JP5621456B2 - Gaze detection device, gaze detection method, computer program for gaze detection, and portable terminal - Google Patents

Description

  The present invention relates to, for example, a gaze detection device that detects a gaze direction by detecting a cornea reflection image, a gaze detection method, a computer program for gaze detection, and a portable terminal using such a gaze detection device.

2. Description of the Related Art Conventionally, a line-of-sight detection device that detects a user's line-of-sight direction by detecting a cornea reflection image (Purkinje image) of a light source by analyzing an image obtained by photographing a user's eyes has been developed.
Such a gaze detection device cannot detect the gaze direction unless it can detect the Purkinje image. For example, when the user is wearing spectacles, the line-of-sight detection apparatus may not be able to detect the Purkinje image because the illumination light from the light source overlaps with the Purkinje image. Therefore, if the eye-gaze detection device cannot detect the Purkinje image on the image obtained by photographing the user's eyes, the photographing condition or the image acquisition condition is changed to photograph the user's eyes again, and the Purkinje image on the re-photographed image. Has been proposed (see, for example, Patent Documents 1 to 4). For example, in one example of such a known technique, the line-of-sight detection apparatus has a plurality of illumination light sources, and images the user's eyes with any one of the light sources turned on. When the Purkinje image cannot be detected on the captured image, the visual line detection device captures the user's eyes again with the other light sources turned on, and detects the Purkinje image on the recaptured image.

JP-A-6-138367 JP 2005-296383 A JP 2003-339642 A JP-T-2001-524703

  However, if the line-of-sight detection device performs a Purkinje image detection process and fails to detect the eye, the user's eyes are re-photographed, so that a time lag occurs until the Purkinje image can be detected. If the user moves his / her eyes between the first shooting and the second shooting, the line-of-sight detection device cannot detect the user's line-of-sight direction at the time of the first shooting.

  In view of this, it is an object of the present specification to provide a line-of-sight detection apparatus and a line-of-sight detection method capable of reducing the time lag from the failure of detection of the Purkinje image of the light source to the detection of the Purkinje image of the light source.

  According to one embodiment, a line-of-sight detection device is provided. The line-of-sight detection device captures a user's eyes while a plurality of light sources arranged at different positions and a first light source of the plurality of light sources illuminating the user's eyes for each predetermined period. The first image, an imaging unit that generates a second image of the user's eyes while the second light source of the plurality of light sources illuminates the user's eyes, and every predetermined period Then, it is determined from one of the first image and the second image whether the cornea reflection image and the user's pupil center of gravity by one of the first light source and the second light source can be detected, and the cornea reflection. When the image and the pupil centroid are detected, the user's line-of-sight direction is detected according to the positional relationship between the pupil centroid and the corneal reflection image. On the other hand, when either the corneal reflection image or the pupil centroid cannot be detected, Pupil centroid and corneal reflection image from the other of the image and the second image Detect, and a control unit for detecting a gaze direction of the user in accordance with the positional relationship between the pupil centroid and the cornea reflection image.

  According to another embodiment, a mobile terminal is provided. In this portable terminal, a display unit, a plurality of light sources arranged at different positions around the display unit, and a first light source of the plurality of light sources illuminate the user's eyes at predetermined intervals. A first image obtained by photographing the user's eyes while the second light source among the plurality of light sources illuminates the user's eyes, and a second image obtained by photographing the user's eyes And a corneal reflection image of one of the first light source and the second light source and the user's pupil center of gravity can be detected from one of the first image and the second image for each predetermined period. If the corneal reflection image and the pupil centroid can be detected, the gaze position on which the user is gazing on the display unit is detected according to the positional relationship between the pupil centroid and the corneal reflection image. If either the reflected image or the pupil center of gravity cannot be detected, the first image and the second A control unit that detects the center of gravity of the pupil and the corneal reflection image from the other of the images, detects the gaze position according to the positional relationship between the center of gravity of the pupil and the corneal reflection image, and executes the application that executes processing associated with the gaze position Part.

  According to still another embodiment, a gaze detection method is provided. In this line-of-sight detection method, a first image obtained by photographing a user's eye while a first light source among a plurality of light sources arranged at different positions is illuminating the user's eye for each predetermined period. And a second image obtained by photographing the user's eye while the second light source among the plurality of light sources illuminates the user's eye, and the first image and the second image are acquired at predetermined intervals. When it is determined from one of the images that the cornea reflection image by one of the first light source and the second light source and the user's pupil centroid can be detected, and the corneal reflection image and the pupil centroid are detected If the user's line-of-sight direction is detected according to the positional relationship between the pupil centroid and the corneal reflection image, and if either the corneal reflection image or the pupil centroid cannot be detected, the first image and the second image Detects the center of gravity of the pupil and the corneal reflection image from the other side of the And detecting a gaze direction of the user in accordance with the positional relationship between Izo.

The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  The line-of-sight detection apparatus and line-of-sight detection method disclosed in this specification can shorten the time lag from when the detection of the Purkinje image of the light source fails until the Purkinje image of the light source can be detected.

It is a hardware block diagram of the portable terminal carrying the eyes | visual_axis detection apparatus by one Embodiment. It is a schematic front view of a portable terminal. It is a figure which shows an example of the image which the Purkinje image of the light source and the image of the light source reflected by the spectacle lens were reflected. It is a functional block diagram of the control part of a gaze detection device. It is a figure which shows an example of a reference table. It is a figure which shows the operation | movement flowchart of a calibration process. It is a figure which shows the operation | movement flowchart of a gaze detection process.

Hereinafter, a gaze detection apparatus according to one embodiment will be described with reference to the drawings.
This line-of-sight detection device has a plurality of light sources arranged at different positions, and illuminates the eyes by photographing the user's eyes a plurality of times while changing the light sources to be lit among the respective light sources at regular intervals. A plurality of images having different light sources are generated. This gaze detection device sequentially detects the Purkinje image and the center of gravity of the pupil for a plurality of images generated at regular intervals, and uses the images from which the Purkinje image and the center of gravity of the pupil are detected to detect the user's gaze. To detect. In this way, this line-of-sight detection device does not re-photograph the user's eyes after failing to detect the Purkinje image or the center of gravity of the pupil, but acquires a plurality of images with different light sources that illuminate the user's eyes in advance. Therefore, even when the detection of the Purkinje image or the center of gravity of the pupil fails on the first analyzed image, the time lag until the user's line-of-sight direction can be detected can be shortened.

  FIG. 1 is a hardware configuration diagram of a mobile terminal equipped with a line-of-sight detection device according to one embodiment. The mobile terminal 1 is a computer having a user interface 2, a memory 3, a control unit 4, a housing 5, and a line-of-sight detection device 10. The mobile terminal 1 is, for example, a mobile phone or a mobile information terminal. Furthermore, the mobile terminal 1 may have a communication interface circuit (not shown) for connecting the mobile terminal 1 to another device. Note that FIG. 1 is a diagram for explaining the constituent elements of the mobile terminal 1 and is not a diagram showing the actual arrangement of the constituent elements of the mobile terminal 1.

The user interface 2 has, for example, a touch panel display. The user interface 2 is arranged on the front surface of the housing 5 such that the display screen of the user interface 2 faces the user facing the front surface of the housing 5. And the user interface 2 displays various icons according to the control signal from the control part 4, for example. In addition, when the user touches the position of the displayed icon, the user interface 2 generates an operation signal corresponding to the position and outputs the operation signal to the control unit 4.
The user interface 2 may have a display device such as a liquid crystal display and a plurality of operation switches such as a keypad.

  The memory 3 includes, for example, a readable / writable nonvolatile semiconductor memory. The memory 3 stores various application programs executed on the control unit 4 and various data. The memory 3 also stores information representing the icon area being displayed on the display screen of the user interface 2.

Furthermore, the memory 3 also functions as a storage unit of the line-of-sight detection device 10. The memory 3 is, for example, a variety of data used for detecting the position at which the user is gazing, such as a reference table that represents the relationship between the position of the pupil centroid relative to the centroid of the Purkinje image and the position at which the user is gazing. Remember. In addition, the memory 3 stores a spectacle wearing flag that is used when the calibration process is performed and that indicates whether the user is wearing spectacles. Furthermore, the memory 3 stores a program for executing the line-of-sight detection process.
The memory circuit that functions as the storage unit of the line-of-sight detection device 10 may be mounted on the mobile terminal 1 separately from the memory 3.

  The line-of-sight detection device 10 obtains the position where the user is gazing on the display screen of the user interface 2, that is, the user's line-of-sight direction. Then, the line-of-sight detection device 10 passes information indicating the position where the user is gazing to the application program being executed by the control unit 4. Details of the line-of-sight detection device 10 will be described later.

The control unit 4 includes one or a plurality of processors and their peripheral circuits. And the control part 4 is connected with each part of the portable terminal 1 via a signal line, and controls the portable terminal 1 whole. For example, the control unit 4 executes processing according to the operation signal received from the user interface 2 and the application program being executed.
The control unit 4 also functions as a control unit of the line-of-sight detection device 10. The function and operation as the control unit of the line-of-sight detection device 10 will be described later.
Further, the control unit 4 is an example of an application execution unit. For example, each position on the display screen of the user interface 2 stored in the memory 3 and the position at which the user is gazing, which is obtained by the line-of-sight detection device 10. Compare the icon display area. And the control part 4 performs the process according to the icon, when the position which the user is gazing and the display area of any icon have overlapped. For example, it is assumed that an icon representing a hotel close to the current location is displayed on the display screen, and each icon is associated with a process of displaying detailed hotel information below the display screen.
In this case, if the position where the user gazes overlaps with the display area of any icon, the control unit 4 causes the user interface 2 to display the detailed information of the hotel related to the icon at the bottom of the display screen. In the present embodiment, since the line-of-sight detection device 10 detects a position at which the user gazes based on a plurality of images with different light sources that illuminate the eyes at regular intervals, the control unit 4 is a position at which the user is gazing. Can be known in each cycle. Therefore, even if the icons that the user watches are changed one after another, the control unit 4 can surely display information associated with the icons that the user has watched.

The line-of-sight detection device 10 includes four light sources 11-1 to 11-4, a camera 12, a memory 3, and a control unit 4 in order to detect a user's line of sight.
The light sources 11-1 to 11-4 illuminate the eyes of the user who operates the mobile terminal 1 and the surroundings thereof. Therefore, each of the light sources 11-1 to 11-4 is, for example, a drive that supplies power from a power source (not shown) to the infrared light emitting diodes according to at least one infrared light emitting diode and a control signal from the control unit 4. Circuit. Each of the light sources 11-1 to 11-4 emits illumination light while receiving a control signal for turning on the light source from the control unit 4.
Note that the number of light sources included in the line-of-sight detection device 10 is not limited to four. The line-of-sight detection device 10 only needs to have two or more light sources arranged at different positions.

The camera 12 is an example of an imaging unit, and generates an image in which the user's eyes are shown. For this purpose, the camera 12 forms an image of a user's eye on the image sensor having a solid-state imaging device arranged in a two-dimensional shape having sensitivity to the light of the light sources 11-1 to 11-4. An imaging optical system. The camera 12 has visible light between the image sensor and the imaging optical system in order to suppress detection of a reflection image by an iris and a corneal reflection image of light from a light source other than the light sources 11-1 to 11-4. You may further have a cut filter.
The number of pixels of the image sensor of the camera 12 is, for example, 1 million, 2 million, or 4 million so that the image has a resolution that can identify the Purkinje images of the light sources 11-1 to 11-4. Can do. An image generated by the image sensor is a color image, and each pixel has three color components of red (R), green (G), and blue (B). Each color component takes any value from 0 to 255, for example. The larger the color component value, the higher the luminance of the color. Alternatively, the image generated by the image sensor may be a monochrome image in which the luminance of each pixel increases as the light intensity increases.
When the camera 12 receives the shooting instruction signal from the control unit 4, the camera 12 takes a picture while any one of the light sources 11-1 to 11-4 is turned on, so that at least the user illuminated by the light source that is turned on. An image with one eye, preferably both eyes, is generated. The camera 12 outputs the generated image to the control unit 4 every time an image is generated.

FIG. 2 is a schematic front view of the mobile terminal 1. In the present embodiment, as shown in FIG. 2, the camera 12 is disposed close to the upper end of the display screen 2 a of the user interface 2 of the mobile terminal 1. The camera 12 is directed so that the user's face facing the display screen 2a can be photographed. The display screen 2a is an example of a gaze area in which the user may gaze.
Each light source 11-1 to 11-4 is arranged along the outer periphery of the user interface 2 so as to illuminate the face of the user facing the display screen 2a. Further, when the user wears spectacles, each of the reflection images of at least one spectacle lens of the light sources 11-1 to 11-4 and the reflection image of the user's cornea are not overlapped on the image plane of the camera 12. The light sources 11-1 to 11-4 are arranged at different positions. For example, at least two of the light sources 11-1 to 11-4 are spaced between the reflected images of the two light sources by the spectacle lens at a distance between the user and the display screen 2a when the user operates the mobile terminal 1. The distance is set to be larger than the diameter of the pupil. For example, the distance between the light source 11-1 and the light source 11-2 is preferably set to 50 mm or more. In this example, the light source 11-1 is arranged on the upper side of the user interface 2. The light sources 11-2 to 11-4 are disposed on the lower side, the left side, and the right side of the user interface 2, respectively.

In general, the surface of the spectacle lens that does not face the user is formed in a convex shape, but its curvature radius is much larger than the curvature radius of the cornea. That is, for the camera 12, both the spectacle lens and the cornea function as convex mirrors, but the absolute value of the optical power of the cornea is stronger than the absolute value of the optical power of the spectacle lens. Therefore, when the position of the light source changes, the amount of change in the position of the reflected image of the light source by the cornea is smaller than the amount of change in the position of the reflected image of the light source by the spectacle lens. Therefore, when the reflected image from the spectacle lens of a certain light source and the reflected image from the cornea overlap on the image plane of the camera 12, the reflected image from the spectacle lens of the light source at another position and the reflected image from the cornea are on the image plane of the camera 12. Become a different position.
Therefore, by changing the light source to be lit among the plurality of light sources 11-1 to 11-4 arranged at different positions, the line-of-sight detection apparatus 1 reflects the reflected image of the illumination light from the light source 11 by the spectacle lens and its The Purkinje image of the illumination light can be separated at different positions on the image plane of the camera 12.

  FIG. 3 is a diagram illustrating an example of an image in which the Purkinje image of the light source and the image of the light source reflected by the spectacle lens are shown. In the image 300 shown in FIG. 3, the eyes of the user illuminated by two light sources arranged in the vertical direction are shown. Bright points 301 and 302 on the image 300 are Purkinje images of two light sources, respectively. On the other hand, the bright spots 311 and 312 are reflection images of two light sources reflected on the surface of the spectacle lens, respectively. Since the reflected image by the spectacle lens is bright, if the reflected image and the Purkinje image of the light source overlap, it is difficult for the line-of-sight detection device 10 to detect the Purkinje image. However, the distance between the bright spot 311 and the bright spot 312 is much larger than the distance between the bright spots 301 and 302. Therefore, when either the upper light source or the lower light source overlaps the Purkinje image of the light source and the reflected image by the spectacle lens, the Purkinje image of the light source and the reflected image by the spectacle lens overlap for the other light source. I understand that it doesn't become.

  Moreover, it is preferable that each light source 11-1 to 11-4 and the camera 12 are arrange | positioned so that the line which connects each light source 11-1 to 11-4 and the camera 12 may remove | deviate from the center of the display screen 2a. By arranging the light sources and the camera 12 in this way, even if the user who is facing the display screen 2a is wearing glasses, the light of the light source reflected by the spectacle lens is directed in a different direction from the camera 12. The possibility increases. Therefore, the line-of-sight detection device 10 can reduce the possibility that the Purkinje image of the light source and the center of gravity of the pupil overlap with the image of the light source by the spectacle lens even when the user is wearing glasses.

As described above, the control unit 4 is also a control unit of the mobile terminal 1 and is connected to each unit of the visual line detection device 10 through a signal line. And the control part 4 controls each part of the visual line detection apparatus 10 by transmitting a control signal to each part of the visual line detection apparatus 10 through a signal line.
Further, the control unit 4 executes a gaze direction detection process for obtaining a position where the user is gazing while an application using information on the position where the user is gazing is executed on the mobile terminal 1. In addition, the control unit 4 pays attention to the relative positional relationship between the Purkinje image of the light source and the pupil center of gravity at a specific timing, such as when the mobile terminal 1 is started or when the above-described application is started. A calibration process is performed to obtain the relationship with the current position.

FIG. 4 is a functional block diagram relating to the line-of-sight detection process of the control unit 4. The control unit 4 includes a light source control unit 21, a camera control unit 22, a face detection unit 23, a Purkinje image detection unit 24, a line-of-sight detection unit 25, and a calibration unit 26.
The light source control unit 21, camera control unit 22, face detection unit 23, Purkinje image detection unit 24, line-of-sight detection unit 25, and calibration unit 26 are functions realized by a computer program executed on a processor included in the control unit 4. It is a module.
Alternatively, the light source control unit 21, the camera control unit 22, the face detection unit 23, the Purkinje image detection unit 24, the line-of-sight detection unit 25, and the calibration unit 26 are each formed as a separate circuit from the processor included in the control unit 4. May be. Alternatively, the light source control unit 21, the camera control unit 22, the face detection unit 23, the Purkinje image detection unit 24, the line-of-sight detection unit 25, and the calibration unit 26 are controlled as one integrated circuit in which circuits corresponding to the respective units are integrated. The mobile terminal 1 may be mounted separately from the processor included in the unit 4. In this case, the integrated circuit may include a storage circuit that stores a reference table or the like separately from the memory 3 and functions as a storage unit included in the visual line detection device 10.

The light source control unit 21 controls the lighting timing of each of the light sources 11-1 to 11-4. Then, the light source control unit 21 transmits a control signal to each light source so that the light source to be turned on is switched at a constant cycle during the execution of the line-of-sight detection process. For example, the light source control unit 21 may transmit a control signal to each light source so that the light source to be turned on is switched in the order of 11-1 → 11-2 → 11-3 → 11-4 at regular intervals. .
The light source control unit 21 may transmit a control signal to each light source so that two or more light sources are turned on simultaneously. When a plurality of light sources are turned on at the same time, the Purkinje image also includes a plurality of light source images that are turned on. Therefore, the line-of-sight detection device 10 uses a light source in which the arrangement pattern of bright spots on the image is actually turned on. It can be checked whether or not it matches the arrangement pattern. Therefore, the line-of-sight detection device 10 can easily distinguish between an image of another light source such as an indoor lamp or the sun and an image of the light source included in the line-of-sight detection device 10.

For example, the light source control unit 21 transmits a control signal to each light source so that the light source to be turned on is switched in the order of (11-1, 11-3) → (11-2, 11-4) at regular intervals. May be. Alternatively, the light source control unit 21 determines that the light source to be lit is (11-1, 11-3) → (11-2, 11-4) → (11-1, 11-4) → (11 at regular intervals. -2, 11-3) A control signal may be transmitted to each light source so as to be switched in the order.
Further, the fixed period is determined so that, for example, a necessary operation is performed following the movement of the user's line of sight. For example, the fixed period is set to 50 milliseconds to 100 milliseconds.
Further, the light source control unit 21 sequentially turns on each light source according to the lighting order of the light sources in one cycle during the execution of the calibration process.
Each time the light source control unit 21 switches the light source to be turned on, the light source control unit 21 notifies the camera control unit 22 accordingly.

The camera control unit 22 controls the timing at which the camera 12 captures an image during the execution of the line-of-sight detection process or the calibration process. When notified from the light source control unit 21 that the light source to be turned on is switched, the camera control unit 22 transmits a shooting instruction signal for causing the camera 12 to perform shooting. When receiving an image from the camera 12, the camera control unit 22 temporarily stores the image in the memory 3.
Therefore, the camera control unit 22 receives a plurality of images from the camera 12 at regular intervals in which the lighting patterns of the light sources 11-1 to 11-4 make a round. For example, as described above, when the light sources 11-1 to 11-4 are turned on one by one within a certain period, the camera control unit 22 displays four images taken while each light source is turned on. receive. In addition, when the light sources 11-1 to 11-4 are turned on in two patterns two by two within a certain period, the camera control unit 22 is photographed while the light sources are turned on with the respective lighting patterns. Receive two images.

The face detection unit 23 performs an operation on the image received from the camera 12 on the image received from the camera 12 in order to identify a region where the user's eyes may be captured on the image during the line-of-sight detection process or the calibration process. Detect the area where the face is reflected.
In the present embodiment, at the time of shooting, the user's face is illuminated with infrared light from the light sources 11-1 to 11-4, and the reflectance of the skin to the infrared light is relatively high (for example, the reflection of the skin). The rate is a few tens of percent in the near-infrared wavelength region), so the brightness of the pixels where the facial skin is reflected on the image is high. On the other hand, since the area of the hair or the user's back area on the image has a low reflectance with respect to infrared light or is far from the light source, the luminance of the pixel in which the hair or the area behind the user is reflected is relatively low.
Therefore, when the value of each pixel of the image is a color image represented by the RGB color system, the face detection unit 23 converts the value of each pixel into a value represented by the YUV color system. Then, the face detection unit 23 extracts pixels whose luminance component (Y component) value of each pixel is equal to or greater than a predetermined threshold as face area candidate pixels that may have a face. When the value of each pixel of the image is a monochrome image representing luminance, the face detection unit 23 compares the value of each pixel with a predetermined threshold value. For example, the predetermined threshold is set to a value obtained by multiplying the maximum value of the luminance component on the image by 0.8.

When the operation of the mobile terminal 1 is executed according to the user's line of sight, the user's face is facing the front of the mobile terminal 1 and at a position approximately several tens of centimeters away from the front of the mobile terminal 1. It is assumed that there is. Therefore, the area occupied by the user's face on the image is relatively large, and the size of the area occupied by the face on the image is estimated to some extent.
Therefore, the face detection unit 23 performs a labeling process on the face area candidate pixels, and sets a set of face area candidate pixels adjacent to each other as a face candidate area. Then, the face detection unit 23 determines whether the size of the face candidate area is included in a reference range corresponding to the size of the user's face. If the size of the face candidate area is included in the reference range corresponding to the size of the user's face, the face detection unit 23 determines that the face candidate area is a face area in which the user's face is reflected.
Note that the size of the face candidate area is represented by, for example, the maximum number of pixels in the horizontal direction of the face candidate area. In this case, the reference range is set to, for example, from 1/4 to 2/3 of the number of pixels in the horizontal direction of the image. Alternatively, the size of the face candidate area may be represented by the number of pixels included in the face candidate area, for example. In this case, the reference range is set to, for example, 1/16 or more to 4/9 or less of the number of pixels of the entire image.

  The face detection unit 23 may add not only the size of the face candidate area but also the shape of the face candidate area to the determination condition for determining the face candidate area as the face area. A human face generally has a substantially elliptical shape. Therefore, the face detection unit 23, for example, when the size of the face candidate area is included in the reference range and the circularity of the face candidate area is equal to or greater than a predetermined threshold corresponding to a general face contour. Alternatively, the face candidate area may be a face area. The face detection unit 23 obtains the total number of pixels located on the contour of the face candidate area as the perimeter of the face candidate area, and multiplies the total number of pixels in the face candidate area by 4π as the square of the perimeter. The circularity can be calculated by dividing.

  Alternatively, the face detection unit 23 may approximate the face candidate area to an ellipse by applying the least square method by applying the coordinates of each pixel on the contour of the face candidate area to an elliptic equation. The face detection unit 23 may use the face candidate area as a face area when the ratio of the major axis to the minor axis of the ellipse is included in the range of the ratio of the major axis to the minor axis of the face. Note that, when the shape of the face candidate region is evaluated by ellipse approximation, the face detection unit 23 may detect an edge pixel corresponding to an edge by performing an inter-pixel calculation on the luminance component of each pixel of the image. . In this case, the face detection unit 23 connects the edge pixels using, for example, a labeling process, and uses the edge pixels connected to a certain length or more as the contour of the face candidate region.

  The face detection unit 23 may detect the face area according to any of various other methods for detecting the face area shown on the image. For example, the face detection unit 23 performs template matching between the face candidate area and a template corresponding to a general face shape, calculates the degree of coincidence between the face candidate area and the template, and the degree of coincidence is predetermined. If the value is greater than or equal to the value, the face candidate area may be determined as a face area.

When the face area can be extracted, the face detection unit 23 generates information representing the face area. For example, the information representing the face area may be a binary image having the same size as the image and having different pixel values for the pixels in the face area and the pixels outside the face area.
The face detection unit 23 passes information representing the face area to the Purkinje image detection unit 24.

The Purkinje image detection unit 24 detects a region in which the user's eyes are reflected in the face region during execution of the line-of-sight detection processing or calibration processing, and detects a Purkinje image in the region in which the eyes are reflected.
The luminance of the pixel corresponding to the eye is significantly different from the luminance of the pixel corresponding to the periphery of the eye. Therefore, the Purkinje image detection unit 24 detects, for each pixel in the face region, an edge pixel whose luminance changes in the vertical direction by performing a difference calculation between adjacent pixels in the vertical direction using, for example, a Sobel filter. The Purkinje image detection unit 24 sets, for example, a region surrounded by two edge lines in which a predetermined number or more of edge pixels are connected in a substantially horizontal direction corresponding to the size of the eye.
Alternatively, the Purkinje image detection unit 24 detects a region that most closely matches the template in the face region by template matching between the template representing the image of the eye on the image and the face region, and uses the detected region as the eye region. It is good.

  Further, the Purkinje image detection unit 24 detects an area where the pupil is reflected in the eye area. In the present embodiment, the Purkinje image detection unit 24 performs template matching between a template corresponding to the pupil and the eye region, and detects a region having the highest degree of matching with the template in the eye region. Then, the Purkinje image detection unit 24 determines that the pupil is reflected in the detected area when the maximum value of the coincidence is higher than a predetermined coincidence threshold. A plurality of templates may be prepared according to the size of the pupil. In this case, the Purkinje image detection unit 24 executes template matching between each template and the eye region, and obtains the highest value of the matching degree. If the highest coincidence value is higher than the coincidence degree threshold, the Purkinje image detection unit 24 determines that the pupil appears in an area overlapping the template corresponding to the highest coincidence value. Note that the degree of coincidence is calculated as, for example, a normalized cross-correlation value between a template and a region overlapping the template. The coincidence threshold is set to 0.7 or 0.8, for example.

The luminance of the region where the pupil is reflected is lower than the luminance of the surrounding region, and the pupil is substantially circular. Therefore, the Purkinje image detection unit 24 sets two concentric rings with different radii in the eye region. When the difference value obtained by subtracting the average luminance value of the inner pixels from the average luminance value of the pixels corresponding to the outer ring is larger than a predetermined threshold value, the Purkinje image detection unit 24 is surrounded by the inner ring. The region may be the pupil region. The Purkinje image detection unit 24 may add to the condition for detecting the pupil region that the average luminance value of the region surrounded by the inner ring is equal to or less than a predetermined threshold value. In this case, for example, the predetermined threshold is set to a value obtained by adding 10% to 20% of the difference between the maximum luminance value and the minimum luminance value in the eye region to the minimum luminance value.
When the Purkinje image detection unit 24 succeeds in detecting the pupil region, the Purkinje image detection unit 24 calculates the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the pupil region as the coordinates of the center of gravity of the pupil region. . On the other hand, if the Purkinje image detection unit 24 fails to detect the pupil region, the Purkinje image detection unit 24 returns a signal indicating that to the control unit 4.

The Purkinje image detection unit 24 detects Purkinje images of the light sources 11-1 to 11-4 in the eye region. The luminance of the region where the Purkinje images of the light sources 11-1 to 11-4 are reflected is higher than the luminance of the surrounding region, and the luminance value is substantially saturated (that is, the luminance value is the pixel value). It is almost the maximum value of brightness to obtain). In addition, the shape of the region in which the Purkinje images of the light sources 11-1 to 11-4 are reflected substantially matches the shape of the light emitting surface of each light source. Therefore, the Purkinje image detection unit 24 sets two rings that have a shape that substantially matches the contour shape of the light emitting surface of the light source in the eye region, and that have different sizes and centers. Then, the Purkinje image detection unit 24 has a difference value obtained by subtracting the average value of the luminance of the outer pixels from the average value of the luminance of the pixels corresponding to the inner ring, greater than a predetermined difference threshold value, and When the inner brightness average value is higher than a predetermined brightness threshold, the area surrounded by the inner ring is used as the Purkinje image of the light sources 11-1 to 11-4. Note that the difference threshold value can be, for example, an average value of difference values between neighboring pixels in the eye region. The predetermined luminance threshold can be set to 80% of the maximum luminance value in the eye region, for example.
Note that the Purkinje image detection unit 24 may detect the region in which the pupil is shown using any of various other methods for detecting the region in which the pupil is shown on the image. Similarly, the Purkinje image detection unit 24 detects the region where the Purkinje image of the light source is captured using any of various other methods for detecting the region where the Purkinje image of the light source is captured on the image. Also good.

When the Purkinje image detection unit 24 succeeds in detecting any of the Purkinje images of the light sources 11-1 to 11-4, the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the light source. Is calculated as the coordinates of the center of gravity of the Purkinje image. On the other hand, if the Purkinje image detection unit 24 fails to detect the Purkinje image of the light source, the Purkinje image detection unit 24 returns a signal indicating that to the control unit 4.
When performing the calibration process, the Purkinje image detection unit 24 notifies the calibration unit 26 of the center of gravity of the Purkinje image and the center of the pupil. On the other hand, the Purkinje image detection unit 24 notifies the gaze detection unit 25 of the center of gravity of the Purkinje image and the center of the pupil when performing the gaze detection process.

  The line-of-sight detection unit 25 determines the position at which the user is gazing on the display screen 2a of the user interface 2, that is, the direction of the line of sight of the user, based on the center of gravity of the Purkinje image and the center of the pupil during the line-of-sight detection process. To detect. Hereinafter, for the sake of convenience, a position where the user is gazing on the display screen 2a is simply referred to as a gazing position.

Since the surface of the cornea is substantially spherical, the position of the Purkinje image of the light source is substantially constant regardless of the line-of-sight direction. On the other hand, the center of gravity of the pupil moves according to the user's line-of-sight direction. Therefore, the line-of-sight detection unit 25 can detect the user's line-of-sight direction by obtaining the relative position of the pupil center of gravity with respect to the center of gravity of the Purkinje image.
In the present embodiment, the line-of-sight detection unit 25 determines the relative position of the pupil centroid relative to the centroid of the Purkinje image of the light source, for example, the horizontal direction of the centroid of the Purkinje image from the horizontal coordinate and the vertical coordinate of the pupil centroid. It is obtained by subtracting the coordinate and the vertical coordinate. The line-of-sight detection unit 25 determines the user's gaze position by referring to a reference table that represents the relationship between the relative position of the pupil center of gravity and the user's gaze position on the display screen 2a.
The line-of-sight detection unit 25 notifies the application program being executed by the control unit 4 of the user's gaze position.

  FIG. 5 is a diagram illustrating an example of a reference table. In each column of the left column of the reference table 500, coordinates of the relative position of the center of the pupil relative to the center of gravity of the Purkinje image of the light source are represented. In addition, in each column in the right column of the reference table 500, the coordinates of the gaze position corresponding to the coordinates of the relative position of the pupil centroid in the same row are represented. Note that the coordinates of the relative position of the center of gravity of the pupil are expressed in units of pixels on the image, while the coordinates of the gaze position are expressed in units of pixels on the display screen 2a. For example, the row 501 indicates that the coordinates of the gaze position with respect to the coordinates (0,0) of the relative position of the pupil center of gravity are (cx, cy). Note that cx and cy are, for example, the horizontal and vertical coordinates of the center of the display screen 2a. The row 502 indicates that the gaze position coordinate is (cx-10, cy) with respect to the relative position coordinate (−2, 0) of the pupil center of gravity.

The calibration unit 26 updates the reference table. The calibration unit 26 determines the execution timing of the calibration process.
For example, when the calibration unit 26 receives a signal indicating that the mobile terminal 1 has been activated from the control unit 4 or receives a signal indicating that a specific application has been activated, the calibration unit 26 performs the first calibration. Execute the process.
When the calibration process is started, the calibration unit 26 displays a calibration mark at a predetermined reference position on the display screen 2 a of the user interface 2. The reference position can be, for example, either the center of the display screen 2a or the four corners of the display screen 2a. In addition, during the calibration, the user gazes at the calibration mark displayed on the display screen 2a.

The calibration unit 26 receives from the Purkinje image detection unit 24 the coordinates of the center of gravity of the Purkinje image and the center of the pupil, which are detected based on the image taken when the mark is displayed at the center of the display screen 2a. Then, the calibration unit 26 updates the reference table so that the relative position of the pupil center of gravity based on the center of the Purkinje image corresponds to the center of the display screen 2a.
Further, the calibration unit 26 obtains the coordinates of the center of gravity of the Purkinje image and the center of the pupil detected based on the image taken when the mark is displayed at the upper left corner of the display screen 2a from the Purkinje image detection unit 24. receive. Then, the calibration unit 26 updates the reference table so that the relative position of the center of the pupil relative to the center of the Purkinje image at this time corresponds to the upper left corner of the display screen 2a. Similarly, the calibration unit 26 determines the relative position of the pupil centroid relative to the centroid of the Purkinje image obtained from the captured image when the mark is displayed at each corner of the display screen 2a. The reference table is updated to correspond to the display position.
Further, the calibration unit 26 corresponds to the relative positions of the pupil centroid relative to the other positions on the display screen 2a with respect to the centroid of the Purkinje image as the positions of the five marks already obtained. For example, linear interpolation is performed using the relative position of the center of gravity of the pupil. And the calibration part 26 updates a reference table so that the relative position of the pupil center of gravity with respect to the position on various display screens 2a may be matched.
When the calibration unit 26 determines the relative position of the center of the pupil with respect to each point on the display screen 2 a, the calibration unit 26 stores the updated reference table in the memory 3.
The reference table may be created in advance for each light source that is turned on, and each reference table may be stored in the memory 3 in association with the identification number of the corresponding light source. In this case, the calibration unit 26 acquires identification information about the light source that is lit from the control unit 4, and specifies a reference table to be updated.

  The calibration unit 26 determines the calibration execution timing based on whether or not the Purkinje image has been successfully detected. For example, when the user wears spectacles, the reflection image of the light source by the spectacle lens may overlap with the pupil center of gravity or the Purkinje image of the light source. Therefore, the Purkinje image detection unit 24 is more effective than when the user is not wearing spectacles. There is a high possibility that the detection of the Purkinje image of the light source will fail. Therefore, the calibration unit 26 determines the calibration execution timing so that the calibration execution frequency when the user is wearing glasses is higher than the execution frequency when the user is not wearing glasses.

In the present embodiment, if the Purkinje image detection unit 24 fails to detect the Purkinje image or pupil center of gravity of the light source for an image when any of the light sources is lit, it is estimated that the user is wearing glasses. . Therefore, in this case, the calibration unit 26 sets the execution timing of the calibration process so that the calibration process is executed every time a predetermined period elapses. The predetermined period is set to 5 minutes, for example. The calibration unit 26 may also execute the calibration process when the portable terminal 1 starts an application that displays information accompanied by some operation on the display screen 2a. Further, the calibration unit 26 may execute the calibration process when the gravity center position of the face area in the latest image is shifted by a predetermined number of pixels or more from the gravity center position of the face area in the previously captured image.
Further, the calibration unit 26 may set the order of images for detecting the pupil center of gravity and the Purkinje image in each cycle when the line-of-sight detection process is executed. For example, the calibration unit 26 may first detect the pupil centroid and the Purkinje image with respect to an image taken while the light source corresponding to the Purkinje image of the light source or the image for which the pupil centroid has been successfully detected. It may be. As a result, the line-of-sight detection device 1 can reduce the possibility that the detection of the pupil center of gravity and the Purkinje image will fail during the execution of the line-of-sight detection process, thereby reducing the computational load required for the line-of-sight detection process.

On the other hand, if the calibration unit 26 succeeds in detecting the Purkinje image or pupil center of gravity of the light source for the image captured when any light source is lit, the user is Presumed not wearing glasses.
Therefore, in this case, the calibration unit 26 executes the calibration process when the mobile terminal 1 starts an application that displays information accompanying some operation on the display screen 2a. The calibration unit 26 may also execute the calibration process when the gravity center position of the face area in the latest image is shifted by a predetermined number of pixels or more from the gravity center position of the face area in the previously captured image. .

FIG. 6 shows an operation flowchart of the calibration process executed by the control unit 4.
As shown in FIG. 6, the control unit 4 displays a predetermined mark at the reference position on the display screen 2 a of the user interface 2. In addition, the control unit 4 sets a spectacle wearing flag indicating whether or not the user is wearing spectacles to a value indicating that the user is not wearing spectacles (hereinafter referred to as OFF for convenience) (step S101).
Next, the control unit 4 acquires a plurality of images taken with different light sources to be lit (step S102). Then, the control unit 4 sets a focused image among unfocused images (step S103).

The face detection unit 23 of the control unit 4 extracts a face region where a face is shown on the image of interest (step S104). Then, the face detection unit 23 passes information representing the face area to the Purkinje image detection unit 24 of the control unit 4.
The Purkinje image detection unit 24 detects an eye region within the face region (step S105). Then, the Purkinje image detector 24 detects the center of gravity of the pupil in the eye region (step S106). The Purkinje image detection unit 24 detects the Purkinje image of the light source that is lit in the eye region (step S107).
The Purkinje image detection unit 24 determines whether or not the Purkinje image detection unit 24 has successfully detected the pupil center of gravity and the Purkinje image of the light source (step S108).
When the Purkinje image detection unit 24 has successfully detected the pupil center of gravity and the Purkinje image of the light source (step S108-Yes), the Purkinje image detection unit 24 uses the calibration unit of the control unit 4 to calculate the center of gravity of the Purkinje image of the light source and the pupil center of gravity. 26 is notified.
The calibration unit 26 obtains the relative position of the pupil centroid relative to the centroid of the Purkinje image. And the calibration part 26 updates a reference table based on the relative position and the gaze position determined by the position of the mark currently displayed on the display screen 2a (step S109).
On the other hand, if the Purkinje image detection unit 24 fails to detect the center of the pupil or the Purkinje image of the light source (step S108-No), the control unit 4 indicates that the user wears spectacles based on the value of the spectacle wearing flag. A value is set (step S110). In the following description, the value of the spectacle wearing flag indicating that the user is wearing spectacles is referred to as ON for convenience.
The control unit 4 determines whether or not all images have been set as the image of interest (step S111). If any image is not set as the image of interest (step S111-No), the control unit 4 executes the processing from step S103 onward again.
On the other hand, if all the images are set as the image of interest (step S111—Yes), the calibration unit 26 determines whether or not the glasses wearing flag is ON (step S112).
If the spectacle wearing flag is ON (step S112—Yes), the calibration unit 26 sets the calibration timing so that the calibration frequency is higher than when the user does not wear spectacles (step S113).
On the other hand, if the spectacle wearing flag is OFF (step S112—No), the calibration unit 26 sets the calibration timing so that the calibration frequency is lower than when the user wears spectacles (step S114). .
Step S113 or S114 Thereafter, the controller 4 ends the calibration process. Note that the Purkinje image detection unit 24 may change the order of the processing in step S106 and the processing in step S107. Further, the Purkinje image detection unit 24 may search for the Purkinje image of the user's pupil and light source in the entire face area. In this case, the process of step S105 is omitted.

FIG. 7 shows an operation flowchart of the line-of-sight detection process executed by the control unit 4.
As shown in FIG. 7, the control unit 4 acquires a plurality of images taken with different light sources to be lit (step S <b> 201). Then, the control unit 4 sets a focused image among unfocused images (step S202).
Here, for example, the control unit 4 may set an image of interest among the plurality of images in the order in which the light source that was turned on at the time of shooting is closer to the camera 12. When the user wears spectacles, generally, the optical axis of the spectacle lens faces downward from the front direction of the user's face. Therefore, when the user is facing the display screen 2a, the light emitted from the light source close to the camera 12 is reflected below the light source by the spectacle lens. As a result, since the image (virtual image) of the light source by the spectacle lens can be above the light source, the image of the light source by the spectacle lens does not overlap with the Purkinje image and pupil center of gravity of the light source. Therefore, the closer the image of the light source lit at the time of shooting to the camera 12 is, the higher the possibility that the control unit 4 can detect the pupil center of gravity and the Purkinje image of the light source. Therefore, the control unit 4 can reduce the probability of redoing the detection of the pupil center of gravity and the Purkinje image of the light source by setting the image of interest in such an order. As a result, the calculation load of the eye gaze detection process in each cycle is reduced. Can be reduced.
Alternatively, in the latest calibration, the control unit 4 makes the image in which the Purkinje image of the light source and the pupil center of gravity are detected become the image of interest earlier than the image in which the Purkinje image of the light source or the pupil center of gravity is not detected. As such, the setting order of the image of interest may be determined.

  The face detection unit 23 of the control unit 4 extracts a face region where a face is shown on the image of interest (step S203). Then, the face detection unit 23 determines whether or not the face area has been successfully extracted (step S204). When face area extraction has failed (step S204—No), the user is not facing the mobile terminal 1. Therefore, the control unit 4 ends the line-of-sight detection process. Thereafter, the control unit 4 may execute a calibration process.

On the other hand, when the face detection unit 23 succeeds in extracting the face area (step S204—Yes), the face detection unit 23 passes information representing the face area to the Purkinje image detection unit 24 of the control unit 4.
The Purkinje image detection unit 24 detects an eye region within the face region (step S205). Then, the Purkinje image detector 24 detects the center of gravity of the pupil in the eye region (step S206). The Purkinje image detection unit 24 detects a Purkinje image of a light source that is lit in the eye region (step S207). Then, the Purkinje image detector 24 determines whether or not the Purkinje image detector 24 has successfully detected the pupil center of gravity and the Purkinje image of the light source (step S208).

When the Purkinje image detection unit 24 fails to detect the pupil center of gravity and the Purkinje image of the light source (No at Step S208), the control unit 4 executes the processing after Step S202 again.
On the other hand, when the Purkinje image detection unit 24 has successfully detected the pupil center of gravity and the Purkinje image of the light source (step S208-Yes), the Purkinje image detection unit 24 uses the line of sight of the Purkinje image of the light source and the pupil center of gravity of the control unit 4. Notify the detection unit 25.
The line-of-sight detection unit 25 refers to the reference table and detects a gaze position corresponding to the position of the pupil center of gravity with reference to the center of gravity of the Purkinje image (step S209). The line-of-sight detection unit 25 passes information indicating the gaze position to the application program being executed by the control unit 4.
Thereafter, the control unit 4 ends the line-of-sight detection process. Note that the Purkinje image detection unit 24 may reverse the order of the processing in step S206 and the processing in step S207. Further, the Purkinje image detection unit 24 may search for the Purkinje image of the user's pupil and light source in the entire face area. In this case, the process of step S205 is omitted. Alternatively, when the Purkinje image detection unit 24 fails to detect either the pupil center of gravity or the Purkinje image of the light source on the image of interest, the control unit 4 performs the processing after step S202 without performing the other detection processing. May be re-executed.

  As described above, this line-of-sight detection device captures the eye by photographing the user's eye a plurality of times while changing the light source to be lit among a plurality of light sources arranged at different positions at regular intervals. A plurality of images with different illuminated light sources are generated. Then, this line-of-sight detection apparatus sequentially detects the center of the pupil and the Purkinje image of the light source for a plurality of images in each cycle, and detects the user's line of sight using the image from which the Purkinje image has been detected. As described above, this line-of-sight detection device does not re-photograph the user's eyes after failing to detect the pupil center of gravity or the Purkinje image of the light source, but acquires a plurality of images with different light source lighting patterns in advance. . Therefore, this line-of-sight detection apparatus can shorten the time lag until the Purkinje image can be detected even if the detection of the pupil center of gravity or the Purkinje image of the light source fails on the first analyzed image.

  In addition, this invention is not limited to said embodiment. For example, the camera may generate an image shot with all light sources turned off in each cycle during execution of the line-of-sight detection process. Then, the Purkinje image detection unit of the control unit detects an image of the light source, and an image taken with any light source turned on, and an image taken with all the light sources turned off, A difference image is calculated by performing a difference calculation between corresponding pixels. In the difference image, the values of the pixels other than the region corresponding to the Purkinje image of the light source are very small. Therefore, the Purkinje image detection unit can detect the Purkinje image of the light source by binarizing the difference image with, for example, the average value of the pixels of the difference image. In addition, this makes it possible for the Purkinje image detection unit to erroneously detect the image of the other light source as the Purkinje image of the light source included in the line-of-sight detection device, even if an image of the other light source such as the sun or room light is captured on the cornea. Can be prevented.

  Further, the camera may generate an image shot with all light sources turned off and an image shot with all light sources turned on in each cycle during execution of the line-of-sight detection process. The Purkinje image detection unit of the control unit calculates a difference between corresponding pixels between an image shot with all the light sources turned on and an image shot with all the light sources turned off. Go to calculate the difference image. In the difference image, the values of the pixels other than the region corresponding to the Purkinje image of the light source are very small. Therefore, the Purkinje image detection unit binarizes the difference image by, for example, the average value of the pixels of the difference image, thereby obtaining a Purkinje image candidate area in which the Purkinje image of the light source may be reflected. Can be identified. Therefore, the Purkinje image detection unit may execute the detection process of the Purkinje image of the light source in the Purkinje image candidate region. In addition, this makes it possible for the Purkinje image detection unit to erroneously detect the image of the other light source as the Purkinje image of the light source included in the line-of-sight detection device, even if an image of the other light source such as the sun or room light is captured on the cornea. Can be prevented.

  In still another modification, the reference table may associate the relative position of the center of the pupil with respect to the Purkinje image of the light source, and the yaw angle and pitch angle representing the user's line-of-sight direction. In this case, for example, when the user is gazing at the center of the display screen, the yaw angle and the pitch angle are each set to 0 ° as the angle representing the user's line-of-sight direction. The yaw angle and pitch angle when the user is gazing at each other point on the display screen is the distance between the three points and the angle by assuming the distance between the user's eyes and the center of the display screen. It is calculated | required by the formula showing a relationship.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
A plurality of light sources arranged at different positions;
For each predetermined period, the first image of the user's eyes while the first light source of the plurality of light sources illuminates the user's eyes, and the second of the plurality of light sources. An imaging unit that generates a second image of the user's eyes while the light source is illuminating the user's eyes;
At each predetermined period, from one of the first image and the second image, a corneal reflection image by one of the first light source and the second light source, and the pupil center of gravity of the user, When the cornea reflection image and the pupil centroid are detected, the gaze direction of the user is detected according to the positional relationship between the pupil centroid and the corneal reflection image, while the cornea When either the reflected image or the pupil centroid cannot be detected, the pupil centroid and the corneal reflection image are detected from the other image of the first image and the second image, and the pupil centroid and the cornea are detected. A control unit that detects the user's line-of-sight direction according to the positional relationship with the reflected image;
A line-of-sight detection apparatus comprising:
(Appendix 2)
The first light source is closer to the imaging unit than the second light source,
The control unit determines whether or not the pupil centroid and the corneal reflection image can be detected from the first image, and when the corneal reflection image or the pupil centroid cannot be detected from the first image, The line-of-sight detection device according to appendix 1, which detects the pupil center of gravity and the corneal reflection image.
(Appendix 3)
The control unit detects the line-of-sight direction by referring to a reference table representing a correspondence between a positional relationship between the cornea reflection image and the pupil center of gravity and the line-of-sight direction, and the control unit includes the first When it is not possible to detect either the cornea reflection image or the pupil center of gravity for at least one of the image and the second image, the execution frequency of the calibration for updating the reference table is set as the frequency of the first and second images. The line-of-sight detection device according to appendix 1, wherein the eye-gaze reflection image and the pupil center of gravity are set to be higher than the calibration execution frequency in both cases.
(Appendix 4)
In the calibration, when the calibration cannot detect either the cornea reflection image or the pupil center of gravity for the first image, the control unit is more than the first image in each predetermined period after the calibration. First, it is determined whether the pupil centroid and the corneal reflection image can be detected from the second image. On the other hand, in the calibration, either the corneal reflection image or the pupil centroid is determined for the second image. If it is not detected, it is determined whether or not the pupil center of gravity and the corneal reflection image can be detected from the first image before the second image in each predetermined period after the calibration. The line-of-sight detection device described.
(Appendix 5)
The plurality of light sources and the imaging unit are arranged around a gaze area where a user may gaze, and for each of the plurality of light sources, a straight line connecting the light source and the imaging unit is the gaze area. The line-of-sight detection device according to any one of appendices 1 to 4, wherein the plurality of light sources and the imaging unit are arranged so as not to pass through a center.
(Appendix 6)
A display unit;
A plurality of light sources arranged at different positions around the display unit;
For each predetermined period, the first image of the user's eyes while the first light source of the plurality of light sources illuminates the user's eyes, and the second of the plurality of light sources. An imaging unit that generates a second image of the user's eyes while the light source is illuminating the user's eyes;
At each predetermined period, from one of the first image and the second image, a corneal reflection image by one of the first light source and the second light source, and the pupil center of gravity of the user, When the cornea reflection image and the pupil centroid can be detected, the gaze that the user is gazing on the display unit according to the positional relationship between the pupil centroid and the corneal reflection image If the position is detected and, on the other hand, either the corneal reflection image or the pupil centroid cannot be detected, the pupil centroid and the corneal reflection image are obtained from the other of the first image and the second image. A control unit that detects and detects the gaze position according to a positional relationship between the pupil center of gravity and the corneal reflection image;
An application execution unit that executes processing associated with the gaze position;
A mobile terminal.
(Appendix 7)
Computer
A first image obtained by photographing the user's eye while a first light source among a plurality of light sources arranged at different positions for each predetermined period illuminates the user's eye; Acquiring a second image of the user's eyes while a second light source of the light sources illuminates the user's eyes;
At each predetermined period, from one of the first image and the second image, a corneal reflection image by one of the first light source and the second light source, and the pupil center of gravity of the user, Whether or not can be detected,
When the cornea reflection image and the pupil centroid are detected, the user's line-of-sight direction is detected according to the positional relationship between the pupil centroid and the corneal reflection image, while either the corneal reflection image or the pupil centroid Is not detected, the pupil centroid and the corneal reflection image are detected from the other image of the first image and the second image, and according to the positional relationship between the pupil centroid and the corneal reflection image. Detecting the gaze direction of the user;
A gaze detection method including the above.
(Appendix 7)
A first image obtained by photographing the user's eye while a first light source among a plurality of light sources arranged at different positions for each predetermined period illuminates the user's eye; Acquiring a second image of the user's eyes while a second light source of the light sources illuminates the user's eyes;
At each predetermined period, from one of the first image and the second image, a corneal reflection image by one of the first light source and the second light source, and the pupil center of gravity of the user, Whether or not can be detected,
When the cornea reflection image and the pupil centroid are detected, the user's line-of-sight direction is detected according to the positional relationship between the pupil centroid and the corneal reflection image, while either the corneal reflection image or the pupil centroid Is not detected, the pupil centroid and the corneal reflection image are detected from the other image of the first image and the second image, and according to the positional relationship between the pupil centroid and the corneal reflection image. Detecting the gaze direction of the user;
A computer program for eye-gaze detection that causes a computer to execute the above.

DESCRIPTION OF SYMBOLS 1 Portable terminal 2 User interface 3 Memory 4 Control part 5 Housing 10 Eye gaze detection apparatus 11-1 to 11-4 Light source 12 Camera 21 Light source control part 22 Camera control part 23 Face detection part 24 Purkinje image detection part 25 Eye gaze detection part 26 Calibration Section

Claims (13)

A plurality of light sources arranged at different positions;
For each predetermined period, the first image of the user's eyes while the first light source of the plurality of light sources illuminates the user's eyes, and the second of the plurality of light sources. An imaging unit that generates a second image of the user's eyes while the light source is illuminating the user's eyes;
For each predetermined period , the first light source and the second light source among the plurality of light sources are switched and turned on, and the first light source is selected from one of the first image and the second image. It is determined whether the cornea reflection image by one of the light source and the second light source and the pupil centroid of the user can be detected, and when the corneal reflection image and the pupil centroid are detected, the pupil centroid and the cornea If the user's line-of-sight direction is detected according to the positional relationship with the reflection image, and if either the cornea reflection image or the pupil center of gravity cannot be detected, the first image and the second image A control unit that detects the pupil centroid and the corneal reflection image from the other image, and detects the user's line-of-sight direction according to a positional relationship between the pupil centroid and the corneal reflection image;
A line-of-sight detection apparatus comprising: The first light source is closer to the imaging unit than the second light source,
The control unit determines whether or not the pupil centroid and the corneal reflection image can be detected from the first image, and when the corneal reflection image or the pupil centroid cannot be detected from the first image, The line-of-sight detection device according to claim 1, wherein the eyeball center of gravity and the cornea reflection image are detected. The control unit detects the line-of-sight direction by referring to a reference table representing a correspondence between a positional relationship between the cornea reflection image and the pupil center of gravity and the line-of-sight direction, and the control unit includes the first When it is not possible to detect either the cornea reflection image or the pupil center of gravity for at least one of the image and the second image, the execution frequency of the calibration for updating the reference table is set as the frequency of the first and second images. The line-of-sight detection device according to claim 1, wherein the line-of-sight detection device is set to be higher than the calibration execution frequency when the cornea reflection image and the pupil center of gravity can be detected for both.   In the calibration, when the calibration cannot detect either the cornea reflection image or the pupil center of gravity for the first image, the control unit is more than the first image in each predetermined period after the calibration. First, it is determined whether the pupil centroid and the corneal reflection image can be detected from the second image. On the other hand, in the calibration, either the corneal reflection image or the pupil centroid is determined for the second image. 4. If not detectable, it is determined whether or not the pupil centroid and the cornea reflection image can be detected from the first image before the second image in each predetermined period after the calibration. The line-of-sight detection device described in 1. A display unit;
A plurality of light sources arranged at different positions around the display unit;
For each predetermined period, the first image of the user's eyes while the first light source of the plurality of light sources illuminates the user's eyes, and the second of the plurality of light sources. An imaging unit that generates a second image of the user's eyes while the light source is illuminating the user's eyes;
For each predetermined period , the first light source and the second light source among the plurality of light sources are switched and turned on, and the first light source is selected from one of the first image and the second image. It is determined whether or not the cornea reflection image by one of the light source and the second light source and the pupil centroid of the user can be detected, and when the corneal reflection image and the pupil centroid can be detected, the pupil centroid and the cornea If the gaze position on which the user is gazing on the display unit is detected according to the positional relationship with the reflection image, and if either the cornea reflection image or the pupil center of gravity cannot be detected, the first image And a control unit that detects the pupil centroid and the corneal reflection image from the other image of the second images, and detects the gaze position according to a positional relationship between the pupil centroid and the corneal reflection image;
An application execution unit that executes processing associated with the gaze position;
A mobile terminal. The first light source is closer to the imaging unit than the second light source,
The control unit determines whether or not the pupil centroid and the corneal reflection image can be detected from the first image, and when the corneal reflection image or the pupil centroid cannot be detected from the first image, The mobile terminal according to claim 5, wherein the center of gravity of the pupil and the corneal reflection image are detected. The control unit detects the line-of-sight direction by referring to a reference table indicating a correspondence between a positional relationship between the cornea reflection image and the pupil center of gravity and the line-of-sight direction; and
When the control unit cannot detect either the corneal reflection image or the pupil center of gravity for at least one of the first image and the second image, the execution frequency of calibration for updating the reference table, The mobile terminal according to claim 5, wherein the mobile terminal is set to be higher than an execution frequency of the calibration when the cornea reflection image and the pupil center of gravity can be detected for both the first and second images. Computer
For each predetermined period, the first light source and the second light source among a plurality of light sources arranged at different positions are switched and turned on,
For each of the predetermined period, the eye of the first image and the previous SL second light source the user the imaging unit has captured the eye of the user while the first light source is illuminating the eye of the user The imaging unit obtains a second image obtained by photographing the user's eyes while illuminating
At each predetermined period, from one of the first image and the second image, a corneal reflection image by one of the first light source and the second light source, and the pupil center of gravity of the user, Whether or not can be detected,
When the cornea reflection image and the pupil centroid are detected, the user's line-of-sight direction is detected according to the positional relationship between the pupil centroid and the corneal reflection image, while either the corneal reflection image or the pupil centroid Is not detected, the pupil centroid and the corneal reflection image are detected from the other of the first image and the second image, and the user is detected according to the positional relationship between the pupil centroid and the corneal reflection image. Detecting the gaze direction of
A gaze detection method including the above. The first light source is closer to the imaging unit than the second light source,
Determining whether the cornea reflection image and the pupil centroid can be detected determines whether the pupil centroid and the corneal reflection image can be detected from the first image, and the corneal reflection image or the pupil The line-of-sight detection method according to claim 8, further comprising detecting the pupil centroid and the corneal reflection image from the second image when any of the centroids cannot be detected. Detecting the line-of-sight direction includes detecting the line-of-sight direction by referring to a reference table representing a correspondence between a positional relationship between the cornea reflection image and the pupil center of gravity and the line-of-sight direction;
When the computer cannot detect either the cornea reflection image or the pupil center of gravity for at least one of the first image and the second image, the computer executes the calibration execution frequency for updating the reference table, The line-of-sight detection method according to claim 8, further comprising setting the corneal reflection image and the pupil centroid for both the first and second images to be higher than the calibration execution frequency. For each predetermined period, the first light source and the second light source among a plurality of light sources arranged at different positions are switched and turned on,
For each of the predetermined period, the eye of the first image and the previous SL second light source the user the imaging unit has captured the eye of the user while the first light source is illuminating the eye of the user The imaging unit obtains a second image obtained by photographing the user's eyes while illuminating
At each predetermined period, from one of the first image and the second image, a corneal reflection image by one of the first light source and the second light source, and the pupil center of gravity of the user, Whether or not can be detected,
When the cornea reflection image and the pupil centroid are detected, the user's line-of-sight direction is detected according to the positional relationship between the pupil centroid and the corneal reflection image, while either the corneal reflection image or the pupil centroid Is not detected, the pupil centroid and the corneal reflection image are detected from the other image of the first image and the second image, and according to the positional relationship between the pupil centroid and the corneal reflection image. Detecting the gaze direction of the user;
A computer program for eye-gaze detection that causes a computer to execute the above. The first light source is closer to the imaging unit than the second light source,
Determining whether the cornea reflection image and the pupil centroid can be detected determines whether the pupil centroid and the corneal reflection image can be detected from the first image, and the corneal reflection image or the pupil The computer program for eye gaze detection according to claim 11, comprising detecting the pupil center of gravity and the cornea reflection image from the second image when any of the centers of gravity cannot be detected. Detecting the line-of-sight direction includes detecting the line-of-sight direction by referring to a reference table representing a correspondence between a positional relationship between the cornea reflection image and the pupil center of gravity and the line-of-sight direction;
When it is not possible to detect either the cornea reflection image or the pupil center of gravity for at least one of the first image and the second image, the execution frequency of the calibration for updating the reference table is set to the first and the second images. The computer program for eye-gaze detection according to claim 11, further causing the computer to execute a setting higher than the calibration execution frequency when the cornea reflection image and the pupil center of gravity can be detected for both of the two images.