JP6897467B2 - Line-of-sight detection device, line-of-sight detection program, and line-of-sight detection method - Google Patents

Description

The present invention relates to a line-of-sight detection device, a line-of-sight detection program, and a line-of-sight detection method.

The line-of-sight detection technique is a technique for identifying a place where an observer observing an object is looking, and is used in various situations. For example, a personal computer (PC) screen, a product display shelf in a store, or the like can be an object to be observed.

A pointing device based on a line-of-sight input that displays a cursor corresponding to the user's gaze point on a display, and a technique for removing a saccade component from the output of a binocular eye movement measurement sensor are also known (for example, Patent Document 1 and Patent). See Reference 2). A method for evaluating visual fatigue based on fixation tremor is also known (see, for example, Non-Patent Document 1).

International Publication No. 2009/019760 Pamphlet Japanese Unexamined Patent Publication No. 7-308290

Go, Kazuyoshi Sakamoto, "Study on visual fatigue evaluation of VDT work by fixation tremor (flick)", Ergonomics, Vol. 32, No. 2, P. 87-97, 1996

When the stay determination with respect to the line of sight of the user is performed using the technique of Patent Document 1, the correct determination result is not always obtained.

It should be noted that such a problem occurs not only in the line of sight of the user who is looking at the PC screen but also in the case of detecting the line of sight of the observer who is observing another object.

In one aspect, the present invention aims to accurately detect the line of sight of an observer observing an object.

In one idea, the program causes the computer to perform the following processes:
(1) The computer obtains an index showing the variation of the observer's line of sight based on the difference between the line-of-sight data of the observer's left eye observing the object and the line-of-sight data of the observer's right eye.
(2) The computer makes a stop determination with respect to the line of sight of the observer based on the index indicating the variation.
(3) The computer determines the line-of-sight position of the observer based on the determination result of the stop determination.

According to the embodiment, the line of sight of the observer observing the object can be detected with high accuracy.

It is a figure which shows the line-of-sight sensor. It is a figure which shows the line-of-sight data which shows the line-of-sight position on the screen. It is a figure which shows the detection result of the line-of-sight position. It is a figure which shows the change of the line-of-sight position. It is a figure which shows the line-of-sight data generated from the low-rate video. It is a figure which shows the relationship between the number of samples of a line-of-sight position, and the minimum inclusion circle. It is a functional block diagram of the line-of-sight detection device. It is a flowchart of a line-of-sight detection process. It is a functional block diagram which shows the specific example of the line-of-sight detection apparatus. It is a figure which shows the line-of-sight detection by a line-of-sight sensor. It is a figure which shows the congestion. It is a figure which shows the difference in the horizontal direction of the line-of-sight data. It is a figure which shows the difference in the vertical direction of the line-of-sight data. It is a figure which shows the change of the line-of-sight position of both eyes. It is a figure which shows the line-of-sight data corresponding to the change of the line-of-sight position of both eyes. It is a flowchart which shows the specific example of the line-of-sight detection processing. It is a block diagram of an information processing apparatus.

Hereinafter, embodiments will be described in detail with reference to the drawings.
FIG. 1 shows an example of a line-of-sight sensor that detects the line of sight of a user observing a PC screen. Images 111 to 113 are displayed on the screen 102 of the display device 101, and a line-of-sight sensor 103 is provided at the lower part of the screen 102. The line-of-sight sensor 103 includes a Light Emitting Diode (LED) 121 and a camera 122. At this time, the line-of-sight position of the user on the screen 102 may move between the images 111 to 113.

The LED 121 irradiates the face of the user who is looking at the screen 102 with infrared rays, and the camera 122 photographs the face of the user. The PC connected to the line-of-sight sensor 103 generates line-of-sight data indicating the line-of-sight position from the image of the user's eyes taken by the camera 122. The state indicated by the line-of-sight data generated from the image of the eye includes a large line-of-sight movement called a saccade, a stagnation in which the line of sight stays in one place, and a variation in the line-of-sight called fixation tremor.

FIG. 2 shows an example of line-of-sight data indicating the line-of-sight position on the screen 102. Assuming that the horizontal coordinates of the screen 102 are the x-coordinates and the vertical coordinates are the y-coordinates, the line-of-sight data can be represented by the coordinates (x, y) of the user's line-of-sight position that change in time series. The horizontal axis of the graph of FIG. 2 represents the number of frames of the image captured by the camera 122, and the vertical axis represents the x-coordinate of the line-of-sight position.

A saccade occurs in which the line of sight stays around each of the gaze positions 201 to 203 that the user gazes at on the screen 102, and the line of sight suddenly moves between the two gaze positions. Then, while the line of sight is stationary, fixation tremors in which the line of sight position varies are observed. The user recognizes the object while the line of sight is stationary.

For example, the saccade between the gaze position 201 and the gaze position 202 appears as gaze data 211 indicating a rapid increase in the x-coordinate, and the fixation tremor at the gaze position 203 is as gaze data 212 indicating a small increase or decrease in the x-coordinate. appear.

FIG. 3 shows an example of the detection result of the line-of-sight position around the gaze position. When the user is gazing at the gaze position 301, the line-of-sight position detection result 302 tends to vary around the gaze position 301. Possible causes of such variations include fine eyeball shaking, fixation tremor, and detection error due to image processing.

Of these, the amplitude of the variation in the line-of-sight angle due to the fixation tremor is considered to be about 0.8 degrees at the maximum and about 0.3 degrees on the average. For example, Non-Patent Document 1 reports that the detection range of the amplitude of a flick included in fixation tremor is 3.0'to 50.0', and the average amplitude is 17.7'. On the other hand, the line-of-sight angle may vary by several degrees or more depending on the distance between the user and the camera 122 or the resolution of the image.

The eye movement detection device of Patent Document 1 calculates the inclusion circle (minimum inclusion circle) of the minimum area including the most recent predetermined number of line-of-sight positions, and the case where the next detected line-of-sight position is included in the minimum inclusion circle. In addition, it is determined that the user's eyeball is stopped. Thereby, even when the line-of-sight position slightly moves within the minimum inclusion circle or there is a detection error of the line-of-sight angle, it can be determined that the user intentionally gazes at one point.

However, as described above, the line-of-sight data includes not only fixed-eye tremors but also large line-of-sight movements, and the line-of-sight position changes depending on the large line-of-sight movements. Not always possible.

FIG. 4 shows an example of a change in the line-of-sight position on the PC screen. The horizontal axis and the vertical axis represent the x-coordinate and the y-coordinate of the screen, respectively. In actual work using a PC, it is difficult to set an appropriate minimum inclusion circle because the line-of-sight data contains many large line-of-sight movements and the variation during the line-of-sight stay is unclear. If the line-of-sight stop determination is made based on an inappropriate minimum inclusion circle, the accuracy of the determination result is reduced.

Further, in an environment such as an embedded device installed on a product shelf, the frame rate of the image of the camera 122 is relatively low, so the number of samples at the line-of-sight position is small, and the amount of variation can be statistically determined. Have difficulty.

FIG. 5 shows an example of line-of-sight data generated from a low-rate video. The horizontal axis represents time, and the vertical axis represents the x-coordinate of the line-of-sight position. The frame rate of the video is 100 ms / frame, and the line-of-sight data 501 to the line-of-sight data 503 represent the line-of-sight position of the stationary line of sight.

Since the average gaze time of a human is about several hundred ms, in the case of 100 ms / frame, only a few gaze positions are generated for one stop location. In this case, if the amount of variation is determined using about 10 line-of-sight positions, a large movement of the line of sight is mixed in, and the accuracy of the amount of variation is lowered. In the example of FIG. 5, when the variation amount is determined by using all of the line-of-sight data 501 to the line-of-sight data 503, a large movement of the line of sight is mixed.

Therefore, a method is conceivable in which the range 511 of the variation in the line-of-sight position is determined in advance, and when the line-of-sight data is included in the range 511, it is determined that the line-of-sight is stationary. However, since the actual variation varies depending on conditions such as the distance between the user and the camera or the individual difference of the user's eyeball, if the stop determination is made based on the predetermined range 511, the correct determination result is not always obtained. Not always available.

FIG. 6 shows an example of the relationship between the number of samples at the line-of-sight position and the minimum inclusion circle. In the detection result of FIG. 6A, since the number of samples is large, the minimum inclusion circle 611 for the gaze position 601 and the minimum inclusion circle 612 for the gaze position 602 can be clearly distinguished. Therefore, it is possible to accurately determine the retention of the line of sight based on the minimum inclusion circle 611 and the minimum inclusion circle 612.

On the other hand, in the detection result of FIG. 6B, since the number of samples is small, the line-of-sight position 614 when looking at another gaze position is mixed in the minimum inclusion circle 613 with respect to the gaze position 603. Therefore, it is difficult to make a stop determination using the minimum inclusion circle 613, and the accuracy of the stop determination is lowered.

FIG. 7 shows a functional configuration example of the line-of-sight detection device of the embodiment. The line-of-sight detection device 701 of FIG. 7 includes a storage unit 711, a calculation unit 712, and a determination unit 713. The storage unit 711 stores the line-of-sight data of the left eye of the observer observing the object and the line-of-sight data of the right eye of the observer. The calculation unit 712 and the determination unit 713 perform the line-of-sight detection process using the line-of-sight data stored in the storage unit 711.

FIG. 8 is a flowchart showing an example of the line-of-sight detection process performed by the line-of-sight detection device 701 of FIG. 7. First, the calculation unit 712 obtains an index indicating the variation in the line of sight of the observer based on the difference between the line of sight data of the left eye and the line of sight data of the right eye (step 801). Next, the determination unit 713 determines the occupancy of the observer's line of sight based on the index calculated by the calculation unit 712 (step 802), and determines the observer's line of sight position based on the determination result of the occupancy determination. (Step 803).

According to the line-of-sight detection device 701 of FIG. 7, the line-of-sight of the observer observing the object can be detected with high accuracy.

FIG. 9 shows a specific example of the line-of-sight detection device 701 of FIG. The line-of-sight detection device 701 of FIG. 9 includes a storage unit 711, a calculation unit 712, a determination unit 713, a detection unit 911, and an output unit 912. The camera 901 corresponds to, for example, the camera 122 of FIG. 1, captures the face of the observer, and outputs the image to the line-of-sight detection device 701. The detection unit 911 detects the line-of-sight data 921 of the left eye and the right eye of the observer from each of the images at a plurality of times included in the image output by the camera 122, and stores the data in the storage unit 711. The image at each time is sometimes called a frame.

For example, the line-of-sight data 921 indicates the line-of-sight positions of the left and right eyes in a plane facing the observer's face. In the case of the display device 101 of FIG. 1, the screen 102 corresponds to a plane facing the observer's face. Further, when the camera 901 is installed on the product shelf, the virtual plane covering the front surface of the product shelf corresponds to the plane facing the observer's face.

FIG. 10 shows an example of line-of-sight detection by the line-of-sight sensor 103 of FIG. The LED 121 irradiates the face 1001 of the observer looking at the screen 102 with infrared rays 1002, and the camera 122 photographs the face 1001. In this case, the detection unit 911 calculates the line-of-sight position of the observer based on the positional relationship between the coordinates of the pupil captured in the image and the coordinates of the corneal reflex by the corneal reflex method.

When the observer is looking at the line-of-sight sensor 103, the difference between the coordinates of the pupil (PupX, PupY) and the coordinates of the corneal reflex (PrkX, PrkY) is 0, and the observer is looking at a position away from the line-of-sight sensor 103. , The difference between those coordinates occurs. Therefore, in the corneal reflex method, the direction of the line of sight is detected from the difference between the coordinates (PupX, PupY) and the coordinates (PrkX, PrkY). The line-of-sight position of one eye detected from the i-th frame by the corneal reflex method is represented by the coordinates (Xi, Yi) on the screen 102 and can be obtained by the following equation.

Xi = A * (PupX-PrkX) + W / 2 (1)
Yi = H + B * (PupY-PrkY) (2)

W in the formula (1) represents the number of pixels in the horizontal direction of the screen 102, and H in the formula (2) represents the number of pixels in the vertical direction of the screen 102. The coefficient A and the coefficient B are obtained in advance by calibration. The line-of-sight position (Xi, Yi) can also be calculated from the line-of-sight vector.

The calculation unit 712 calculates an index 923 indicating the variation in the line of sight based on the difference between the line-of-sight data 921 of the left eye and the line-of-sight data 921 of the right eye, and stores the index 923 in the storage unit 711. The determination unit 713 calculates the threshold value 924 for the retention determination based on the index 923 and stores it in the storage unit 711. Further, the determination unit 713 calculates the variation 922 of the line-of-sight data at each time from the line-of-sight data 921 of the left eye and the right eye detected at each of the plurality of times within a predetermined period, and stores the variation 922 in the storage unit 711.

Next, the determination unit 713 makes a stop determination for the variation 922, and if the variation 922 at each time within the predetermined period is smaller than the threshold value 924, it determines that the observer's line of sight is stationary. Then, the determination unit 713 determines the line-of-sight position 925 of the observer based on the determination result of the stop determination, and the output unit 912 outputs the line-of-sight position 925.

By calculating the difference between the line-of-sight data 921 of the left eye and the line-of-sight data 921 of the right eye, the saccade included in the line-of-sight data 921 of each eye is canceled, and it becomes possible to extract the variation representing the fixation tremor. When the observer is looking at an object such as the screen 102, both eyes are gazing at the same position, so that the gazing positions of both eyes are considered to be the same except for the difference in variation. At this time, the difference in the line-of-sight data 921 of both eyes in the left-right direction (horizontal direction) is affected by the congestion, but the difference in the vertical direction is not affected by the congestion.

FIG. 11 shows an example of congestion. Congestion refers to the movement of the eyeball with both eyes facing inward at the same time when the observer gazes at the object. When the object moves back and forth, congestion occurs as the focus is adjusted to the object. Although the eyeball moves in the left-right direction due to the congestion, it does not move in the up-down direction, so that the difference in the vertical direction of the line-of-sight data 921 is not affected by the congestion.

FIG. 12 shows an example of the difference in the horizontal direction of the line-of-sight data 921 of both eyes. The horizontal axis represents the number of frames, and the vertical axis represents the x-coordinate of the line-of-sight position. Graph 1201 shows the change of the x-coordinate of the line-of-sight position of the left eye, graph 1202 shows the change of the x-coordinate of the line-of-sight position of the right eye, and graph 1203 shows the difference between the graph 1201 and the graph 1202. From the graph 1203, it can be seen that the difference in the line-of-sight position in the horizontal direction changes greatly depending on the congestion.

FIG. 13 shows an example of the difference in the vertical direction of the line-of-sight data 921 of both eyes. The horizontal axis represents the number of frames, and the vertical axis represents the y-coordinate of the line-of-sight position. Graph 1301 shows the change in the y-coordinate of the line-of-sight position of the left eye, graph 1302 shows the change of the y-coordinate of the line-of-sight position of the right eye, and graph 1303 shows the difference between the graph 1301 and the graph 1302. From Graph 1303, it can be seen that the change in the vertical difference in the line-of-sight position is smaller than the change in the horizontal difference and is not affected by the congestion.

Therefore, the calculation unit 712 calculates the difference between the vertical line-of-sight position indicated by the left-eye line-of-sight data 921 and the vertical line-of-sight position indicated by the right-eye line-of-sight data 921, and calculates the index 923 based on the difference. .. For example, the vertical change ΔLy of the line-of-sight position of the left eye and the vertical change ΔRy of the line-of-sight position of the right eye are expressed by the following equations.

ΔLy = ΔSLy + ΔVLy (3)
ΔRy = ΔSRy + ΔVRy (4)

ΔSLy in the formula (3) represents a large vertical movement due to the saccade of the left eye, and ΔVLy represents a vertical variation in the line-of-sight position of the left eye. ΔSRy in the equation (4) represents a large vertical movement due to the saccade of the right eye, and ΔVRy represents a vertical variation in the line-of-sight position of the right eye. Since the left eye and the right eye are looking at the same height position, ΔSLy and ΔSRy can be regarded as substantially the same value. Therefore, the difference between ΔLy and ΔRy is as follows.

ΔLy−ΔRy = (ΔSLy + ΔVLy) − (ΔSRy + ΔVRy)
= ΔVLy−ΔVRy (5)

From equation (5), the variance of the difference Ly-Ry between the y-coordinate Ly of the line-of-sight position of the left eye and the y-coordinate Ry of the line-of-sight position of the right eye is the sum of the variance due to the variation in Ly and the variance due to the variation in Ry. You can see that it is represented. Further, the variance due to the variation in Ly and the variance due to the variation in Ry can be regarded as substantially the same value. Therefore, half of the dispersion of Ly-Ry can be used as the dispersion due to the variation in Ly or the dispersion due to the variation in Ry. Further, the dispersion of the horizontal variation of the line-of-sight position can be regarded as substantially the same value as the dispersion of the vertical variation of the line-of-sight position.

FIG. 14 shows an example of a change in the line-of-sight positions of both eyes. The line-of-sight position 1401 of the left eye and the line-of-sight position 1402 of the right eye of the observer looking at the screen 102 move on the screen 102 according to the position of the image to be gazed at.

FIG. 15 shows an example of the line-of-sight data 921 corresponding to the change in the line-of-sight position of FIG. The horizontal axis represents the number of frames, and the vertical axis represents the y-coordinate of the line-of-sight position. Graph 1501 shows the change of the y-coordinate of the line-of-sight position of the left eye, graph 1502 shows the change of the y-coordinate of the line-of-sight position of the right eye, and graph 1503 shows the difference between the graph 1501 and the graph 1502.

By collecting logs of the line-of-sight position 1401 and the line-of-sight position 1402 in a certain period and calculating the difference in the y-coordinates of the line-of-sight positions, the amount of variation in the line-of-sight position is always calculated even when the line-of-sight moves frequently. It becomes possible to grasp. As a result, an appropriate threshold value for the stop determination can be determined based on the amount of variation in the line-of-sight position, and the accuracy of the stop determination is improved. As the index 923 indicating the variation in the line of sight, for example, the variance or standard deviation of the difference in the y-coordinates of the line-of-sight positions detected in a certain period can be used.

The difference between the line-of-sight data 921 of the left eye and the line-of-sight data 921 of the right eye does not necessarily have to be the difference in the vertical direction of the line-of-sight position, and may be the difference in the line-of-sight position in a predetermined direction intersecting the horizontal direction. .. For example, if the horizontal straight line connecting the left eyeball and the right eyeball is not parallel to the x-axis on the screen 102, the y-axis on the screen 102 is not perpendicular to the horizontal straight line but at a predetermined angle. Crossing at. In this case, the difference in the y-coordinates of the line-of-sight positions of both eyes represents the difference in the line-of-sight positions in a predetermined direction intersecting the left-right direction.

FIG. 16 is a flowchart showing a specific example of the line-of-sight detection process performed by the line-of-sight detection device 701 of FIG. First, the detection unit 911 detects the line-of-sight data 921 of the left eye and the right eye of the observer from the image output by the camera 122 (step 1601).

For example, from the image of the i-th frame, the x-coordinate LXi of the line-of-sight position of the left eye, the y-coordinate LYi of the line-of-sight position of the left eye, the x-coordinate RXi of the line-of-sight position of the right eye, and the y-coordinate RYi of the line-of-sight position of the right eye are detected. To. LXi, LYi, RXi, and RYi may be the coordinates of the line-of-sight position on the screen 102 of FIG. 1, and are the coordinates of the line-of-sight position on a virtual plane set at a position where an object such as a product exists. It may be.

Next, the detection unit 911 checks whether or not the observer to be monitored has switched to another observer (step 1602). For example, the detection unit 911 can determine that the observer has been switched when a new observer appears after the observer disappears from the image.

When the observer is switched (step 1602, YES), the detection unit 911 resets the detection process by erasing the line-of-sight data 921 stored in the storage unit 711 (step 1603). Then, the detection unit 911 checks whether or not the line-of-sight data 921 of the latest N frames (N is an integer of 2 or more) is stored in the storage unit 711 (step 1604).

When the line-of-sight data 921 of the latest N frames is not accumulated (step 1604, NO), the determination unit 713 uses the default threshold value as the threshold value TH for the retention determination (step 1606).

Next, the determination unit 713 calculates the variation 922 of the line-of-sight data at each time from the line-of-sight data 921 of the left eye and the right eye detected at each of the plurality of times within the predetermined period, and makes a stop determination for the variation 922 ( Step 1607).

For example, when the variance of the line-of-sight position coordinates is used as the variation 922 of the line-of-sight data 921 detected from the i-th frame and the latest M frames (M is an integer of 2 or more) are used as a predetermined period, the determination unit. In 713, Vi representing the variation 922 can be calculated by the following equation.

Vi = ((LXi + RXi) / 2-Avex) ²
+ ((LYi + RYi) / 2-Avey) ² (6)

However, it is assumed that M <N and i ≧ M. For example, when the frame rate is 100 ms / frame, a value of about 5 to 20 may be used as N, and a value of 2 or more and less than N may be used as M.

(LXi + RXi) / 2 in the formula (6) represents the average value of LXi and RXi, and (LYi + RYi) / 2 represents the average value of LYi and RYi. Avex of the formula (7) represents a value obtained by averaging the average values of LXj and RXj over M frames in the range of j = i-M + 1 to i. Further, Avey in the formula (8) represents a value obtained by averaging the average values of LYj and RYj over the same M frames.

In the stop determination, the determination unit 713 compares the M variation Vj in the range of j = i-M + 1 to i with the threshold value TH, and when Vj ≦ TH for all js (step 1607, YES), the line of sight is It is determined that the vehicle is stationary (step 1608). In this case, the determination unit 713 calculates the line-of-sight position 925 from the coordinates of the line-of-sight positions of M frames. As the line-of-sight position 925, statistical values such as the average value and the median value of M coordinates can be used. Then, the line-of-sight detection device 701 updates i by i = i + 1, and repeats the processing after step 1601 for the image of the next frame.

On the other hand, when Vj> TH for one or more js (step 1607, NO), the determination unit 713 determines that the line of sight is not stationary, and the line-of-sight detection device 701 determines that the image of the next frame is in step 1601. The subsequent processing is repeated.

If i <M in step 1607, the determination unit 713 omits the stop determination and repeats the processes after step 1601. If the observer has not been switched in step 1602 (step 1602, NO), the detection unit 911 performs the process of step 1604.

When the line-of-sight data 921 of the latest N frames is accumulated (step 1604, YES), the calculation unit 712 calculates the index 923 using the accumulated line-of-sight data 921, and the determination unit 713 calculates the index 923. The threshold value 924 is calculated from (step 1605).

For example, the calculation unit 712 can use the moving average of the variance of the difference between LYi and RYi in N frames as the index 923. In this case, the calculation unit 712 calculates the div representing the index 923 by the following equation.

The ave in the formula (10) represents the average value of RYi-LYi in N frames in the range of j = i-N + 1 to i, and the div in the formula (9) is the RYi-LYi in the same N frames. Represents variance. In this case, the determination unit 713 can calculate TH representing the threshold value 924 by the following equation using a positive real number K.

TH = K * (div / 2) (11)

Then, the determination unit 713 performs the processing after step 1607 using the threshold value TH of the equation (11). In step 1604, once the line-of-sight data 921 of N frames is accumulated, the threshold value TH calculated in step 1605 is used unless the observer switches.

According to such a line-of-sight detection process, even when the frame rate of the image is low, an appropriate threshold value for the stop determination can be determined based on the difference between the line-of-sight data 921 of both eyes, and the stop determination can be made. The accuracy of is improved.

By the way, in the line-of-sight detection process of FIG. 16, since the moving average of the variance of the difference between LYi and RYi is used as the index 923, the period of the latest N frames is used as the accumulation period for accumulating the line-of-sight data 921. .. When M <N, the period of the most recent M frames used in the calculation of the variation Vi is included in the accumulation period. However, the accumulation period does not necessarily have to include the period of M frames, and may be a period earlier than that.

Further, the determination unit 713 may use the standard deviation as the variation 922 instead of the variance of the coordinates of the line-of-sight position. In this case, the calculation unit 712 calculates the index 923 by using the standard deviation instead of the variance of the difference between LYi and RYi in N frames.

The configuration of the line-of-sight detection device 701 in FIGS. 7 and 9 is only an example, and some components may be omitted or changed depending on the application or conditions of the line-of-sight detection device 701. For example, in the line-of-sight detection device 701 of FIG. 9, when the line-of-sight data 921 is detected by an external device, the detection unit 911 can be omitted.

The flowcharts shown in FIGS. 8 and 16 are merely examples, and some processes may be omitted or changed depending on the configuration or conditions of the line-of-sight detection device 701. For example, in the line-of-sight detection process of FIG. 16, if the observer does not switch, the processes of step 1602 and step 1603 can be omitted.

The line-of-sight sensor 103 of FIG. 1 and the line-of-sight detection shown in FIG. 10 are merely examples, and the configuration and position of the line-of-sight sensor 103 vary depending on the application or conditions of the line-of-sight detection device 701. When the camera 122 uses visible light to photograph the observer, the LED 121 can be omitted.

The line-of-sight data and the line-of-sight position shown in FIGS. 2 to 6 and 12 to 15 are merely examples, and the line-of-sight data and the line-of-sight position change according to the captured image. The congestion shown in FIG. 11 is only an example, and the degree of congestion varies depending on the observer.

Equations (1) to (10) are merely examples, and the line-of-sight detection device 701 may perform the line-of-sight detection process using another calculation formula.

FIG. 17 shows a configuration example of an information processing device (computer) used as the line-of-sight detection device 701 of FIGS. 7 and 9. The information processing device of FIG. 17 includes a Central Processing Unit (CPU) 1701, a memory 1702, an input device 1703, an output device 1704, an auxiliary storage device 1705, a medium drive device 1706, and a network connection device 1707. These components are connected to each other by bus 1708. The camera 901 of FIG. 9 may be connected to the bus 1708.

The memory 1702 is, for example, a semiconductor memory such as a Read Only Memory (ROM), a Random Access Memory (RAM), or a flash memory, and stores a program and data used for processing. The memory 1702 can be used as the storage unit 711 of FIGS. 7 and 9.

The CPU 1701 (processor) operates as the calculation unit 712 and the determination unit 713 of FIGS. 7 and 9 by executing a program using, for example, the memory 1702. The CPU 1701 also operates as the detection unit 911 in FIG. 9 by executing the program using the memory 1702.

The input device 1703 is, for example, a touch panel, a keyboard, a pointing device, or the like, and is used for inputting instructions or information from a user or an operator. The output device 1704 is, for example, a display device, a printer, a speaker, or the like, and is used for making an inquiry to a user or an operator and outputting a processing result. The output device 1704 can be used as the output unit 912 of FIG.

The auxiliary storage device 1705 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 1705 may be a hard disk drive. The information processing device can store programs and data in the auxiliary storage device 1705 and load them into the memory 1702 for use. The auxiliary storage device 1705 can be used as the storage unit 711 of FIGS. 7 and 9.

The medium drive device 1706 drives the portable recording medium 1709 to access the recorded contents. The portable recording medium 1709 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 1709 may be a Compact Disk Read Only Memory (CD-ROM), a Digital Versatile Disk (DVD), or a Universal Serial Bus (USB) memory. The user or operator can store the programs and data in the portable recording medium 1709 and load them into the memory 1702 for use.

In this way, computer-readable recording media that store programs and data used for processing include physical (non-temporary) recording media such as memory 1702, auxiliary storage device 1705, and portable recording medium 1709. A recording medium is included.

The network connection device 1707 is a communication interface circuit that is connected to a communication network such as a Local Area Network (LAN) or the Internet and performs data conversion associated with communication. The information processing device can receive programs and data from an external device via the network connection device 1707, load them into the memory 1702, and use them. The network connection device 1707 can be used as the output unit 912 of FIG.

It is not necessary for the information processing apparatus to include all the components shown in FIG. 17, and some components may be omitted depending on the application or conditions. For example, if the interface with the user or operator is unnecessary, the input device 1703 and the output device 1704 may be omitted. If the information processing device does not use the portable recording medium 1709 or the communication network, the medium driving device 1706 or the network connecting device 1707 may be omitted.

Having described in detail the embodiments of the disclosure and its advantages, those skilled in the art will be able to make various changes, additions and omissions without departing from the scope of the invention as expressly stated in the claims. Let's do it.

The following additional notes will be further disclosed with respect to the embodiments described with reference to FIGS. 1 to 17.
(Appendix 1)
Based on the difference between the line-of-sight data of the left eye of the observer observing the object and the line-of-sight data of the right eye of the observer, an index showing the variation of the line-of-sight of the observer is obtained.
Based on the index, the occupancy determination with respect to the line of sight of the observer is performed.
The line-of-sight position of the observer is determined based on the determination result of the stop determination.
A line-of-sight detection program that allows a computer to perform processing.
(Appendix 2)
The line-of-sight according to Appendix 1, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting the left-right direction in a plane facing the observer's face. Detection program.
(Appendix 3)
The line-of-sight detection program according to Appendix 2, wherein the predetermined direction is a direction perpendicular to the left-right direction.
(Appendix 4)
The computer calculates the threshold of the occupancy determination based on the index, and from the line-of-sight data of the left eye and the right eye detected at each of the plurality of times within a predetermined period, the variation of the line-of-sight data at each of the plurality of times. Is calculated, and when the variation of the line-of-sight data at each of the plurality of times is smaller than the threshold value, it is determined that the line-of-sight of the observer is stationary. The line-of-sight detection program described.
(Appendix 5)
The appendix 4 is characterized in that the index represents the variance or standard deviation of the difference detected at each of a plurality of times within the first period including the predetermined period or the second period prior to the predetermined period. Line-of-sight detection program.
(Appendix 6)
A storage unit that stores the line-of-sight data of the left eye of the observer observing the object and the line-of-sight data of the right eye of the observer.
A calculation unit that obtains an index indicating variation in the line of sight of the observer based on the difference between the line of sight data of the left eye and the line of sight data of the right eye.
A determination unit that makes a stop determination for the observer's line of sight based on the index and determines the line-of-sight position of the observer based on the determination result of the stop determination.
A line-of-sight detection device comprising.
(Appendix 7)
The line of sight according to Appendix 6, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting the left-right direction in a plane facing the observer's face. Detection device.
(Appendix 8)
The line-of-sight detection device according to Appendix 7, wherein the predetermined direction is a direction perpendicular to the left-right direction.
(Appendix 9)
The determination unit calculates the threshold value of the retention determination based on the index, and from the line-of-sight data of the left eye and the right eye detected at each of the plurality of times within a predetermined period, the line-of-sight data at each of the plurality of times Item 1 of Appendix 6 to 8, wherein the variation is calculated, and when the variation of the line-of-sight data at each of the plurality of times is smaller than the threshold value, it is determined that the line-of-sight of the observer is stationary. The line-of-sight detection device according to.
(Appendix 10)
The appendix 9 is characterized in that the index represents the variance or standard deviation of the difference detected at each of a plurality of times within the first period including the predetermined period or the second period prior to the predetermined period. Line-of-sight detector.
(Appendix 11)
The computer
Based on the difference between the line-of-sight data of the left eye of the observer observing the object and the line-of-sight data of the right eye of the observer, an index showing the variation of the line-of-sight of the observer is obtained.
Based on the index, the occupancy determination with respect to the line of sight of the observer is performed.
The line-of-sight position of the observer is determined based on the determination result of the stop determination.
A line-of-sight detection method characterized by this.
(Appendix 12)
The line-of-sight according to Appendix 11, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting the left-right direction in a plane facing the observer's face. Detection method.
(Appendix 13)
The line-of-sight detection method according to Appendix 12, wherein the predetermined direction is a direction perpendicular to the left-right direction.
(Appendix 14)
The computer calculates the threshold of the occupancy determination based on the index, and from the line-of-sight data of the left eye and the right eye detected at each of the plurality of times within a predetermined period, the variation of the line-of-sight data at each of the plurality of times. Is calculated, and when the variation of the line-of-sight data at each of the plurality of times is smaller than the threshold value, it is determined that the line-of-sight of the observer is stationary. The line-of-sight detection method described.
(Appendix 15)
The appendix 14 is characterized in that the index represents the variance or standard deviation of the difference detected at each of a plurality of times within the first period including the predetermined period or the second period prior to the predetermined period. Line-of-sight detection method.

101 Display device 102 Screen 103 Line-of-sight sensor 111-113 Image 121 LED
122, 901 Cameras 201-203, 301, 601-603 Gaze position 211, 212, 501-503, 921 Line-of-sight data 302 Detection result 511 Range 611-613 Minimum inclusion circle 614, 925, 1401, 1402 Line-of-sight position 701 Line-of-sight detection device 711 Storage unit 712 Calculation unit 713 Judgment unit 911 Detection unit 912 Output unit 922 Variation 923 Index 924 Threshold 1001 Face 1002 Infrared 1201-1203, 1301-1303, 1501-1503 Graph 1701 CPU
1702 Memory 1703 Input device 1704 Output device 1705 Auxiliary storage device 1706 Media drive device 1707 Network connection device 1708 Bus 1709 Portable recording medium

Claims (7)

Based on the difference between the line-of-sight data of the left eye of the observer observing the object and the line-of-sight data of the right eye of the observer, an index showing the variation of the line-of-sight of the observer is obtained.
Based on the index, a stop determination is made with respect to the line of sight of the observer,
The line-of-sight position of the observer is determined based on the determination result of the stop determination.
A line-of-sight detection program that allows a computer to perform processing. The difference according to claim 1, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting the left-right direction in a plane facing the observer's face. Line-of-sight detection program. The line-of-sight detection program according to claim 2, wherein the predetermined direction is a direction perpendicular to the left-right direction. The computer calculates the threshold of the occupancy determination based on the index, and from the line-of-sight data of the left eye and the right eye detected at each of the plurality of times within a predetermined period, the variation of the line-of-sight data at each of the plurality of times. Any one of claims 1 to 3, wherein when the variation of the line-of-sight data at each of the plurality of times is smaller than the threshold value, it is determined that the line-of-sight of the observer is stationary. The line-of-sight detection program described in. 4. The index is characterized in that it represents the variance or standard deviation of the difference detected at each of a plurality of times within the first period including the predetermined period or the second period prior to the predetermined period. The line-of-sight detection program described. A storage unit that stores the line-of-sight data of the left eye of the observer observing the object and the line-of-sight data of the right eye of the observer.
A calculation unit that obtains an index indicating variation in the line of sight of the observer based on the difference between the line of sight data of the left eye and the line of sight data of the right eye.
A determination unit that makes a stop determination for the observer's line of sight based on the index and determines the line-of-sight position of the observer based on the determination result of the stop determination.
A line-of-sight detection device comprising. The computer
Based on the difference between the line-of-sight data of the left eye of the observer observing the object and the line-of-sight data of the right eye of the observer, an index showing the variation of the line-of-sight of the observer is obtained.
Based on the index, a stop determination is made with respect to the line of sight of the observer,
The line-of-sight position of the observer is determined based on the determination result of the stop determination.
A line-of-sight detection method characterized by this.

Images