KR101619656B1 - Gaze Tracking Apparatus and Method for Detecting Pupil thereof - Google Patents

Abstract

The present invention relates to a line-of-sight tracking device and a pupil detecting method thereof which can predict information on a pupil within a current acquired eye image based on pupil information detected from a previous eye image when detecting a pupil for tracking a line-of-sight, and detect a pupil based on the predicted pupil information, such that pupil detection accuracy can be improved. An eye image is inputted by a camera. A pupil state (a position and a size) within the eye image is predicted. Then, pupil candidates are selected from the eye image, and a pupil candidate which is most similar to the pupil state is detected among the pupil candidates as a pupil.

Description

Technical Field [0001] The present invention relates to a gaze tracking apparatus and a pupil detection method thereof,

The present invention relates to a line-of-sight tracking apparatus, and more particularly to a line-of-sight tracking apparatus for predicting information on a pupil in a currently acquired eye image based on pupil information detected in a previous eye image, The present invention relates to a line-of-sight tracking apparatus and a pupil detection method thereof, which can improve pupil detection and accuracy.

In general, the gaze tracker captures facial images and eye images, detects pupil in the image, detects illumination reflections in the pupil, and calculates gaze vectors using the detected reflections.

Such a conventional gaze tracker detects a pupil and detects an illumination reflection point, and applies a Kalman filter or the like to smoothen and correct the gaze vector.

However, when a filter such as a Kalman filter is used as it is in the prior art, there is a problem in that accuracy is poor because the gaze movement is not linear.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the problems of the conventional art described above, and it is an object of the present invention to provide a method and apparatus for predicting information on a pupil in a currently acquired eye image based on pupil information detected in a previous eye image, A gaze tracking apparatus and pupil detection method capable of detecting pupil based on pupil information and improving accuracy.

According to an aspect of the present invention, there is provided a method of detecting a pupil of a line-of-sight tracing apparatus, including the steps of receiving an eye image through a camera, predicting a pupil state in the eye image, Selecting pupil candidates from the pupil candidates, and detecting the pupil candidates most similar to the pupil candidates out of the pupil candidates as pupils.

The pupil state predicting step predicts the pupil state using a recursive filter such as a Kalman filter.

Further, the pupil state includes a pupil position and a size.

Further, the recursive filter corrects the pupil state in the filter by receiving the detected state information of the pupil.

The pupil candidate selection step may include generating a binary image by binarizing the eye image, extracting an outline on the binary image, and designating an outline of a predetermined size or more out of the extracted outline as a pupil candidate .

The pupil detection step may include calculating a distance between the pupil position predicted in the eye image and the pupil candidate position and detecting pupil candidates having the smallest distance as the pupil .

According to another aspect of the present invention, there is provided a gaze tracking apparatus including a camera for capturing an image of a user, an illumination unit for emitting light to the user, a pupil state estimation unit for estimating a pupil state in the eye image detected from the user's image, And a control unit for detecting the pupil based on the pupil state.

The control unit may include an eye detection unit for detecting the eye image from the user's image, a pupil prediction unit for predicting the pupil state including the position and size of the pupil in the eye image, A pupil detection unit which selects a pupil candidate from the image and detects a pupil candidate having the highest degree of similarity with the pupil candidate among the pupil candidates as a pupil; an illumination reflection point detection unit that detects an illumination reflection point by the illumination unit in the pupil; And a line-of-sight calculating unit for calculating a line-of-sight vector using the illumination reflection point.

The pupil prediction unit may be implemented as a Kalman filter.

Further, the pupil detecting unit feeds back the detected position and size of the pupil to the pupil predicting unit.

The present invention predicts information on a pupil in a currently acquired eye image based on pupil information detected in a previous eye image in pupil detection, and improves accuracy by detecting pupil based on the predicted pupil information.

1 is a block diagram showing a gaze tracking apparatus according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a pupil detection method of a gaze tracking apparatus according to an embodiment of the present invention. FIG.

The terms "comprises", "comprising", "having", and the like are used herein to mean that a component can be implanted unless otherwise specifically stated, Quot; element ".

Also, the terms " part, "" module, " and" module ", as used herein, refer to a unit that processes at least one function or operation and may be implemented as hardware or software or a combination of hardware and software . It is also to be understood that the articles "a", "an", "an" and "the" Can be used.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a gaze tracking apparatus according to an embodiment of the present invention.

1, the eye-gaze tracking apparatus 100 includes a camera 110, an illumination unit 120, and a control unit 130. [

The camera 110 is installed facing the user and acquires the image of the user. The camera 110 is composed of one or more image sensors (not shown). Here, the image sensor may be a charge coupled device (CCD) image sensor, a complementary metal oxide semi-conductor (CMOS) image sensor, a charge priming device (CPD) image sensor, a charge injection device (CID) And may be implemented as any one of image sensors such as a sensor.

For example, the camera 110 may be a stereo camera composed of two infrared image sensors and an infrared filter.

The illumination unit 120 is disposed side by side with the camera 110 to irradiate light toward the user. The illumination unit 120 may be composed of one or more infrared light emitting devices (IR LEDs).

The control unit 130 detects the eye image from the user's image acquired through the camera 110, detects the pupil and illumination reflection points in the eye image, and tracks the eyes using pupil and illumination reflection points. The controller 130 may be implemented as an image processor.

The control unit 130 includes an eye detection unit 131, a pupil detection unit 133, a pupil prediction unit 135, an illumination reflection point detection unit 137, and a gaze calculation unit 139.

The eye detecting unit 131 extracts a face image from a user's image input from the camera 110. [ Then, the eye detecting unit 131 detects the eye image from the face image. At this time, the eye detecting unit 131 may use a known face detection algorithm and an eye detection algorithm.

The pupil detection unit 133 converts the eye image into a binary image through thresholding. The pupil detection unit 133 detects one or more contours on the binary image. The pupil detection unit 133 selects a contour similar to the pupil shape among the detected contours as pupil candidates. For example, the pupil detection unit 133 detects a contour line having a horizontal / vertical ratio of the contour line of 0.9 or more and 1.1 or less as a pupil candidate. Alternatively, the pupil detecting section 133 detects a circular contour line having a diameter of 2 mm or more and 6 mm or less as pupil candidates.

The pupil detection unit 133 finally detects any one of the pupil candidates as the pupil using the state information of the pupil in the eye image predicted by the pupil predictor 135. [ Here, the state information includes the position (coordinate) and size (e.g., radius) of the pupil.

For example, the pupil detection unit 133 determines the pupil closest to the contour located closest to the position of the pupil predicted by the pupil predictor 135.

The pupil prediction unit 135 predicts the state information of the pupil in the eye image at the current time using the pupil state information (position and size) previously detected by the pupil detection unit 133. [ For example, the pupil prediction unit 135 predicts the position of the pupil in the current eye image using the pupil position information (pupil center point coordinate) detected in the previous eye image.

The pupil prediction unit 135 is implemented with a recursive filter such as a Kalman filter. The Kalman filter recursively operates on the predictive and corrective steps. The prediction step generates an estimated value of the current time point based on the data measured at the previous time point. The measurement update step is a step of receiving the measured value at the current point and correcting the measured value through the previous values, thereby removing the noise.

Thus, the gaze tracking apparatus predicts the pupil position through the filter value at the time of pupil detection, and weights the pupil position instead of using the filter value as a post-processing.

The pupil prediction unit 135 predicts the pupil state information in the eye image (current frame) for a predetermined frame (for example, 10 frames) in the initial operation, and corrects the state information of the pupil detected from the pupil detection unit 133.

The illumination reflection point detection unit 137 detects an illumination reflection point (glint) generated by the detected intra-pupil illumination unit 120. The pupil detection method can be used for detection of the illumination reflection point.

The gaze calculation unit 139 calculates the gaze vector using the coordinates of the center point of the pupil and the illumination reflection point detected by the pupil detection unit 133 and the illumination reflection point detection unit 137 and the camera information. For example, the direction perpendicular to the straight line connecting the center point coordinates of the pupil and the illumination reflection point is calculated as a line of sight.

2 is a flowchart illustrating a pupil detection method of a gaze tracking apparatus according to an embodiment of the present invention.

First, the gaze tracking device 100 acquires a snow image through the camera 110 (S101). Upon receiving the user's image from the camera 110, the control unit 130 detects the face region and detects the eye region in the detected face region.

The control unit 130 predicts the pupil state in the eye image (S103). In other words, the pupil prediction unit 135 of the control unit 130 generates state information including the position (pupil center coordinate) and size (e.g., radius, diameter, etc.) of the pupil in the eye image. The pupil prediction unit 135 transmits the predicted pupil state information to the pupil detection unit 133. [

The control unit 130 converts the eye image into a binary image through binarization (S105).

The control unit 130 extracts contour lines on the binary image using the contour detection technique (S107). At this time, the control unit 130 calculates the position and size of the extracted contour line.

The control unit 130 selects pupil candidates among the detected contour lines (S109). For example, the control unit 130 selects a contour line having a diameter of 2 mm or more and 6 mm or less among the detected contour lines as pupil candidates.

The control unit 130 calculates the degree of similarity between the predicted pupil state and the pupil candidate (S111). For example, the pupil detection unit 133 calculates the distance between the predicted position of the pupil and the position of the pupil candidate.

The control unit 130 detects pupil candidates having the highest similarity as a pupil (S113). The pupil detection unit 133 detects pupil candidates that are closest to the predicted position of the pupil and the pupil candidate as the pupil.

The control unit 130 feeds back the detected pupil state information to the pupil prediction unit 135 (S115). The pupil detection unit 133 transmits state information including the position and size of the detected pupil to the pupil prediction unit 135.

Then, the control unit 130 detects the detected intra-pupillary illumination reflex points and computes the line-of-sight vector using the coordinates of the center point of the detected pupil and the illumination reflex points.

The embodiments described above are those in which the elements and features of the present invention are combined in a predetermined form. Each component or feature shall be considered optional unless otherwise expressly stated. Each component or feature may be implemented in a form that is not combined with other components or features. It is also possible to construct embodiments of the present invention by combining some of the elements and / or features. The order of the operations described in the embodiments of the present invention may be changed. Some configurations or features of certain embodiments may be included in other embodiments, or may be replaced with corresponding configurations or features of other embodiments. It is clear that the claims that are not expressly cited in the claims may be combined to form an embodiment or be included in a new claim by an amendment after the application.

Embodiments in accordance with the present invention may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, an embodiment of the present invention may include one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) field programmable gate arrays, processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, a procedure, a function, or the like which performs the functions or operations described above. The software code can be stored in a memory unit and driven by the processor. The memory unit is located inside or outside the processor, and can exchange data with the processor by various known means.

It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit of the invention. Accordingly, the foregoing detailed description is to be considered in all respects illustrative and not restrictive. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

100: eye tracking device
110: camera
120:
130:
131: eye detection unit
133: Pupil detection unit
135: Pupil predictor
137: Illumination reflection point detector
139: line of sight calculation unit

Claims (10)

Receiving an eye image through a camera;
Estimating a pupil state including a pupil position and a size in the eye image;
Selecting pupil candidates from the eye image,
Detecting a pupil candidate most similar to the pupil state among the pupil candidates as a pupil,
The pupil detection step may include:
Calculating a distance between a pupil position predicted in the eye image and a pupil candidate position;
And detecting a pupil candidate having the minimum distance as a pupil. The method according to claim 1,
Wherein the pupil state predicting step comprises:
Wherein the pupil state is predicted using a recursive filter such as a Kalman filter. delete 3. The method of claim 2,
Wherein the recursive filter comprises:
And the state of the detected pupil is fed back to correct the pupil state in the filter. The method according to claim 1,
Wherein the pupil candidate selection step comprises:
Generating a binary image by binarizing the eye image;
Extracting a contour on the binary image;
And designating a contour of a predetermined size or more out of the extracted contours as pupil candidates. delete A camera for acquiring a user's image,
A lighting unit for emitting light to the user;
And a controller for predicting a pupil state including a position and a size of a pupil in the eye image detected from the image of the user and detecting the pupil based on the predicted pupil state,
Wherein the control unit selects a pupil candidate from the eye image through contour detection and detects a pupil candidate having a minimum distance from the predicted pupil position of the pupil candidate as a pupil. 8. The method of claim 7,
Wherein,
An eye detector for detecting the eye image from the user's image;
A pupil prediction unit for predicting the pupil state in the eye image,
A pupil detection unit for detecting pupil from the eye image,
An illumination reflection point detection unit for detecting an illumination reflection point by the illumination unit in the pupil,
And a line-of-sight calculating unit for calculating a line-of-sight vector using the pupil and the illumination reflection point. 9. The method of claim 8,
Wherein the pupil prediction unit comprises:
And a Kalman filter. 9. The method of claim 8,
Wherein the pupil detection unit comprises:
And the position information and the size of the detected pupil are fed back to the pupil predicting unit.

Images