KR102001950B1 - Gaze Tracking Apparatus and Method - Google Patents

Abstract

The eye tracking apparatus according to an embodiment of the present invention includes: acquiring a first image; Generating a second image having a higher resolution than the first image using the acquired first image; Recognizing a user's eye region within the generated second image; And tracking the user's gaze by recognizing the user's pupil within the recognized eye region.

Description

[0001] Gaze Tracking Apparatus and Method [0002]

An embodiment relates to a gaze tracking apparatus and a gaze tracking method thereof.

Recently, it has been proposed to measure the viewer's advertisement effect based on the user's gaze tracking, analyze driver's driving behavior and prevent drowsiness, rearrange the menu through analysis of web page consumption behavior, control game using eye interaction, And placement of shelves through analysis of technical training programs and consumption behavior.

Patent Document 1 relates to a computer vision-based gaze tracking method.

This method constitutes a look-up table for the user ' s face and eye characteristic values. In actual line-of-sight tracing, the gaze direction is calculated by measuring the face direction and the center point of the iris, and then comparing the characteristic value with the measured value. The gaze direction is calculated by combining the measured face direction coordinate system and the iris center coordinate system. The direction of the face is measured by finding a correct angle by wearing a T-stick type reference model. The center of the iris is obtained by finding the center of gravity of the iris color by analyzing the color distribution of the iris, Finds the longest horizontal line in the elliptical iris edge, and determines the center point of the horizontal line as the center point of the iris.

However, in this method, in order to increase the reliability in constructing the look-up table by measuring characteristic values of various faces and eyes in advance, there is a restriction to collect and collect characteristics of a number of characteristics in advance. The eye tracking accuracy may be lowered.

The conventional technique disclosed in Patent Document 2 tracks the user's face by recognizing the face of the user as a feature surface by a plurality of feature points and recognizing the direction of the user's face by translating and rotating the feature surface. As a result, highly reliable and accurate pointing and tracking systems can be provided at a price that is affordable for general use. However, since only the feature points of the face are used and the rotation of the eyeball is not considered, it is not an intuitive eye tracking interface. In order to move the cursor to the desired position, the head should be moved continuously.

Non-patent papers propose a method of estimating the 3D position of eyeballs using a stereo camera device composed of two cameras and three infrared lights, and obtaining a gaze vector through this to obtain eye gaze position on a two-dimensional plane screen.

In order to perform this method, the calibration process between two cameras must precede the stereo camera configuration, and the position between the three lights must be accurately set.

This method has a problem that it is expensive to construct a large number of cameras and lighting devices.

Patent Document 1: KR 10-0311605

Patent Document 2: KR 10-0325365

Non-Patent Documents: S. W. Shih and J. Liu, "A Novel Approach to 3-D Gaze Tracking Using Stereo Cameras," IEEE Transactions on Systems, Man and Cybernetics, Part B, vol. 34, no. 1, pp. 234-245, Feb. 2004

The embodiment provides a gaze tracking apparatus and a tracking method thereof that can track a user's gaze by applying a super resolution technique.

It is to be understood that the technical objectives to be achieved by the embodiments are not limited to the technical matters mentioned above and that other technical subjects not mentioned are apparent to those skilled in the art to which the embodiments proposed from the following description belong, It can be understood.

The eye tracking apparatus according to an embodiment of the present invention includes: acquiring a first image; Generating a second image having a higher resolution than the first image using the acquired first image; Recognizing a user's eye region within the generated second image; And tracking the user's gaze by recognizing the user's pupil within the recognized eye region.

According to another aspect of the present invention, there is provided a gaze tracking apparatus comprising: a first image acquiring unit acquiring a first image; A second image acquiring unit for acquiring a second image having a higher resolution than the first image using the first image acquired through the first image acquiring unit; A gaze tracking unit for recognizing a user's eye region from a second image acquired through the second image acquisition unit and recognizing a pupil included in the recognized eye region; And a controller for tracking the user's gaze using the pupil recognized through the gaze tracking unit.

According to the embodiment, by tracking the user's gaze by applying the super resolution technique, various functions according to gaze tracking can be provided while reducing the cost for constructing the gaze tracking device.

1 is a view for explaining a gaze tracking apparatus according to an embodiment.
2 is a detailed configuration diagram of the image acquisition unit shown in FIG.
3A to 3C are views illustrating a high-resolution image generation concept according to an embodiment.
4 is a view showing a high-resolution image formed by the embodiment.
5 is a view for explaining a gaze tracking method according to an embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which: FIG. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise.

In order to clearly illustrate the present invention in the drawings, thicknesses are enlarged in order to clearly illustrate various layers and regions, and parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification .

1 is a view for explaining a gaze tracking apparatus according to an embodiment.

Referring to FIG. 1, the gaze tracking apparatus includes an image acquisition unit 110, a high-resolution image generation unit 120, a gaze tracking unit 130, and a control unit 140.

The image acquiring unit 110 acquires a plurality of low resolution images.

2, the image acquisition unit 110 includes a main lens 111 for forming an image of an object, a light source 111 for separating light beams passing through the main lens 111 based on the direction of light rays, Lens array 113 and photosensor array 115 that directs the light beam to the light source 115, as shown in FIG.

The microlens array 113 and the photosensor array 115 included in the image acquisition unit 110 may be implemented as an image sensor 112.

In the embodiment, a plurality of low-resolution images can be obtained by using the image acquisition unit 110 as described above.

The operation of the image acquisition unit 110 will be described in detail as follows.

The rays from a single point on the subject 105 in the imaged scene can reach a single convergent point on the focal plane of the microlens array 113. [

The microlens 114 at this convergence point separates these rays of light based on the direction of the light and produces a focused image of the aperture of the main lens 111 on the photosensor of the microlens array.

The photosensor array 115 detects the light incident thereon and produces an output that is processed using one or more of the various components. The output optical data includes, for example, a high-resolution image generation unit 120 using data together with positional information on each photosensor providing data when generating an image of a scene including the subjects 105, 106, and 107 ).

The high-resolution image generation unit 120 is implemented to process the image data and calculate an image of a scene including the subjects 105, 106, and 107.

Refocusing using the detected light or the characteristics of the detected light together with the known direction (calculated using the known position of each photosensor) in which the high-resolution image generating section 120 light reaches the microlens array, Selectively refocusing and / or calibrating the data as it forms an image that can be corrected.

At this point, the microlens 113 is shown as several distinct microlenses, for example, but the array is typically implemented with multiple (e.g., thousands or millions) microlenses.

In addition to the microlens array 113, an optical coding mask or the like may be used as long as the light beams passing through the main lens 111 are separated based on the directions of the light beams. Those skilled in the art will recognize that the main lens 111 and the microlens array 113 may be implemented using various lenses and / or microlens arrays that are currently available or will be developed in the future will be.

The photosensor array 115 generally has several photosensors for each microlens in the microlens array 113. The size, or pitch, of each pixel of the photosensor array 115 is relatively finer than the microlens array 114. In addition, the microlenses in the microlens array 113 and the photosensors in the photosensor array 115 are generally positioned such that light traveling through each microlens to the photosensor array does not overlap with light traveling through the adjacent microlenses .

The main lens 111 has the same performance as moving in the horizontal direction along the optical axis to focus on a subject of interest at the desired depth "d" as illustrated between the main lens 111 and the exemplary imaging subject 105 . Thus, it is possible to perform refocusing at each position of interest based on the obtained optical field data.

By way of example, rays from a single point of the subject 105 are shown for this illustration. These rays can reach a single convergence point in the microlens 114 on the focal plane of the microlens array 113. [ The microlens 114 separates based on the direction of these rays to produce focused image and light field data of the aperture of the main lens 111 on the set of pixels in the pixel array below the microlens.

The optical field data L (u, v, s, t) intersects the main lens 111 at (u, v), and (s, t ) Along the light ray intersecting the plane of the microlens array 113. [ For example, intensities passing through the position (u, v) of each sub-aperture of the main lens 111 in each microlens s, t. Here, the sub-aperture refers to the number of directional resolutions of the main lens 111. For example, if the number of sub-apertures is 196, each microlens array 113 may be composed of 196 pixels.

Each photosensor in the photosensor array 115 may be implemented to provide a value representative of the set of rays directed to the photosensor through the main lens 111 and the microlens array 113. That is, each photosensor generates an output in response to light incident on the photosensor, and the position of each photosensor with respect to the microlens array 113 is used to provide direction information on the incident light.

The high resolution image generating unit 120 may generate the re-focusing image using the optical field data, i.e., L (u, v, s, t). At this time, the high-resolution image generator 120 can determine the direction of light on each photo sensor using the position of each photo sensor with respect to the microlenses. Also, the high-resolution image generation unit 120 determines the depth of field of the subject in the scene in which the detected light is emitted, and uses the depth of field and the direction of the detected light to generate a composite image focused on a focal plane different from the focal plane Can be calculated.

On the other hand, the image formed below the specific microlens 114 in the microlens array 113 indicates the directional resoultion of the system relative to its position on the imaging plane. The main lens 111 is effectively at an optical infinity of the microlens and the photo sensor array 115 can be positioned in one plane at the focal depth of the microlens in order to focus the microlenses.

The separation distance "s" between the main lens 111 and the microlens array 113 can be selected to achieve a sharp image within the field of view of the microlens.

The aperture size of the main lens 111 and the aperture size of the microlenses in the microlens array 113 (e.g., the effective size of the aperture in the lens) may be selected to match the particular application in which the eye tracking device is implemented. This method is facilitated by fitting to the f-number (focal ratio: the effective focal length of the lens to the aperture diameter) of the main lens and the microlens.

3A to 3C are views illustrating a high-resolution image generation concept according to an embodiment.

Captures the light field data as shown in FIG. 3A. As described above, when an image is acquired using optical field data, a plurality of low-resolution images or sub-sampled images having reduced spatial resolution as compared with a general image pickup apparatus using the same photo sensor array are obtained. According to an exemplary embodiment, one of the plurality of low-resolution images may be determined as a reference image frame, and the sub-sampled low resolution image frames other than the reference image frame may be selected as at least one reference image frame.

3B shows the result of registering the optical field data on the HR grid.

3B is a diagram illustrating registration of optical field data constituting a reference image frame and a reference image frame to be selected according to an embodiment on a high resolution grid. In Fig. 3B, & cir &, DELTA, and * represent optical field data that passes through the position of another main lens as optical field data for one point of the object.

According to one embodiment, if the image frame formed using the light field data indicated by a circle including points 311 and 312 is a reference image frame, it may be registered on the high resolution grid as shown in FIG. 3B.

The high-resolution image frame is reconstructed as shown in FIG. 3C. According to one embodiment, the resolution of the reference image frame can be increased by interpolating the values between the optical field data constituting the reference image frame using the optical field data for one point of the object. For example, the value 313 on the HR grid between the light field data 311 and the light field data 312 can be interpolated using the light field data belonging to the region 310 on the HR grid of Figure 3b.

4, the image acquisition unit 110 acquires a plurality of images 410 having a low resolution, and the high-resolution image generation unit 120 acquires a plurality of images 410 using the obtained plurality of images 410 And generates a high-resolution image 420.

The ultimate goal of the above principle is to detect the eye region of the user in the low-resolution image, and thereby to generate the high-resolution image to enlarge the detected eye region.

Accordingly, the high-resolution image generating unit 120 detects a user's eye region from the low-resolution image, and generates a high-resolution image obtained by enlarging the detected eye region.

The gaze tracking unit 130 detects the user's eye area within the high-resolution image generated through the high-resolution image generation unit 120. [

In particular, the gaze tracking unit 130 acquires the pupil of the user by driving an image processing algorithm and a circular detection algorithm on the high-resolution image. That is, the gaze tracking unit 130 detects the user pupil region from the high-resolution image generated through the high-resolution image generating unit 120, and drives the circular detection algorithm to acquire the center point of the pupil from the detected pupil region.

The circular detection algorithm is a method of detecting a portion having the largest gray level sum of an inner circle and an outer circle of a template by moving a circular detection template composed of an inner circle and an outer circle to a pupil region and detecting a center point of the detected region Can be obtained as a center point.

In addition, the gaze tracking unit 130 may further perform the local binarization technique in addition to the circular detection algorithm to obtain the pupil center point.

In this manner, the gaze tracking unit 130 performs local binarization in addition to the circular detection algorithm to accurately obtain the center point of the pupil, determines the dark part of the binarized area as a pupil area, It can be judged as a practical center point of the user's pupil.

Alternatively, the gaze tracking unit 130 acquires an image of the eye region from the high-resolution image. Thereafter, the gaze tracking unit 130 detects the center of the pupil and the center of the reflex point using image processing techniques such as threshold binarization, open computation, closure computation, median filtering, labeling, and center of gravity I ask.

The control unit 140 tracks the user's gaze based on the distance between the center point of the left pupil and the center point of the right pupil obtained through the gaze tracking unit 130. [

In addition, the controller 140 tracks which portion of the screen the user is viewing by using the distance between the center of the pupil and the two reflection points.

If the user's gaze is tracked, the control unit 140 displays the gaze effect based on the tracked gaze, such as viewer advertisement effect measurement, driver driving behavior analysis and drowsiness prevention, menu rearrangement through analysis of web page consumption behavior, Control, iris information based user authentication, exercise and expertise training programs, and disposition of shelves through consumption behavior analysis.

In the embodiment described above, the user's gaze is tracked through acquisition of a high-resolution image.

According to the embodiment, by tracking the user's gaze by applying the super resolution technique, various functions according to gaze tracking can be provided while reducing the cost for constructing the gaze tracking device.

5 is a view for explaining a gaze tracking method according to an embodiment.

Referring to FIG. 5, in the gaze tracking method, a plurality of low-resolution images are preferentially acquired (Step 110).

When the plurality of low resolution images are acquired, a high resolution image is generated based on the plurality of low resolution images using an image interpolation method (operation 120).

Then, an eye region in which the user's eyeball is located is identified from the generated high-resolution image in operation 130.

When the eyeball region is identified, the left pupil and the right pupil are recognized in the confirmed eyeball region (Step 140).

Thereafter, using the distance relationship between the left pupil and the right pupil, a line of sight of which part of the current screen the user is viewing is tracked (operation 150).

On the other hand, in the above, the eyes of a plurality of users can be tracked.

Accordingly, when a plurality of users are located in front of the gaze tracking device, the control unit 140 performs gaze tracking for each of the plurality of positioned users.

In addition, when the line-of-sight tracking is performed, the line-of-sight information tracked for each of the plurality of users is displayed, and information on which part of the current screen is viewed by each of the plurality of users is provided.

For this purpose, the control unit 140 acquires eye position information for a plurality of users from the high-resolution image, recognizes pupils corresponding to the respective eye positions based on the obtained eye position information, do.

Meanwhile, in order to provide various functions according to the line-of-sight tracking, the gaze of one user must be tracked. At this time, if the eyes of the plurality of users are tracked, the various functions as described above can be effectively provided.

On the other hand, in general, the viewer who is watching the display device is located at the closest distance from the display device.

Also, the pupil of the viewer located at the closest distance occupies the largest portion in the high-resolution image. This is similar to the principle that images located close to each other are largely displayed on the screen.

Accordingly, the control unit 140 acquires only the eye position of the user located at the closest distance from the high-resolution image, and performs gaze tracking using the obtained eye position.

According to the embodiment, by tracking the user's gaze by applying the super resolution technique, various functions according to gaze tracking can be provided while reducing the cost for constructing the gaze tracking device.

The foregoing description is merely illustrative of the technical idea of the present invention and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

110:
120: a high-resolution image generating unit
130: eye tracking unit
140:
111: main lens
112: Image sensor
113: micro lens array
115: Photo sensor array

Claims (19)

Obtaining a plurality of first image frames having different focal distances from each other using a photosensor array including a plurality of photosensors for acquiring optical information;
Acquiring a direction in which the optical information is incident using the photosensor position information on the photo sensor array;
Generating a second image frame having a higher resolution than the first image frame using the optical information of the plurality of first image frames and the direction in which the optical information of the plurality of first image frames are incident;
Recognizing an eye region within the generated second image frame;
Recognizing a pupil region in the eye region;
Recognizing the center of the pupil in the pupil region; And
And tracking the line of sight of the subject using the center information of the pupil,
Wherein the plurality of first image frames are images photographed differently in focus with respect to the same subject by moving a lens located in a single image acquisition unit in the direction of an optical axis,
The step of recognizing the pupil region performs a binarization technique,
The step of recognizing the center of the pupil may include performing a circular detection algorithm and the binarization technique on the pupil region to obtain a pupil center position,
Wherein the circular detection algorithm detects a portion having a greatest gray level sum difference between the inner circle and the outer circle of the template by moving a circular detection template composed of an inner circle and an outer circle to the pupil region in the eye region, Acquiring a center point of an area as a center point of the pupil,
Wherein the binarization technique divides a shadow region into a dark region and a bright region based on a threshold value in the eye region and determines a center of gravity of the dark region as a substantial center point of the pupil. The method according to claim 1,
Wherein a direction in which the light is incident is a direction of light incident on the photosensor. delete delete The method according to claim 1,
Wherein the generating the second video frame comprises:
Wherein the second image frame is generated by applying an image interpolation algorithm to the plurality of first image frames. The method according to claim 1,
Wherein the generating the second video frame comprises:
And generating an enlarged image obtained by enlarging the eye region of the subject included in the plurality of first image frames. The method according to claim 1,
The step of recognizing the eye region
And generating an enlarged image of the eyeball region for each of the plurality of subjects when a plurality of the objects exist in the second image frame. 8. The method of claim 7,
Wherein the step of tracking the subject's eye-
And recognizing the pupil in the eye region for each of the plurality of subjects to track the gaze for each of the plurality of subjects. 9. The method of claim 8,
And displaying information on the gaze tracked for each of the plurality of subjects. A main lens on which light is incident;
A microlens array for separating the direction of light incident on the main lens;
A photosensor array including a plurality of photosensors, the photosensor array acquiring the light information and the photosensor position information;
A single image acquiring unit which is composed of the main lens, the microlens array, and the photo sensor array and moves the main lens in the optical axis direction to acquire a plurality of first image frames having different focal lengths;
A high-resolution image generation unit for acquiring a direction in which the light is incident using the position information of the photosensor and generating a second image frame using the light incidence direction and the light information;
A gaze tracking unit for recognizing an eye region from the second image frame generated through the high resolution image generating unit and acquiring a center point of the pupil included in the eye region; And
And a control unit for tracking the gaze of the subject using the information of the pupil recognized through the gaze tracking unit,
Wherein the second video frame has a higher resolution than the first video frame,
Wherein the gaze tracking unit acquires a center position of the pupil by performing a circular detection algorithm and a binarization technique on the eye region,
Wherein the circular detection algorithm detects a portion having a greatest gray level sum difference between the inner circle and the outer circle of the template by moving a circular detection template composed of an inner circle and an outer circle to the pupil region in the eye region, Acquiring a center point of an area as a center point of the pupil,
The binarization technique divides the eye region into a dark region and a bright region based on a threshold value, determines the dark region as the pupil region, and determines a center of gravity of the dark region as a substantial center point of the pupil Device. 11. The method of claim 10,
Wherein the direction in which the light is incident is a direction of light incident on the photosensor. 11. The method of claim 10,
Wherein the microlens array is composed of a plurality of microlenses,
Wherein the number of sub-apertures through which the light passes through the main lens is equal to the number of pixels of the photo sensor corresponding to the microlens array. delete 11. The method of claim 10,
Wherein the high-
Wherein the image interpolation algorithm is applied to the plurality of first image frames. 11. The method of claim 10,
Wherein the high-
Wherein the eyeball region is detected in the plurality of first image frames and an enlarged image in which the eyeball region recognized in the plurality of first image frames is enlarged in the second image frame is generated. 11. The method of claim 10,
Wherein,
An image enlarging unit that enlarges the eyeball region for each of the plurality of subjects when the plurality of subjects are included, and recognizes the pupil and tracks the gaze for each of the plurality of subjects. 17. The method of claim 16,
Wherein,
Eye line information for each of the plurality of tracked subjects. delete delete

Images