CN112528713B - Gaze point estimation method, gaze point estimation system, gaze point estimation processor and gaze point estimation equipment - Google Patents

Abstract

The present invention discloses a gaze point estimation method and system, which is applied to a sight tracking device provided with a single camera and a single coaxial light source, and includes: obtaining an original image captured from a single camera; collecting human eye feature information of the original image, and calculating pupil spot center data of both eyes based on the human eye feature information; calculating and obtaining an initial PCR vector based on the pupil spot center data, the PCR vector representing a vector from the spot center to the pupil center; normalizing the initial PCR vector using a preset distance factor to obtain a target PCR vector; and calculating and obtaining gaze point information based on the target PCR vector. When performing gaze point estimation, only the pupil spot center data is used, which solves the problem that two sets of light sources are required to perform gaze point estimation, thereby reducing the number of light sources of the existing sight tracking device, and realizing miniaturization and lightweighting of the sight tracking device.

Description

A method, system, processor and device for estimating gaze point

Technical Field

The present invention relates to the field of information processing technology, and in particular to a gaze point estimation method, system, processor and device.

Background Art

With the development of human-computer interaction technology, eye tracking technology has been widely used. Eye tracking, also known as gaze tracking, is a technology that estimates the eye's gaze and/or gaze point by measuring eye movement.

Existing gaze tracking technologies are generally based on a multi-light source single camera or a multi-light source multi-camera setting. There are generally two types of light sources: one is that the light source is separated from the camera position to form a normal pupil face image (also called a dark pupil), which is called a dark pupil light source; the other is that the iris reflected light on the same axis as the camera causes the camera pupil to shine (also called a bright pupil), which is called a bright pupil light source. In order for the existing technology to achieve the purpose of gaze tracking, the light source combination is generally a combination of multiple dark pupils, or multiple dark pupils combined with one bright pupil, or one dark pupil combined with one bright pupil. The distance between the two light sources needs to be set relatively large, which makes the gaze tracking device produced by the existing gaze tracking method larger in size and cannot meet the miniaturization and lightweight purposes required by users.

Summary of the invention

In view of the above problems, the present invention provides a gaze point estimation method and system. Based on the gaze point estimation of a single light source, the number of light sources of the existing gaze tracking device can be reduced, thereby achieving the miniaturization and lightweight of the gaze tracking device.

In order to achieve the above object, the present invention provides the following technical solutions:

A gaze point estimation method is applied to a gaze tracking device provided with a single camera and a single coaxial light source, comprising:

Acquire an original image captured from the single camera;

Collecting human eye feature information of the original image, and calculating pupil spot center data of both eyes based on the human eye feature information;

Based on the pupil spot center data, an initial PCR vector is calculated and obtained, where the PCR vector represents a vector pointing from the spot center to the pupil center;

The initial PCR vector is normalized using a preset distance factor to obtain a target PCR vector, wherein the preset distance factor represents a normalization parameter for normalizing the initial PCR vector;

According to the target PCR vector, the fixation point information is calculated and obtained.

Optionally, the pupil spot center data of the two eyes include pupil image coordinates and spot image coordinates, and the collecting of human eye feature information of the original image and calculating the pupil spot center data of the two eyes based on the human eye feature information include:

Collecting human eye feature information of the original image;

According to the human eye feature information, pupil image features and spot image features of both eyes are acquired;

Calculating and obtaining pupil image coordinates according to the pupil image features;

The light spot image coordinates are calculated based on the light spot image features.

Optionally, the preset distance factor represents a function of a distance parameter, wherein the distance parameter includes a distance between pupils of two eyes, a distance between light spots of two eyes, or a distance between designated feature points of two eyes.

Optionally, the acquiring the original image captured from the single camera includes acquiring the original image captured from the single camera according to a set exposure gain, and the method further includes:

According to the original image, calculating the average gray value of the pupil area;

According to the average gray value, it is determined whether to adjust the set exposure gain so that the acquired original image meets the light spot search condition.

Optionally, judging whether to adjust the set exposure gain according to the average gray value includes:

It is determined whether the average grayscale value exceeds a preset grayscale threshold, and if so, the set exposure gain is adjusted.

Optionally, the method further comprises:

Adjusting the set exposure gain to obtain a target exposure gain;

The single camera is controlled to perform image acquisition according to the target exposure gain, so that the acquired original image meets the target exposure gain.

Optionally, calculating and obtaining the fixation point information according to the target PCR vector includes:

The gaze point information is calculated based on a preset mapping relationship and the target PCR vector, wherein the preset mapping relationship represents a mapping relationship between the PCR vector and the gaze point and/or gaze direction.

A gaze point estimation system is applied to a gaze tracking device provided with a single camera and a single coaxial light source, comprising:

an acquisition unit, configured to acquire an original image captured from the single camera;

A first calculation unit is used to collect human eye feature information of the original image, and calculate pupil spot center data of both eyes based on the human eye feature information;

A second calculation unit is used to calculate and obtain an initial PCR vector based on the pupil spot center data, where the PCR vector represents a vector pointing from the spot center to the pupil center;

A normalization unit, configured to perform normalization processing on the initial PCR vector using a preset distance factor to obtain a target PCR vector, wherein the preset distance factor represents a normalization parameter for normalizing the initial PCR vector;

The third calculation unit is used to calculate and obtain the fixation point information according to the target PCR vector.

Optionally, the first computing unit includes:

A collection subunit, used for collecting human eye feature information of the original image;

An acquisition subunit, used to acquire pupil image features and spot image features of two eyes according to the human eye feature information;

A first calculation subunit, configured to calculate and obtain pupil image coordinates according to the pupil image features;

The second calculation subunit is used to calculate and obtain the spot image coordinates according to the spot image features; the pupil spot center data of the two eyes includes pupil image coordinates and spot image coordinates.

Optionally, the acquisition unit is specifically used to acquire the original image captured from the single camera according to the set exposure gain, and the system further includes:

A gray value calculation unit, used for calculating an average gray value of a pupil area according to the original image;

A judging unit, used for judging whether to adjust the set exposure gain according to the average gray value, so that the acquired original image meets the light spot search condition;

Wherein, the judging unit is specifically used for:

Determining whether the average grayscale value exceeds a preset grayscale threshold, and if so, adjusting the set exposure gain;

The system also includes:

An adjusting unit, used for adjusting the set exposure gain to obtain a target exposure gain;

The re-capturing unit is used to control the single camera to perform image capture according to the target exposure gain, so that the acquired original image meets the target exposure gain.

The third computing unit is specifically used for:

The gaze point information is calculated based on a preset mapping relationship and the target PCR vector, wherein the preset mapping relationship represents a mapping relationship between the PCR vector and the gaze point and/or gaze direction.

A processor is used to run a program, wherein the program executes the gaze point estimation method as described above when running.

A device, comprising a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, at least:

Acquire an original image captured from the single camera;

Collecting human eye feature information of the original image, and calculating pupil spot center data of both eyes based on the human eye feature information;

Based on the pupil spot center data, an initial PCR vector is calculated and obtained, where the PCR vector represents a vector pointing from the spot center to the pupil center;

The initial PCR vector is normalized using a preset distance factor to obtain a target PCR vector, wherein the preset distance factor represents a normalization parameter for normalizing the initial PCR vector;

According to the target PCR vector, the fixation point information is calculated and obtained.

Compared with the prior art, the present invention provides a gaze point estimation method, system, processor and device. In the gaze point estimation process, pupil spot center data is obtained, and a target PCR vector is obtained after calculation and normalization based on the pupil spot center data, and gaze point information is obtained by calculation of the target PCR vector. The gaze point estimation method only uses pupil spot center data when performing gaze point estimation, that is, only one spot and pupil position information can be used to estimate the distance information between the human eye and the camera, without using two spot information in one eye, thus solving the problem that two sets of light sources are required to achieve gaze point estimation, thereby reducing the number of light sources of existing sight tracking devices, and realizing miniaturization and lightweight of sight tracking devices.

Glossary:

PCR (Pupil Corneal Reflection) is the pupil-corneal reflection method, which is a kind of optical recording method.

The method steps are:

First, an eye image with a light spot (also called a Purkinje spot) is obtained, and the reflection point of the light source on the cornea, namely the light spot, is obtained; as the eyeball rotates, the relative position relationship between the pupil center and the light spot changes accordingly, and the correspondingly collected eye images with the light spot reflect such a position change relationship; the line of sight/fixation point is estimated based on the position change relationship.

IPD (Inter Pupillary Distance) is the distance between the pupils of two eyes (left eye and right eye).

IGD (Inter Glint Distance) is the distance between two light spots in the eye image.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

FIG1 is a schematic diagram of a light source combination in the prior art;

FIG2 is a schematic structural diagram of a module of a gaze tracking device provided in an embodiment of the present application;

FIG3 is a flow chart of a method for estimating a gaze point provided in Embodiment 1 of the present application;

FIG4 is a schematic diagram of a PCR vector provided in an embodiment of the present application;

FIG5 is a flow chart of a method for calculating pupil spot center data of two eyes provided in Embodiment 2 of the present application;

FIG6 is a schematic flow chart of an exposure gain adjustment method provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of the structure of a gaze point estimation system provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The terms "first" and "second" and the like in the specification and claims of the present invention and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

In an embodiment of the present invention, a method for estimating a gaze point is provided, which can be applied to the field of eye tracking. Eye tracking, which can also be called line of sight tracking, is a technology for estimating the line of sight and/or gaze point of an eye by measuring eye movement. Eye tracking technology requires special equipment, such as an eye tracking device.

Among them, the line of sight can be understood as a three-dimensional vector, and the fixation point can be understood as the two-dimensional coordinates of the above three-dimensional vector projected on a certain plane. At present, the optical recording method is widely used, which uses a camera or a video camera to record the eye movement of the subject, that is, to obtain an eye image reflecting the eye movement, and extract eye features based on the acquired eye image to establish a line of sight/fixation point estimation model. Among them, eye features may include: pupil position, pupil shape, iris shape, iris position, eyelid position, eye corner position, light spot (also called Purkinje spot) position, etc.

Eye tracking methods can be roughly divided into two types: intrusive and non-intrusive. In the current gaze tracking system, most non-intrusive eye tracking methods are used, especially the pupil corneal reflection method is the most widely used. According to the physiological characteristics of the human eye and the principle of visual imaging, the collected eye diagram is processed by image processing technology to obtain the human eye characteristic parameters for gaze estimation. Taking the obtained human eye characteristic parameters as the reference point, the coordinates of the gaze point can be obtained by using the corresponding mapping model to achieve gaze tracking. This method has high accuracy, does not interfere with the user, and allows the user's head to rotate freely. The hardware equipment used includes a light source and an image acquisition device, wherein the light source is generally an infrared light source, because infrared light will not affect the vision of the eye, and can be multiple infrared light sources arranged in a predetermined manner, such as a herringbone shape, a straight line shape, etc.; the image acquisition device can be an infrared camera device, an infrared image sensor, a camera or a video camera, etc. In the corneal reflection method, in order to overcome the error caused by the asymmetric light spot, a multi-light source single camera or a multi-light source multi-camera is usually used to achieve gaze point estimation with free head movement.

For example, the light source is generally of the following two types: one is that the light source is separated from the camera position to form a normal pupil face image (also called a dark pupil); the other is that the iris reflects light due to the coaxiality of the light source and the camera, so that the camera can obtain a face image with a bright pupil (also called a bright pupil). Referring to FIG1 , which shows a schematic diagram of a light source combination in the prior art, it can be seen that the light source combination in the prior art is generally a combination of multiple dark pupils, or multiple dark pupils combined with one bright pupil, or one dark pupil combined with one bright pupil. In order to achieve a multi-light source solution, the distance between the two light sources is usually greater than 150 mm, which makes the existing line of sight tracking equipment larger in size.

Therefore, in an embodiment of the present application, a gaze point estimation method is provided, which is applied to a gaze tracking device provided with a single camera and a single coaxial light source. Since only one light source is provided in the gaze tracking device, the size of the gaze tracking device will be smaller.

Embodiment 1

Referring to FIG. 2, it shows a schematic diagram of the structure of a module of a gaze tracking device provided by an embodiment of the present application. In the module, only one infrared light source and one infrared camera are required, wherein the infrared light source is used as a bright pupil light source. It should be noted that for a bright pupil to appear, the light source needs to be located near the optical axis of the camera or on the same straight line as the optical axis of the camera. In this way, due to the mirror reflection principle of the pupil, the image of the pupil formed on the image is not black, but a very bright image. Using the module of the light source and the camera in the gaze tracking device can greatly reduce the size of the module, achieving the purpose of miniaturization and lightweight of the gaze tracking device. The infrared light source in FIG. 2 represents a single light source in the embodiment of the present application, that is, the single light source in the present application can be a light source, or a group of light sources presented as shown in FIG. 2, but the distance between each light source position point in the group of light sources is small, so that the overall light presentation is similar to a single light source, which is different from the scene in the prior art where the distance between the two light sources is greater than 150 mm.

In order to realize the estimation of the sight line/gaze point of the sight line tracking device provided with a single camera and a single coaxial light source, a gaze point estimation method is also provided in the first embodiment of the present application. Referring to FIG. 3 , the method may include the following steps:

S101 : Acquire an original image captured from a single camera.

S102: Collect human eye feature information of the original image, and calculate pupil spot center data of both eyes based on the human eye feature information.

The scene is illuminated by a single light source in the gaze tracking device, and an image is collected by a single camera in the gaze tracking device, wherein the collected image is an image including human eye feature information, such as human eye image, human face image, etc. Specifically, the human eye feature information may include information related to the pupil, such as the pupil position and pupil shape, and some iris-related information, such as the iris position and iris shape, and may also include light spot information formed by the light source irradiating the eye, etc.

After detecting the human eyes according to the original image, the pupil spot center data of the two eyes are calculated, that is, the center coordinate data of the pupil images of the two eyes and the center coordinate data of the spot images of the two eyes. Specifically, a two-dimensional coordinate system of the human eye image can be set, and the above-mentioned pupil spot center data can be determined according to the origin position and coordinate scale of the two-dimensional coordinate system.

S103, calculating and obtaining an initial PCR vector based on pupil spot center data;

S104. Normalize the initial PCR vector according to a preset distance factor to obtain a target PCR vector.

It should be noted that the PCR (Pupil Corneal Reflection) vector represents the vector from the center of the light spot to the center of the pupil, that is, the vector formed by the line connecting the center of the pupil and the center of the light spot. See Figure 4, which shows a schematic diagram of the PCR vector, and the arrow represents the PCR vector formed from the center of the light spot to the center of the pupil. Assuming the center of the pupil is (x1, y1) and the center of the light spot is (x2, y2), the PCR vector is (x1-x2, y1-y2). At this time, the calculation obtains an initial PCR vector.

In order to reduce the sensitivity of the PCR vector to the distance from the human eye to the lens in the embodiment of the present application, the initial PCR vector needs to be normalized using a distance factor. Specifically, the horizontal and vertical coordinate values of the initial PCR vector are divided by the distance factor. It should be noted that the distance factor is not a distance value, but a function of a related distance parameter. For example, taking the inter pupil distance (IPD) between the pupils of the two eyes as an example, IPD ² can be used as a function of the distance parameter, that is, the distance factor.

The above distance parameters may include the pupil distance between the two eyes, the spot distance between the two eyes, or the distance between any feature points of the two eyes, for example, the distance between the inner corners of the two eyes, the distance between the outer corners of the two eyes, the distance between the upper and lower eyelids, the size of the face, the scale of the key points of the face, etc. Correspondingly, in a possible implementation, the distance factor represents the functional expression of the square of the spot distance D between the two eyes, that is, D ² is used as the function of the distance parameter, which is the distance factor.

In different processing processes, other functions, such as cubic or square root, may also be used, which needs to be determined in combination with specific eye features of the user. Therefore, the embodiment of the present invention does not limit the specific form of the function of the distance parameter representing the normalization factor.

For example, in another possible implementation, the distance factor represents a function of the distance between the designated feature points of two eyes. If the distance between the feature points of two eyes is d, then the distance factor = d ³ .

S105. Calculate and obtain fixation point information according to the target PCR vector.

The corneal reflection method can be used to estimate the gaze point information, and the target PCR vector can be input into the preset regression model to calculate the gaze point information, or the target PCR vector and the human eye parameters can be input into the sight line calculation model to calculate the user's sight line coordinates, thereby obtaining the gaze point information that matches the user. Among them, the preset regression model and the sight line calculation model can be obtained by training the model using training samples including the PCR vector and the gaze point information.

The present invention provides a gaze point estimation method, in which pupil spot center data is obtained during the gaze point estimation process, and a target PCR vector is obtained after calculation and normalization based on the pupil spot center data, and gaze point information is obtained through the target PCR vector calculation. The gaze point estimation method only uses pupil spot center data when performing gaze point estimation, that is, only one spot and pupil position information can be used to estimate the distance information between the human eye and the camera, without using two spot information in one eye, thus solving the problem that two sets of light sources are required to achieve gaze point estimation, thereby reducing the number of light sources of existing sight tracking equipment, and realizing miniaturization and lightweight of sight tracking equipment.

Embodiment 2

In the second embodiment of the present application, a method for calculating pupil spot center data of two eyes is provided, referring to FIG5 , the method includes:

S201, collecting human eye feature information of the original image;

S202, acquiring pupil image features and spot image features of both eyes according to the human eye feature information;

S203, calculating and obtaining pupil image coordinates according to the pupil image features;

S204: Calculate and obtain the light spot image coordinates according to the light spot image features.

In this embodiment, the pupil spot center data of both eyes includes pupil image coordinates and spot image coordinates.

After obtaining the human eye feature information in the original image, the spot image feature can be used as a search condition to search in the original image to obtain the spot image. For example, the search can be performed based on the grayscale value of the spot image. The original image needs to be grayscale converted to obtain the grayscale value of each pixel of the original image, and then the spot image in the original image is determined based on the grayscale value range of the spot image.

After obtaining the spot image, the coordinates of the center point of the spot image can be used as the spot image coordinates. Correspondingly, when determining the pupil image coordinates, the pupil image can also be obtained by searching in the original image according to the pupil image features, and the center coordinates of the pupil image are determined as the pupil image coordinates.

In a specific implementation method, the pupil area is first extracted from the original image to facilitate the extraction of line of sight feature parameters. Since the pupil area is darker in the human eye image and has a very low grayscale value, a lower grayscale threshold can be set. By comparing the grayscale value in the image with the threshold, the candidate area of the pupil area is determined. The pupil area is the deepest part of the human eye image, and the pupil area is a circular shape. The human eye image is converted into binary image information according to the grayscale difference between the eye skin, sclera, iris and pupil in the human eye image, and the pupil area can be extracted. In the pupil area image, it includes the corneal reflection spot area, the complete pupil area and the partial iris area. Compared with other areas, the corneal reflection spot area is the part with the highest grayscale value, with a smaller area and brighter color, and the corneal reflection spot contained in each pupil area is distributed in a circular horizontal direction. Therefore, according to the characteristics of the corneal reflection spot, the pupil area is first binarized, a gray value threshold is set, and the area greater than the gray value threshold is taken as the candidate area, that is, the bright spot area of the pupil area is extracted, and then the noise bright spots in the pupil area are removed according to the bright spot area and shape to obtain the spot image corresponding to the spot area.

In another possible implementation, the center of the light spot may be used as the center, the pupil search range may be set, the pupil grayscale threshold may be obtained by calculating the grayscale histogram, and then the pupil center coordinates may be obtained.

Assume that the pupil image coordinates are (P _x , P _y ), the spot image coordinates are (G _x , G _y ), and the PCR vector coordinates are (x, y), then:

x＝norm( _Px - _Gx )

y＝norm(P _y −G _y )

Among them, norm represents normalization calculation, that is, in order to reduce the sensitivity of PCR to the distance from the human eye to the lens, PCR needs to be normalized using a distance factor, and the distance factor can be a related function of the pupil distance, the spot distance, and the distance of any feature points between the two eyes. It should be noted that in the embodiment of the present application, the spot distance refers to the spot distance between the two eyes. This is because the normalization factor used in the prior art is the distance between the two spots in the image, usually represented by IGD (Inter Glint Distance), but for the embodiment of the present application, there is no IGD in the case of using fewer light sources. Therefore, for some gaze point estimation scenes, new normalization factors that rely on fewer light sources and can achieve an ideal normalization effect should be considered. Here, the scale information on other images is used as a normalization factor to resist the influence of distance changes. That is, the normalization factor can be a related function of information such as pupil distance, the distance between the inner corners of the eyes, the distance between the outer corners of the eyes, the distance between the upper and lower eyelids, the size of the face, the spot distance between the eyes, and the scale of the key points of the face.

Embodiment 3

Since there are many light sources in the existing gaze tracking device, the control logic of brightness in order to achieve effective gaze point estimation is also relatively complex. In the third embodiment of the present application, an exposure gain control logic is provided, which only needs to be controlled according to the gray value of the pupil part.

Referring to FIG6 , it shows a schematic flow chart of an exposure gain adjustment method, the method comprising:

S301, calculating the average gray value of the pupil area according to the original image;

S302: judging whether to adjust the set exposure gain according to the average gray value so that the acquired original image satisfies the light spot search condition.

According to the image features of the pupil area, the pupil area is searched in the original image, and then the image corresponding to the pupil area is converted into a grayscale image, so that the grayscale value of each pixel in the grayscale image can be obtained, and then the average grayscale value of the pupil area is calculated. According to the set grayscale threshold, it is judged whether the average grayscale value of the pupil area exceeds the grayscale threshold. If so, the judgment result is fed back to the camera in the sight tracking device so that the camera can adjust the exposure gain. If not, the camera can shoot the image according to the current exposure gain. The specific adjustment method is to adjust the parameters related to exposure first. If the exposure-related parameters are adjusted to the limit value (i.e., the maximum value) and still cannot meet the spot search condition, the gain-related parameters can be adjusted again until the gain is adjusted to the limit value. It should be noted that the above-mentioned limit values all represent the maximum values of the corresponding parameters.

Specifically, the camera continuously acquires images. If the grayscale value of the pupil area of the current frame is calculated to know that the exposure gain needs to be adjusted, the camera starts adjusting from the current frame. The effect of the adjustment will be determined according to the actual hardware conditions of the camera. It may be effective in the next frame or five frames later. Therefore, in actual operation, the statistical threshold will continuously count the average value of five to ten frames, and detect every five or ten frames to prevent insufficient adjustment. Generally, the sign of the end of adjustment is determined according to the specific situation of the image. The main thing is that the edges of the pupil and the light spot should be clear and easy to extract the contour through the image algorithm. Specifically, the grayscale difference between the pupil and iris area should be sufficient, and the grayscale difference between the light spot and the pupil should be sufficient.

Of course, this can also be achieved by adjusting the brightness of the light source, such as adjusting the brightness of the infrared fill light to prevent the problem of spot search failure caused by excessive brightness.

A method for calculating gaze point information is also provided in an embodiment of the present application, specifically comprising:

The gaze point information is calculated based on a preset mapping relationship and the target PCR vector, wherein the preset mapping relationship represents a mapping relationship between the PCR vector and the gaze point and/or gaze direction.

Input the PCR of both eyes, and estimate and output the gaze point/gaze direction according to the established mapping relationship between the PCR vector and the gaze point/annotation direction.

X＝a0+a1x+a2x2+a3y+a4y2+a5xy

Y＝b0+b1x+b2x2+b3y+b4y2+b5xy

Where x, y are the coordinates of the PCR vector in the two-dimensional coordinate system; X, Y are the coordinates of the gaze point in the two-dimensional coordinate system, and related parameters such as a0, a1, a2, a3, a4, a5, b0, b1, b2, b3, b4, b5, etc. can be fitted during calibration.

After tracking and obtaining the user's gaze point information, the corresponding user's gaze point can be displayed on a display or a display module.

In the embodiment of the present application, since there is only pupil and spot information of the left and right eyes, different from the traditional solution of using the spots of two light sources to estimate the distance between the human eye and the camera, this solution uses the pupil distance as the evaluation scheme for the eye distance, solving the problem that two sets of light sources are required to estimate the sight line.

The module of the single camera and the single coaxial light source provided in the embodiment of the present application can be placed below or above the computer monitor to realize the eye tracking of the monitor; the module can also be placed in the mobile phone module to realize the eye tracking of the mobile phone. In addition to the monitor and the mobile phone, other devices can have similar sight lines to meet the device's eye tracking needs, which will not be explained one by one in the embodiment of the present invention.

Embodiment 4

In the fourth embodiment of the present application, a gaze point estimation system is also provided. The system is applied to a gaze tracking device provided with a single camera and a single coaxial light source, as shown in FIG7 , and includes:

An acquisition unit 10, configured to acquire an original image captured from the single camera;

A first calculation unit 20 is used to collect human eye feature information of the original image, and calculate pupil spot center data of both eyes based on the human eye feature information;

A second calculation unit 30 is used to calculate and obtain an initial PCR vector based on the pupil spot center data, where the PCR vector represents a vector pointing from the spot center to the pupil center;

A normalization unit 40, configured to perform normalization processing on the initial PCR vector using a preset distance factor to obtain a target PCR vector, wherein the preset distance factor represents a normalization parameter for normalizing the initial PCR vector;

The third calculation unit 50 is used to calculate and obtain the fixation point information according to the target PCR vector.

Based on the above embodiment, the first calculation unit includes:

A collection subunit, used for collecting human eye feature information of the original image;

An acquisition subunit, used to acquire pupil image features and spot image features of two eyes according to the human eye feature information;

A first calculation subunit, configured to calculate and obtain pupil image coordinates according to the pupil image features;

The second calculation subunit is used to calculate and obtain the spot image coordinates according to the spot image features; the pupil spot center data of the two eyes includes pupil image coordinates and spot image coordinates.

On the basis of the above embodiment, the preset distance factor represents a function of a distance parameter, wherein the distance parameter includes the distance between pupils of two eyes, the distance between light spots of two eyes, or the distance between specific feature points of two eyes.

Based on the above embodiment, the acquisition unit is specifically used to acquire the original image captured from the single camera according to the set exposure gain, and the system also includes:

A gray value calculation unit, used for calculating an average gray value of a pupil area according to the original image;

A judging unit, configured to judge whether to adjust the set exposure gain according to the average gray value so that the acquired original image satisfies the light spot search condition;

Wherein, the judging unit is specifically used for:

Determining whether the average grayscale value exceeds a preset grayscale threshold, and if so, adjusting the set exposure gain;

The system also includes:

An adjusting unit, used for adjusting the set exposure gain to obtain a target exposure gain;

The re-capturing unit is used to control the single camera to perform image capture according to the target exposure gain, so that the acquired original image meets the target exposure gain.

Based on the above embodiment, the third calculation unit is specifically used for:

The gaze point information is calculated based on a preset mapping relationship and the target PCR vector, wherein the preset mapping relationship represents a mapping relationship between the PCR vector and the gaze point and/or gaze direction.

The present invention provides a gaze point estimation system, in which pupil spot center data is obtained during the gaze point estimation process, and a target PCR vector is obtained after calculation and normalization based on the pupil spot center data, and gaze point information is obtained through the target PCR vector calculation. The gaze point estimation method only uses pupil spot center data when performing gaze point estimation, that is, only one spot and pupil position information can be used to estimate the distance information between the human eye and the camera, without using two spot information in one eye, thus solving the problem that two sets of light sources are required to achieve gaze point estimation, thereby reducing the number of light sources of existing sight tracking equipment, and realizing miniaturization and lightweight of sight tracking equipment.

Embodiment 5

Embodiment 5 of the present invention provides a processor, which is used to run a program, wherein the gaze point estimation method described in any one of Embodiments 1 to 3 is executed when the program is running.

Embodiment 6

Embodiment 6 of the present invention provides a device, the device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, the following steps are performed:

Acquire an original image captured from the single camera;

Collecting human eye feature information of the original image, and calculating pupil spot center data of both eyes based on the human eye feature information;

Based on the pupil spot center data, an initial PCR vector is calculated and obtained, where the PCR vector represents a vector pointing from the spot center to the pupil center;

The initial PCR vector is normalized using a preset distance factor to obtain a target PCR vector, wherein the preset distance factor represents a normalization parameter for normalizing the initial PCR vector;

According to the target PCR vector, the fixation point information is calculated and obtained.

Furthermore, the pupil spot center data of the two eyes include pupil image coordinates and spot image coordinates, and the collecting of human eye feature information of the original image and calculating the pupil spot center data of the two eyes based on the human eye feature information include:

Collecting human eye feature information of the original image;

According to the human eye feature information, pupil image features and spot image features of both eyes are acquired;

Calculating and obtaining pupil image coordinates according to the pupil image features;

The light spot image coordinates are calculated based on the light spot image features.

Furthermore, the preset distance factor represents a function of a distance parameter, wherein the distance parameter includes a distance between pupils of two eyes, a distance between light spots of two eyes, or a distance between designated feature points of two eyes.

Further, the acquiring of the original image captured from the single camera includes acquiring the original image captured from the single camera according to a set exposure gain, and the method further includes:

According to the original image, calculating the average gray value of the pupil area;

According to the average gray value, it is determined whether to adjust the set exposure gain so that the acquired original image meets the light spot search condition.

Further, judging whether to adjust the set exposure gain according to the average gray value includes:

It is determined whether the average grayscale value exceeds a preset grayscale threshold, and if so, the set exposure gain is adjusted.

Furthermore, the method further comprises:

Adjusting the set exposure gain to obtain a target exposure gain;

The single camera is controlled to perform image acquisition according to the target exposure gain, so that the acquired original image meets the target exposure gain.

Further, the calculating and obtaining the fixation point information according to the target PCR vector includes:

The gaze point information is calculated based on a preset mapping relationship and the target PCR vector, wherein the preset mapping relationship represents a mapping relationship between the PCR vector and the gaze point and/or gaze direction.

The devices in this article can be servers, PCs, PADs, mobile phones, etc.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

The present application is described with reference to the flowchart and/or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, and the combination of the process and/or box in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

In a typical configuration, a computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.

The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and/or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims (12)

1. A gaze point estimation method, characterized in that the method is applied to a gaze tracking device having a single camera and a single on-axis light source, comprising:

Acquiring an original image captured from the single camera;

Collecting human eye characteristic information of the original image, and calculating pupil spot center data of two eyes based on the human eye characteristic information;

based on the pupil spot center data, calculating to obtain an initial PCR vector, wherein the PCR vector represents a vector of the spot center pointing to the pupil center;

Normalizing the initial PCR vector by using a preset distance factor to obtain a target PCR vector, wherein the preset distance factor represents a normalization parameter for normalizing the initial PCR vector, and the preset distance factor represents a function of a distance parameter, and the distance parameter comprises the distance between pupils of two eyes, the distance between light spots of two eyes or the distance between appointed characteristic points of two eyes;

And calculating and obtaining the gaze point information according to the target PCR vector.

2. The method of claim 1, wherein the pupil spot center data of both eyes includes pupil image coordinates and spot image coordinates, wherein the acquiring human eye feature information of the original image and calculating pupil spot center data of both eyes based on the human eye feature information includes:

Collecting human eye characteristic information of the original image;

according to the human eye characteristic information, pupil image characteristics and facula image characteristics of two eyes are obtained;

calculating to obtain pupil image coordinates according to the pupil image characteristics;

and calculating to obtain the spot image coordinates according to the spot image characteristics.

3. The method of claim 1, wherein the acquiring the raw image captured from the camera comprises acquiring the raw image captured from the camera according to a set exposure gain, the method further comprising:

Calculating an average gray value of the pupil area according to the original image;

And judging whether the set exposure gain is adjusted according to the average gray value, so that the obtained original image meets the light spot searching condition.

4. A method according to claim 3, wherein said determining whether to adjust said set exposure gain based on said average gray value comprises:

judging whether the average gray value exceeds a preset gray threshold value, and if so, adjusting the set exposure gain.

5. The method of claim 4, further comprising:

Adjusting the set exposure gain to obtain a target exposure gain;

And controlling the camera to acquire images according to the target exposure gain, so that the acquired original images meet the target exposure gain.

6. The method of claim 1, wherein said computing gaze point information from said target PCR vector comprises:

And calculating and obtaining the gaze point information according to a preset mapping relation and the target PCR vector, wherein the preset mapping relation represents the mapping relation between the PCR vector and the gaze point and/or the gaze direction.

7. A gaze point estimation system, characterized in that the system is applied to a gaze tracking device having a single camera and a single on-axis light source, comprising:

an acquisition unit configured to acquire an original image captured from the single camera;

The first calculation unit is used for collecting human eye characteristic information of the original image and calculating pupil spot center data of two eyes based on the human eye characteristic information;

the second calculation unit is used for calculating and obtaining an initial PCR vector based on the pupil spot center data, wherein the PCR vector represents a vector of the spot center pointing to the pupil center;

The normalization unit is used for carrying out normalization processing on the initial PCR vector by utilizing a preset distance factor to obtain a target PCR vector, wherein the preset distance factor represents normalization parameters for carrying out normalization processing on the initial PCR vector, and the preset distance factor represents a function of distance parameters, wherein the distance parameters comprise the distance between pupils of two eyes, the distance between light spots of two eyes or the distance between appointed characteristic points of two eyes;

and a third calculation unit, configured to calculate and obtain gaze point information according to the target PCR vector.

8. The system of claim 7, wherein the first computing unit comprises:

the acquisition subunit is used for acquiring the human eye characteristic information of the original image;

The acquisition subunit is used for acquiring pupil image characteristics and facula image characteristics of two eyes according to the human eye characteristic information;

the first calculating subunit is used for calculating and obtaining pupil image coordinates according to the pupil image characteristics;

The second calculating subunit is used for calculating and obtaining the spot image coordinates according to the spot image characteristics; the pupil spot center data of the two eyes comprise pupil image coordinates and spot image coordinates.

9. The system according to claim 7, wherein the acquisition unit is specifically configured to acquire the original image captured from the single camera according to the set exposure gain, the system further comprising:

A gray value calculation unit, configured to calculate an average gray value of a pupil area according to the original image;

The judging unit is used for judging whether the set exposure gain is adjusted according to the average gray value so that the obtained original image meets the light spot searching condition;

wherein, the judging unit is specifically configured to:

judging whether the average gray value exceeds a preset gray threshold value, and if so, adjusting the set exposure gain;

the system further comprises:

an adjusting unit for adjusting the set exposure gain to obtain a target exposure gain;

And the re-acquisition unit is used for controlling the single camera to acquire images according to the target exposure gain so that the acquired original images meet the target exposure gain.

10. The system according to claim 7, wherein the third computing unit is specifically configured to:

And calculating and obtaining the gaze point information according to a preset mapping relation and the target PCR vector, wherein the preset mapping relation represents the mapping relation between the PCR vector and the gaze point and/or the gaze direction.

11. A processor for running a program, wherein the program is operative to perform the gaze point estimation method of any one of claims 1-6.

12. An electronic device comprising a processor, a memory and a program stored on the memory and executable on the processor, the processor implementing the gaze point estimation method of any of claims 1-6 when the program is executed by the processor.