JP2016173313A - Visual line direction estimation system, visual line direction estimation method and visual line direction estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a visual line direction estimation system, a visual line direction estimation method and a visual line direction estimation program capable of estimating a visual line direction by reducing the load of arithmetic processing.SOLUTION: A visual line direction estimation system 1 comprises a camera 10 for acquiring a two-dimensional image and a distance image to be photographed and a visual line direction estimation device 30. The visual line direction estimation device 30 includes a coordinate system setting unit 31 for using a two-dimensional image and a distance image of a face to set a coordinate system, and a visual line direction estimation unit 32. The visual line direction estimation unit 32 calculates first and second rotation angles by fitting the outline of an iris obtained by projecting a three-dimensional outline of an iris based on an eyeball model to an image plane to the outline of an iris extracted from a two-dimensional image of the face of an object person to be photographed, and estimates the visual line direction of the object person to be photographed on the basis of an optical axis calculated from the pupil center of the object person to be photographed determined on the basis of the calculated first and second rotation angles and the eyeball model, and the original point of a third coordinate system.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a gaze direction estimation system, a gaze direction estimation method, and a gaze direction estimation program.

  Estimating the human gaze direction is important in, for example, behavior analysis of car drivers, security monitoring, human behavior research, and human-computer interfaces. In recent years, an RGB-D camera capable of acquiring a two-dimensional image to be imaged and a distance image including distance information to the imaged object has been used as a conventional technique for estimating the gaze direction. An example of the RGB-D camera is Kinect (registered trademark) manufactured by Microsoft Corporation. For example, in Patent Document 1, the line-of-sight direction is estimated using captured image information from Kinect and an eyeball model. And in patent document 1, it is reported that the estimation error of a gaze direction averages about 10 degrees. Further, in Patent Document 2, it is reported that the estimation error of the gaze direction averages about 3.4 ° by further combining a learning process with the combination of photographed image information and eyeball model from Kinect.

J. Li, S. Li, “Eye-Model-Based Gaze Estimation by RGB-D Camera,” Computer Vision and Pattern Recognition Workshops, pp. 592-596, 2014 K. A. F Mora, J. Odobez, “Geometric Generative Gaze Estimation for Remote RGB-D Cameras,” In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014), pp. 1773-1780, 2014.

  However, for example, in the technique described in Non-Patent Document 2, it is considered that the calculation process increases because the learning process is included, and the load of the calculation process increases.

  Therefore, an object of the present invention is to provide means capable of reducing the processing load required to estimate the line-of-sight direction.

  A gaze direction estimation system according to the present invention is a gaze direction estimation system that estimates a gaze direction of a subject to be photographed, and a camera that acquires a distance image including a two-dimensional image of the photographing target and distance information to the photographing target; And a gaze direction estimation device that estimates the gaze direction of the subject to be photographed based on the two-dimensional image and the distance image of the face of the subject to be photographed by the camera. The gaze direction estimation device calculates posture information of the subject's head relative to the first coordinate system of the camera from the two-dimensional image of the face and the distance image, sets the second coordinate system, and sets the second coordinate system. A third coordinate system of the eyeball whose origin is the center of the eyeball of the subject to be imaged expressed by translational movement of the coordinate system, the origin of the third coordinate system is the origin, and the horizontal direction relative to the third coordinate system A coordinate system setting unit that sets a fourth coordinate system of the subject's iris represented by a first rotation angle that is a rotation angle and a second rotation angle that is a rotation angle in the vertical direction; When the three-dimensional contour of the iris in the fourth coordinate system is projected onto the image plane in the first coordinate system via the camera, the contour of the iris is obtained from the two-dimensional image of the face of the subject. First and second rotation angles by fitting to the extracted iris contour The subject is imaged based on the optical axis calculated from the pupil center of the subject to be imaged determined based on the calculated first and second rotation angles and the eyeball model, and the origin of the third coordinate system. And a gaze direction estimation unit that estimates the gaze direction of the subject.

  In the above configuration, by capturing the face of the subject to be photographed as a subject to be photographed with a camera, a two-dimensional image of the face of the subject and the distance image to the face can be obtained. Since the posture information of the head of the person to be photographed can be calculated using the two-dimensional image and the distance image acquired as described above, the coordinate system setting unit has the second coordinate system relative to the first coordinate system. Can be set. When this second coordinate system is set, the third coordinate system can be set as a coordinate system obtained by translating the second coordinate system. Since the third coordinate system is fixed with respect to the second coordinate system, if a translation vector indicating the transformation of the second and third coordinate systems is defined for the subject, The first to third coordinate systems can be mutually converted. Furthermore, the fourth coordinate system, which is a coordinate system for the iris of the eye, is represented by the first and second rotation angles with respect to the third coordinate system due to the structure of the eye. When the line-of-sight direction moves, the iris also moves, so the first and second rotation angles are parameters for estimating the line-of-sight direction.

  If the coordinate system setting unit sets the first to fourth coordinate systems, the gaze direction estimation unit estimates the gaze direction using the first to fourth coordinate systems and the eyeball model. Specifically, by using the first to fourth coordinate systems and the eyeball model that have been set, the contour of the iris in the eyeball model is converted into an image plane in the first camera coordinate system that is the camera coordinate system. Can project. The first and second rotation angles can be calculated by fitting the projected contour to the contour of the iris extracted from the actually captured two-dimensional image. Thereby, the fourth coordinate system is determined with respect to the third coordinate system, and the position of the iris in the eyeball model is known, so that the position of the pupil center can be calculated. An axis indicated by a straight line connecting the center of the pupil and the center of the eyeball, that is, the origin of the third coordinate system is known as an optical axis. Therefore, when the pupil center is calculated as described above, the optical axis can be calculated from the pupil center and the origin of the third coordinate system. Since the line-of-sight direction is known as a direction inclined by a predetermined angle with respect to the optical axis, the line-of-sight direction can be estimated by calculating the optical axis.

  In this case, two parameters of the first and second rotation angles are used as the unknown parameters for estimating the gaze direction by using the two-dimensional image of the face of the person to be imaged, the distance image to the face, and the eyeball model. Only. Therefore, according to the said gaze direction estimation system, the load of arithmetic processing can be reduced.

  The line-of-sight direction estimation unit is associated with each edge point when the three-dimensional contour of the iris in the fourth coordinate system, which is associated by fitting, is projected onto the image plane, and in the contour of the iris in the two-dimensional image. The first and second rotation angles can be calculated by solving the objective function that indicates the corresponding relationship with the corresponding edge point and includes the first and second rotation angles as unknown parameters.

  In one embodiment, the third coordinate system is based on a two-dimensional image and a distance image of the face of the subject to be photographed while the subject is facing the camera and gazing at the camera. You may further provide the calibration part which calibrates an origin. In this way, by calibrating the origin of the third coordinate system, the eyeball position corresponding to the person to be imaged can be known, so that the line-of-sight direction can be estimated more accurately.

  A gaze direction estimation method according to another aspect of the present invention is a gaze direction estimation method for estimating a gaze direction of a subject to be photographed, and includes a two-dimensional image of a photographing target and a distance image including distance information to the photographing target. A photographing process for photographing the face of the subject to be photographed using the camera to be obtained, and the head of the subject to be photographed with respect to the first coordinate system of the camera from the two-dimensional image and the distance image of the face obtained in the photographing step A third coordinate system of the eyeball with the origin of the center of the eyeball of the subject to be imaged expressed by translationally moving the second coordinate system, The iris of the subject to be imaged represented by the first rotation angle that is the horizontal rotation angle with respect to the third coordinate system and the second rotation angle that is the vertical rotation angle with respect to the third coordinate system. A coordinate system setting step for setting the fourth coordinate system of the Extracting the iris contour from the two-dimensional image of the face of the subject to be photographed when the three-dimensional contour of the iris in the fourth coordinate system is projected onto the image plane in the first coordinate system via the camera The first and second rotation angles are calculated by fitting to the contour of the iris to be determined, and the pupil center of the subject to be imaged is determined based on the calculated first and second rotation angles and the eyeball model. And a gaze direction estimation step of estimating the gaze direction of the subject to be photographed based on the optical axis calculated from the origin of the third coordinate system.

  A line-of-sight direction estimation program according to still another aspect of the present invention is a line-of-sight direction estimation program for estimating a line-of-sight direction of a subject to be photographed, and a computer is provided with a two-dimensional image of the photographing target and distance information to the photographing target The head of the subject to be photographed with respect to the first coordinate system of the camera from the two-dimensional image of the face and the distance image obtained by photographing the face of the subject to be photographed using a camera that acquires a distance image including A third coordinate system of an eyeball having an origin at the center of the eyeball of the subject to be imaged expressed by translational movement of the second coordinate system and calculating the posture information and setting the second coordinate system; A coordinate system setting for setting a fourth coordinate system of the subject's iris represented by a first rotation angle that is a horizontal rotation angle and a second rotation angle that is a vertical rotation angle with respect to the three coordinate systems. The process and the first through the camera, based on the eyeball model The iris contour when the three-dimensional contour of the iris in the first coordinate system is projected onto the image plane in the first coordinate system is used as the iris contour extracted from the two-dimensional image of the face of the subject. The first and second rotation angles are calculated by fitting, the pupil center of the subject to be imaged determined based on the calculated first and second rotation angles and the eyeball model, and the third coordinate system And a line-of-sight direction estimating step of estimating the line-of-sight direction of the subject to be imaged based on the optical axis calculated from the origin.

  By using the two-dimensional image of the face of the subject to be photographed and the distance image to the face obtained by photographing with the camera, the posture information of the head of the subject can be calculated. In the gaze direction estimation method and the gaze direction estimation program, the second coordinate with respect to the first coordinate system is obtained using the posture information calculated from the two-dimensional image of the face of the subject and the distance image to the face. A system is set, and third and fourth coordinate systems having a certain relationship with the second coordinate system are set.

  Since these first to fourth coordinate systems can be converted to each other, by using the set first to fourth coordinate systems and the eyeball model, the contour of the iris can be expressed in the camera coordinate system. Projection can be performed on an image plane in a certain first camera coordinate system. The first and second rotation angles can be calculated by fitting the projected contour to the contour of the iris extracted from the actually captured two-dimensional image. Thereby, the fourth coordinate system is determined with respect to the third coordinate system, and the position of the iris in the eyeball model is known, so that the position of the pupil center can be calculated. An axis indicated by a straight line connecting the center of the pupil and the center of the eyeball, that is, the origin of the third coordinate system is known as an optical axis. Therefore, when the pupil center is calculated as described above, the optical axis can be calculated from the pupil center and the origin of the third coordinate system. Since the line-of-sight direction is known as a direction inclined by a predetermined angle with respect to the optical axis, the line-of-sight direction can be estimated by calculating the optical axis.

  In this case, only two parameters, the first and second rotation angles, are used to estimate the line-of-sight direction by using the two-dimensional image of the subject to be imaged, the distance image to the face, and the eyeball model. It is. Therefore, according to the said gaze direction estimation method and gaze direction estimation program, the load of calculation processing can be reduced.

  In the line-of-sight direction estimation step, each edge point when the three-dimensional contour of the iris in the fourth coordinate system associated with the fitting is projected onto the image plane and the contour of the iris in the two-dimensional image The first and second rotation angles can be calculated by solving the objective function that indicates the corresponding relationship with the corresponding edge point and includes the first and second rotation angles as unknown parameters.

  The gaze direction estimation method according to an embodiment further includes a calibration step of calibrating the origin of the third coordinate system based on the two-dimensional image and the distance image of the face of the subject to be imaged, and the calibration step is performed. In the photographing step, the subject person may be photographed while the subject person is facing the camera and gazing at the camera.

  Similarly, in the gaze direction estimation program according to one embodiment, the computer captures the subject to be photographed while the subject is facing the camera and gazing at the camera. A calibration step for calibrating the origin of the third coordinate system may be further executed based on the two-dimensional image and the distance image of the face of the person to be photographed.

  In this way, by performing the calibration process or causing the computer to execute the eyeball position corresponding to the subject to be imaged, the line-of-sight direction can be estimated more accurately.

  ADVANTAGE OF THE INVENTION According to this invention, the means which can shorten the processing time required for estimation of a gaze direction can be provided, maintaining the precision of gaze direction estimation.

The schematic diagram of the camera which the gaze direction estimation system which concerns on one Embodiment uses appears. It is a schematic diagram of the eyeball model used when estimating a gaze direction with the gaze direction estimation system according to an embodiment. It is a schematic diagram of four coordinate systems used when estimating a gaze direction with the gaze direction estimation system which concerns on one Embodiment. It is a block diagram of the gaze direction estimation system concerning one embodiment. It is drawing for demonstrating the position coordinate of the point on the outline of the iris in an eyeball model. It is drawing which shows the experimental result of the iris fitting experiment. It is drawing which shows the experimental result of a gaze direction estimation. It is a graph which shows the average error of a gaze direction estimation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals are used for the same elements, and redundant descriptions are omitted.

  The gaze direction estimation system according to an embodiment is for photographing a subject to be photographed with a camera, and estimating the gaze direction of the subject to be photographed using the photographed image and the eyeball model 20. The gaze direction estimation system is a system that estimates a human gaze direction, and can be applied to, for example, driver behavior analysis, security monitoring, human behavior research, and a human-computer interface.

  The gaze direction estimation system uses the camera 10 schematically shown in FIG. The camera 10 is a so-called RGG-D camera that can acquire a distance image including distance information to the shooting target together with an RGB image (two-dimensional image) of the shooting target. An example of the camera 10 is Microsoft's Kinect (registered trademark).

  The camera 10 includes an RGB camera 11 and a distance image acquisition unit 12. The RGB camera 11 is a two-dimensional image acquisition unit that acquires a two-dimensional color image to be photographed. The distance image acquisition unit 12 includes an infrared (IR) camera 12A and an IR projector 12B that projects a predetermined pattern with infrared rays.

  In the camera 10, an RGB image is acquired by the RGB camera 11. Further, the IR camera 12A captures a predetermined pattern projected using IR (infrared rays) by the IR projector 12B, thereby acquiring a distance image based on the distortion state of the predetermined pattern.

  The eyeball model 20 used in the gaze direction estimation system will be described with reference to FIG. In FIG. 2, the camera 10 is schematically shown for convenience of explanation. The eyeball model 20 is obtained by modeling the structure of the human eye with two spheres 21 and 22 having different sizes. In FIG. 2, a sphere 21 is an eyeball, which corresponds to a so-called white eye. The sphere 22 is located in the vicinity of the surface of the sphere 21 and is a so-called corneal model, which corresponds to the black eye. The iris and pupil are in the cornea.

And the center C of the sphere 21, the pupil (or iris) virtual line that connects the center P _f corresponds to the optical axis S _O. Hereinafter, the center C is also referred to as an eyeball center C. Center _{C P} of the sphere 22 is located on the optical axis _{S O.} The viewing direction D _G herein means a direction of the visual axis, and the center C _P of the sphere 22 is a model of the cornea, which is the direction of a virtual line connecting the gaze point G. Viewing direction D _G includes an optical axis S _O, which form a predetermined angle theta. The predetermined angle θ can be regarded as a substantially constant angle although there are individual differences. In the following description, unless otherwise specified, the angle θ is constant regardless of the subject to be imaged in which the line-of-sight direction _DG is estimated.

As described above, the iris and pupil are located inside the cornea, but the camera 10 images the corneal surface. In FIG. 2, the intersection of the optical axis _SO and the sphere 22 is defined as the pupil center P. _{Illustrated as f} . If the imaging subject is photographed by the camera 10, P _f point indicated in FIG. 2 appears on the RGB image. A curve A formed by the intersection of the sphere 22 and the sphere 21 is defined as the outline of the iris. Hereinafter, the curve A is also referred to as a contour A.

In the eyeball model 20, the distance between the center C _p of the center C and the sphere 22 of the sphere 21 and K _0, and the center C, the distance between the pupil center P _f and K. K ₀ and K are constants set in the eyeball model 20. Although K ₀ and K are actually different among individuals, it is sufficient to set values based on the general number of human eyeballs, assuming that individual differences are almost negligible. For example, K = 13.1 mm and K ₀ = 5.3 mm.

Next, a coordinate system used in the gaze direction estimation system will be described with reference to FIG. To estimate the gaze direction D _G of the imaging subject, as shown in FIG. 3, four coordinate system, i.e., the camera coordinate system (first coordinate system) S _C, head coordinate system (second coordinate system) _{S H,} eyeball coordinate system (third coordinate system) _{S e} and iris (iris) coordinate system (fourth coordinate system) using the _{S p.} The coordinate system may be a right-handed system or a left-handed system, but in the following, all four coordinate systems are right-handed coordinate systems unless otherwise specified. In the eyeball model 20 shown in FIG. 3, the sphere 22 is omitted and only the sphere 21 is shown. However, for convenience of explanation, the outline A of the iris is shown. 3, the plane Q indicated by hatching is a plane containing the contour A, the point q is the intersection of the plane Q and Z _p axis.

The camera coordinate system S _C is a three-dimensional coordinate system (O _C —X _C Y _C Z _C ) set for the camera 10. In the present specification, Z _C axis of the camera coordinate system S _C is oriented in the imaging subject side, Y _C axis is oriented vertically. In the following, unless otherwise specified, the camera coordinate system _{S C} is the coordinate system of the RGB camera 11. In one embodiment, the origin _{O C} of the camera coordinate system _{S C} can be set as the optical center of the RGB camera 11. However, since the coordinate system and the coordinate system of the IR camera 12A of the RGB camera 11 can be converted to each other, it may be set in the coordinate system of the IR camera 12A and camera coordinate system _{S C.}

Head coordinate system _{S H} is a three-dimensional coordinate system set with respect to the imaging subject's head _{(O H -X H Y H Z} H). In the present embodiment, from the viewpoint of facilitating the conversion between the coordinate system, the vertical direction in correspondence to the camera coordinate system S _C and Y _H axis direction. Further, Z _H axis, and facing the back of the head. Head coordinate system S _H and the camera coordinate system S _C is interconvertible, the coordinate transformation is represented by translation vector T and the rotation matrix R. The translation vector T and the rotation matrix R are also posture information with respect to the camera 10 of the head.

Eyeball coordinate system _{S e} is a 3-dimensional coordinate system set with respect to the eye of the imaging subject _{(O e -X e Y e Z} e), the eye model 20 shown in FIG. 1, the sphere 21 This is a coordinate system set for the user. Therefore, the origin O _e corresponds to the center C of the sphere 21. Human structural, ocular coordinate system S _e can be set the head coordinate system S _H as obtained by translation. Therefore, the eye coordinate system S _e and the head coordinate system S _H can be converted to each other, the coordinate transformation is represented by translation vector T _e. Therefore, Y _e axis of the eyeball coordinate system S _e is the vertical direction, Z _e axis is oriented to the back of the head.

Iris coordinate system _{S p} is a three-dimensional coordinate system set with respect to the iris of the imaging subject _{(O p -X p Y p Z} p). The origin O _p of the iris coordinate system S _p is the same as the origin O _e of the eyeball coordinate system S _e , and the Z _p axis is set as an axis that passes through the pupil center P _f and is orthogonal to the iris plane. Since the origin of the iris coordinate system S _p and eye coordinate system S _e is the same, between the iris coordinate system S _p and the eyeball coordinate system S _e is roll alpha, yaw (first rotation angle) beta and pitch ( Mutual conversion is possible using each rotation angle of (second rotation angle) γ.

However, when the iris moves on the eyeball, due to the human structure, the iris is only rotated around the vertical direction (ie, horizontal movement) and rotated around the horizontal direction (ie, vertical movement). . Therefore, the iris coordinate system _Sp can set the roll α to 0. Therefore, the conversion between the iris coordinate system S _p and eye coordinate system S _e is expressed by only the yaw β and pitch γ is a two rotation angles. Note that the yaw beta, the rotation angle around the X _e axis of the eyeball coordinate system S _e, shows a lateral movement when viewed face of the imaging subject. The pitch γ indicates a rotation angle around the Y _e axis, and indicates a vertical movement when the face of the person to be imaged is viewed.

Camera 10 has been described above, the line-of-sight direction estimation system 1 for estimating a gaze direction D _G by using eye model 20 and the four coordinate system, as the block diagram shown in FIG. 4, the camera 10, the line-of-sight direction estimation Device 30. FIG. 4 is a functional block diagram illustrating the main functions of the camera 10 and the line-of-sight direction estimation device 30. The description of the camera 10 is omitted since it has been described above, but in the following, the camera 10 is referred to as Kinect.

  The line-of-sight direction estimation device 30 is electrically connected to the camera 10 via wiring, and an image (specifically, image data) acquired by the camera 10 is input to the line-of-sight direction estimation device 30 via wiring. The The image from the camera 10, for example, may be input to the gaze direction estimation device 30 using wireless communication.

Gaze direction estimation unit 30, a coordinate system setting unit 31 for setting a four coordinate system shown in FIG. 2, the line-of-sight direction estimation unit 32 for estimating a gaze direction D _G, the origin O _e of the eyeball coordinate system S _e, That is, it has the calibration part 33 which calibrates the position of the eyeball center, and the memory | storage part 34 which stores the various programs and data containing a gaze direction estimation program.

  The line-of-sight direction estimation apparatus 30 is an information processing apparatus (computer apparatus) that includes a CPU, and the information processing apparatus executes the relative orientation information creation program stored in the storage unit 34 so that the information processing apparatus can detect the line-of-sight direction estimation apparatus 30. Function as.

  Each component (such as the coordinate system setting unit 31, the line-of-sight direction estimation unit 32, and the calibration unit 33) included in the line-of-sight direction estimation device 30 is connected by a bus or the like, and can communicate data and the like. The gaze direction estimation device 30 executes the gaze direction estimation program stored in the storage unit 34, thereby realizing the functions of the coordinate system setting unit 31, the gaze direction estimation unit 32, and the calibration unit 33. The line-of-sight direction estimation device 30 includes other components included in a normal information processing device, that is, an input unit that receives input of data and instructions (for example, user instructions), an output unit that outputs data, and the like. Below, it demonstrates focusing on the characteristic part of the gaze direction estimation apparatus 30. FIG.

  The coordinate system setting unit 31 is based on the RGB image (two-dimensional image) and the distance image of the face of the person to be imaged captured by the RGB camera 11 and the distance image acquisition unit 12 of the camera 10 as shown in FIG. Set one coordinate system.

  Specifically, based on the RGB image and the distance image of the face of the subject to be photographed input from the camera 10, the posture information of the head of the subject, ie, the rotation matrix R and the translation vector T are calculated. . This can be calculated by a known technique based on, for example, a change in the image when the head moves. When the camera 10 is Kinect, it is automatically calculated by the Kinect function.

Using the calculated R and T, it sets the head coordinate system S _H relative to the camera coordinate system S _C. A head coordinate system _{S H,} to set the eye coordinate system _{S e} on the basis of the translation vector _{T e.} Furthermore, with respect to the eyeball coordinate system S _e, to set the iris coordinate system S _p represented by the yaw β and pitch gamma.

The calibration unit 33 calibrates the eyeball center C based on the RGB image and the distance image when the subject is gazing at the RGB camera 11. Specifically, for calibrating the translation vector T _e to convert a head coordinate system S _H and the eyeball coordinate system S _e. The calibration method is the same method as that of Non-Patent Document 1, except that the person to be imaged is gazing at the direction of the RGB camera 11.

Specifically, in FIG. 1, when the imaging subject is gazing the RGB camera 11, the gaze point G in FIG. 1 correspond to the origin O _C of the camera coordinate system S _C. The position vector of each of the camera coordinate system _{S C} transformed eyeball center C (center of the sphere 21) and _{C C,} the position vector of the center Cp of the sphere 22 and _{C pC,} the position vector of the pupil center _{P f P fC} And

The following equation (1) is established from the relationship that the angle formed by the vector [C _pC O _C ] and the vector [C _pC P _fC ] is θ. Incidentally, _[C pC _{O C]} shows a vector directed to the position vector _{O C} from the position vector _{C pC,} shows a vector directed to _[C pC _{P fC],} the position vector _{P fC} from position vector _{C pC} Yes.

The position vector _{C pC,} between the position vector _{C C} and the position vector _{P fC,} the following formula (2) is satisfied.

In the formula (2), K and _{K 0} is a constant. Further, P _fc can be calculated based on an RGB image that is captured when a non-photographing subject is gazing at the direction of the RGB camera 11. Specifically, in the following reference, as proposed by Febian Timm and Erhardt Barth et al., P _fc as the pupil center is calculated using the gradient of the edge point of the iris extracted from the RGB image. The
Reference: F. Timm, E. Barth. Accurate eye center localization by means of gradients. In Proceedings of the International Conference on Computer Theory and Applications, volume 1, pp. 125-130, 2011.

Accordingly, the nonlinear equation shown in the equation (1) can be expressed by, for example, the Levenberg-Marquardt Algorithm while using P _fc obtained based on the image taken while changing the position of the head of the subject to be photographed a plurality of times. The eyeball center C can be calibrated by solving using (LMA). As a result, a calibrated translation vector _Te is obtained. Since the eyeball center C is represented by three-dimensional coordinates, it has three parameters. Therefore, the position of the head of the subject to be imaged may be changed at least three times.

Gaze direction estimation unit 32, via the RGB camera 11, projecting a 3-dimensional contour of the iris in the iris coordinate system S _p based on the eyeball model 20 the image plane in the camera coordinate system S _C. Then, the line-of-sight direction estimation unit 32 calculates the pupil center of the subject to be photographed by fitting the projected iris contour to the iris contour extracted from the RGB image of the face of the subject. To do. Gaze direction estimation unit 32, the direction of this from the calculated pupil center and the origin O _e of the eyeball coordinate system S _e to calculate the optical axis S _O, viewing direction D _G is a predetermined angle θ relative to the optical axis S _O estimating the line of sight direction D _G of the imaging subject based on the relationship between the viewing direction D _G and the optical axis S _O that is say it.

  The principle of the gaze estimation in the gaze direction estimation unit 32 will be described more specifically.

<Calculating the initial value of the pupil center>
Using the RGB image, the pupil center is calculated by the method described in the above-mentioned reference. Specifically, the eyes on the RGB image are extracted based on the feature points of the RGB image. For example, when the camera 10 is Kinect, the eye can be extracted by using Kinect's SDK library. Further, an iris is extracted from the extracted eye region based on the feature points. This extraction may be performed in a known direction. Next, based on the gradient of the edge point on the iris outline, the position of the iris center, that is, the pupil center is calculated. The position of the pupil center is taken as the initial value (or initial position) of the iris center.

<Iris Fitting>
Iris, the structure of the human eye, the shape when viewed from the normal direction (Z _p-axis direction of the optical axis S ₀ or diagram of Fig. 2 3) is circular, the table in ellipse on normal image Is done. Therefore, the contour of the iris in three-dimensional space, the shape when viewed from the normal direction of the iris approximate ellipse, the coordinates in the iris coordinate system S _p of the point (edge point) P _pi on the iris outline, Figure As shown in FIG. 3, (a · cos (t _i ), b · sin (t _i ), L) is expressed as (i is an arbitrary integer of 1 or more, and is used to distinguish edge points. is there).

Here, L, a, the b and _{t i} is described using the coordinate display of the point _{P pi} using 3 and 5. As shown in FIG. 3, the intersection point between the plane Q and the Zp axis is _defined as a point q. L is the distance between the point q and the origin O _p, and is set in the eyeball model 20. Although L has individual differences, it can be regarded as a substantially constant value, and the value of L is, for example, 10.5 mm. Contour A is included in a plane Q, to perpendicular to the Z _p axis and the plane Q, the value of Z _p coordinates in the iris coordinate system S _p of points on the contour A becomes L.

FIG. 5 is a schematic diagram of the contour A viewed from the Zp-axis direction. In FIG. 5, the contour A is represented as an ellipse. For illustration, the axis parallel to the street X _p axis point q and x _p axis, and y _p axis parallel to the Y _p axis. It said a is the half of the length of the contour A (ellipse) in x _p axis, the contour A, corresponding to the transverse radius of the ellipse when viewed from the Zp-axis direction. Meanwhile, b is half the length of the contour A (ellipse) in y _p direction, a contour A, corresponding to the longitudinal radius of the ellipse when viewed from the Zp-axis direction. t _i is a parameter (parameter) indicating an angle formed by the straight line connecting the point q and the point P _pi . When a = b, the contour A is circular. In this case, the X _pi (x _pi ) coordinates and the Y _pi (y _pi ) coordinates of the point P _pi are r · cos (t _i ) and r · sin (t _i ), respectively, where r is the radius of the circle. expressed. 3 and 5, since the contour A is approximated by an ellipse, when representing X _pi (x _pi ) coordinates, a is used instead of r, and when representing Y _pi (y _pi ) coordinates, That is, b is used instead of r.

Position vector _{P pi} of the point _{P pi} is the following equation can be converted into the position vector _{P ei} on the eyeball coordinate system _{S e.}

R _P is the iris coordinate system S _p is a transformation matrix for varying the eye coordinate system S _e, roll (roll) alpha, is represented by the yaw β and pitch gamma, as described above, the roll alpha is 0. “·” In the equation (3) indicates an inner product.

Then converted position vector _{P ei} in the eye coordinate system _{S e} to position vector _{P hi} head coordinate system _{S H.} Eyeball coordinate system S _e, since corresponding to that translates with respect to the head coordinate system S _H, the position vector P _hi is expressed as follows.

Subsequently, the position vector _{P hi,} into a position vector _{P ci} of the camera coordinate system _{S C.} This conversion is expressed by the following equation. “·” In the equation (5) indicates an inner product.

Position vector P _ei represents the three-dimensional position of the edge points on the contour of the iris in the camera coordinate system S _C. If P _ci is represented as (x _ci , y _ci , z _ci ), and the position vector P _ci is the position vector of the image point on the image plane projected by the RGB camera 11, I _Pi (u _i , v _i ) I _Pi is expressed as follows.

Here, M is a projection matrix including internal parameters of the camera 10 (two-dimensional image unit). From the above equations, u _i and v _i are expressed as follows using the functions f and g.

From the above equations (7a) and (7b), the following equations are derived.

In Expression (8a) and Expression (8b), functions h and k are functions obtained by solving Expression (7a) and Expression (7b).

If the relationship of sin ² (t _i ) + cos ² (t _i ) = 1 is used, the objective function Ψ is expressed by equation (9). The objective function Ψ is a function that indicates a correspondence relationship between the point P _p on the iris outline projected on the image plane based on the eyeball model 20 and the edge point of the iris outline on the corresponding RGB image.

In Expression (9), R and T are obtained from the RGB image and the distance image acquired by the camera 10. M is known because it is a projection matrix unique to the camera 10. Since L, a, b, T _e , u _i , and v _i are set in the eyeball model 20, they are known. Therefore, in Equation (9), two parameters β and γ are unknown parameters.

  One edge point group is obtained from one RGB image. Furthermore, the initial value of the pupil center can be obtained by the method described in <Calculation of initial value of pupil center>. Therefore, the initial β and γ are known. In this state, Equation (9) can be solved using the Levenberg-Marquardt Algorithm (LMA). Specifically, since the equation (9) is obtained for each edge point constituting the edge point group, it is optimal to minimize the error in the plurality of equations (9) obtained for the edge point group. β and γ are calculated.

When β and γ are calculated, the position vector P _fc of pupil center in the camera coordinate system S _C has the formula (3), (4) and iris coordinate system by utilizing the same conversion as the formula (5) S _p Is calculated from the position vector (0, 0, K). It is assumed that the distance K between the pupil center _Pf and the eyeball center C is a constant.

<Gaze estimation>
If the calculated pupil center P _f as described above, and the eyeball center C in the eyeball model 20, since the optical axis S _O shown in FIG. 1 from the pupil center P _f is calculated using a predetermined angle θ A line-of-sight direction _DG is calculated.

Gaze direction estimation unit 32, based on the above principle, by calculating the β and γ using the equation (9), to identify the optical axis S _O, estimates the viewing direction D _G. In the camera coordinate system S _C, the viewing direction D _G, X _C Z and projection line when projected onto the _C plane, the angle between the Z _C-axis and [delta], the angle between the projection line and the line-of-sight direction D _G (in other words, the angle between a sight line direction _{D G} and _X C _{Z C} plane) is defined as [psi.

  Next, a gaze estimation method for the person to be imaged using the gaze direction estimation system 1 will be described. First, a gaze estimation method in a state where the eyeball center C is calibrated will be described.

  The camera 10 captures the face of the person to be imaged, and acquires an RGB image and a distance image of the face (imaging process). At this time, it is only necessary that the head of the person to be imaged is photographed so that the face is captured. Therefore, the upper body of the subject person may be photographed, or the whole body of the subject person may be photographed.

  When the line-of-sight direction estimation device 30 receives an image captured by the camera 10, the coordinate system setting unit 31 of the line-of-sight direction estimation device 30 sets the four coordinate systems shown in FIG. 2 (coordinate system setting step).

Next, line-of-sight direction estimation unit 32, by using the RGB image and the distance image from the camera 10, estimates the viewing direction D _G of the imaging subject in the above-described method (line-of-sight direction estimation step).

Specifically, first, to identify the edge points on the contour of the iris included in the eye of the RGB image, and calculates the initial value of the pupil center P _f (initial value calculating step).

Next, when the iris contour in the eyeball model 20 is projected on the RGB image represented by the image plane of the RGB camera 11, the projected contour and the iris contour in the actually acquired RGB image are used. Fi and coating for, calculates a yaw β and pitch γ is the rotation angle with respect to the eyeball coordinate system S _e iris coordinate system S _p. Specifically, yaw β and pitch γ are calculated using equation (9) and the initial value calculated in the initial value calculation step (fitting step (or rotation angle calculation step)).

Based on the calculated yaw β and pitch γ and eye model 20 determines the optical axis S _O, by calculating a predetermined angle θ to the optical axis S _O, it calculates the viewing direction D _G (gaze direction calculating step) .

  Next, the case where the calibration process of the eyeball center C is included will be described.

When the eyeball center C is calibrated, in the photographing step, the face of the subject is photographed while the subject is facing the camera 10 and the camera 10 is being watched. In the present embodiment, the camera coordinate system of the RGB camera 11, since the set to the camera coordinate system S _C of the camera 10, the imaging subject is gazing at the orientation and RGB camera 11 toward the RGB camera 11 Preferably it is.

When the calibration image is input to the line-of-sight direction estimation device 30, the coordinate system setting process is performed as described above. Thereafter, the calibration unit 33 calibrates the position of the eyeball center C by the method described above. Specifically, the translation vector _Te is calculated. The calculated translation vector _Te is stored in the storage unit 34, for example.

The calibration of the eyeball center C may be performed once for the same subject. Then, after performing the calibration process, the line-of-sight direction estimation, calibrated position of the eyeball center C in the calibration step, in other words, may be used to translation vector T _e. Whether or not the line-of-sight direction estimation device 30 performs the calibration process is determined, for example, by the presence or absence of a mode switching input whether or not the calibration is performed via the input unit of the line-of-sight direction estimation device 30. That's fine. Further, for example, when the gaze direction estimation device 30 is used in a driving support system for a car, if the calibration process is performed once for each driver, then each driver can calibrate his / her own calibration data via the input unit. Can be selected to use.

  In the gaze direction estimation system 1 (gaze direction estimation program) and the gaze direction estimation method, the iris contour is represented by two unknown parameters β and γ by using the eyeball model 20. Normally, there are 5 parameters indicating an ellipse (a parameter indicating the long axis, a parameter indicating the short axis, a parameter indicating the inclination, and two parameters indicating the center of the ellipse), but the number of parameters is reduced. The processing load required for line-of-sight estimation is reduced. As a result, the speed can be improved, and the calculation processing time can be shortened.

Further, only a part of the iris may be reflected on the RGB image due to the direction of the eye of the subject to be photographed or the closed state of the eyelid. Therefore, the calculation error of the pupil center P _f becomes large when calculating the pupil center P _f on the basis of only the iris extracted from the RGB image. On the other hand, in this embodiment, the pupil center _Pf is calculated by fitting the iris based on the eyeball model 20 to the iris on the RGB image. Therefore, it is possible to obtain the pupil center P _f more accurately. Therefore, it is possible more accurately calculate the viewing direction D _G.

Further, in the calibration unit 33, since it is calibrated eye center C, it can be more accurately estimate the gaze direction D _G. When the eyeball center C is calibrated by the calibration unit 33, an image photographed while the subject is facing the camera 10 and gazing at the camera 10 is used. For this reason, for example, the target is positioned between the camera 10 and the person to be imaged and cannot be calibrated as in the conventional case where the object person is looking at a target different from the camera 10. Absent. Therefore, the eyeball center C can be calibrated more reliably and accurately.

Next, experimental results will be described. In the following experiments 1 and 2, K, K ₀ and L shown in FIG. 2 were set to K = 13.1 mm, K ₀ = 5.3 mm, and L = 10.5 mm, respectively. The camera 10 used Kinect. The number of pixels of the RGB image obtained by Kinect was 1280 × 960 pixels, and the number of pixels of the distance image was 640 × 480 pixels. The distance between the subject and the camera 10 was about 70 cm (0.7 m).

<Experiment 1>
In Experiment 1, the head of the person to be imaged was fixed, and various images were taken with the camera 10 with the eyes directed in various directions, and the results shown in FIG. 4 are based on the obtained RGB images and distance images. The line-of-sight direction estimation unit 32 of the line-of-sight direction estimation apparatus 30 calculates the yaw β and the pitch γ. For the camera 10, Kinect was used. Then, by projecting the contour and pupil center P _f of the iris represented by iris coordinate system S _p on the basis of the yaw β and pitch γ calculated on each RGB image.

  FIG. 6 is a diagram showing the result of iris fitting. 6A is a diagram illustrating an image when the subject is facing the camera 10 and is gazing at the camera 10, and is illustrated in FIGS. 6B to 6E. 6A and 6B show images when the eyes are looking at the upper side, the lower side, the right side (viewed from the subject) and the left side (viewed from the subject) from the state of FIG. It is.

  In FIG. 6A to FIG. 6E, the white circle is the pupil center calculated by the eyeball model 20 in the gaze direction estimation unit 32, and the x mark is the pupil center based only on the RGB image, This is the initial value of the above description. Further, Δ marks indicate the edge points of the iris extracted from the RGB image. On the other hand, the white circle is an iris fitting result based on the yaw β and the pitch γ calculated by the line-of-sight direction estimation unit 32.

FIG. 6A shows a case where the subject is facing the camera 10 and is gazing at the camera 10, so that the entire iris appears in the image. Therefore, the position of the pupil centers P _f is substantially coincident with the case of using the result of solving the initial value calculated in the method described in <Initial value calculation of pupil center> described above, equation (9) . Also, the iris edge points extracted from the RGB image are almost the same as the fitting results. When the above-described calibration image of the eyeball center C is captured, it is preferable to capture in the state of FIG.

In FIG. 6B to FIG. 6E, the iris edge points extracted from the RGB image are almost the same as the fitting result, but the eyes are not facing the front, and partly with eyelids or the like. Therefore, there is a difference between the case where the pupil center P _f is calculated using β and γ calculated by the equation (9) and the initial value.

Therefore, if the pupil center _Pf is calculated using the gaze direction estimation device 30 shown in FIG. 4, for example, even if the iris is hidden by a cocoon or the like and only a part of it appears in the image, the pupil center is accurately determined. P _f can be calculated. As a result, it is understood that more accurately estimated even viewing direction D _G.

<Experiment 2>
Next, the subject to be photographed is photographed a plurality of times by the camera 10 while the head is moving with respect to the camera 10 while constantly gazing at the RGB camera 11 of the camera 10, and based on each photographing. the viewing direction D _G of the imaging subject was estimated by eye direction DG estimation method described above. Specifically, (δ, ψ) was calculated. The camera 10 used Kinect. In addition, the number of times of photographing was 48.

  FIG. 7 is a diagram illustrating a gaze direction estimation result. Specifically, (a) in FIG. 7 is an experimental result for the left eye of the subject, and (b) in FIG. 7 is an experimental result for the right eye of the subject. In FIGS. 7A and 7B, the horizontal axis represents the number of times of photographing, and the vertical axis represents the angle. 7A and 7B show δ and ψ as estimated values.

  In addition, during the experiment, since the subject to be photographed is always gazing at the RGB camera 11, a virtual linear direction that connects the position of the subject's eye at each photographing point and the RGB camera 11. Is the true value of the line-of-sight direction DG at the photographing point. In FIGS. 7A and 7B, δ and ψ as true values are also shown for comparison.

7 from (a) and (b), a slight blur viewing direction D _G close to the true value of what is it can be seen that can be calculated.

  The experiment 2 was performed on different subjects H1, H2, and H3. As a result, an average of errors of δ and ψ in the eye direction estimation results of the left eye and the right eye was calculated. The calculation results are shown in FIG. In FIG. 8, the average of the error of the left eye and the right eye of each of the subjects H1, H2, and H3 was averaged by three persons was 3.25 °.

  Although the experimental conditions and the like are different, they cannot be generally compared. However, in Non-Patent Document 2, since the average estimation error in the line-of-sight direction is reported to be 3.4 °, the method described in this embodiment uses Non-Patent Document 2. That is, the gaze direction can be estimated with almost the same accuracy. On the other hand, in the present embodiment, as described above, since the arithmetic processing is reduced, the line-of-sight direction can be estimated more quickly while improving accuracy.

Although various embodiments and experimental examples of the present invention have been described above, the present invention is not limited to the above-described various embodiments and experimental examples. For example, the calibration unit (calibration process) is not necessarily provided. For example, the eyeball center C as a general average value of a human (adult), that is, the translation vector _Te may be used. In the form in which the line-of-sight direction estimation device 30 includes the calibration unit 33, when the subject is the same, the calibration process by the calibration unit 33 may be performed once at the beginning. Moreover, although the initial value of the iris has been described as being calculated by the gaze direction estimation unit 32, for example, an initial value calculation unit of the iris may be provided separately. Furthermore, various illustrated embodiments and the like can be appropriately combined without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Gaze direction estimation system, 10 ... Camera, 11 ... RGB camera (two-dimensional image acquisition part), 12 ... Distance image acquisition part, 30 ... Gaze direction estimation apparatus, 31 ... Coordinate system setting part, 32 ... Gaze direction estimation part 33 ... proofreading part.

Claims (9)

A gaze direction estimation system that estimates a gaze direction of a subject to be photographed,
A camera that acquires a distance image including a two-dimensional image to be imaged and distance information to the imaged object;
A gaze direction estimation device that estimates the gaze direction of the subject to be photographed based on the two-dimensional image and the distance image of the face of the subject to be photographed by the camera;
With
The gaze direction estimating device is:
From the two-dimensional image of the face and the distance image, posture information of the head of the subject to be imaged with respect to the first coordinate system of the camera is calculated to set a second coordinate system, and the second coordinate system A third coordinate system of the eyeball, whose origin is the center of the eyeball of the subject to be photographed expressed in translation, and the origin of the third coordinate system is the origin, and the third coordinate system A coordinate system setting unit for setting a fourth coordinate system of the iris of the subject to be imaged represented by a first rotation angle that is a horizontal rotation angle with respect to and a second rotation angle that is a vertical rotation angle;
Based on the eyeball model, the iris contour when the three-dimensional contour of the iris in the fourth coordinate system is projected onto the image plane in the first coordinate system via the camera, The first and second rotation angles are calculated by fitting the outline of the iris extracted from the two-dimensional image of the face of the subject to be imaged, and the calculated first and second rotation angles And a line of sight that estimates the line-of-sight direction of the subject based on an optical axis calculated from the pupil center of the subject to be photographed determined based on the eyeball model and the origin of the third coordinate system A direction estimator;
Having
Gaze direction estimation system. The line-of-sight direction estimator associates each edge point when the three-dimensional contour of the iris in the fourth coordinate system associated with the fitting is projected onto the image plane, and the two-dimensional image Calculating the first and second rotation angles by solving an objective function that indicates a corresponding relationship with a corresponding edge point in the contour of the iris and includes the first and second rotation angles as unknown parameters;
The gaze direction estimation system according to claim 1. Based on the two-dimensional image and the distance image of the face of the subject to be photographed while the subject is facing the camera and gazing at the camera, the third coordinates A calibration unit for calibrating the origin of the system;
The gaze direction estimation system according to claim 1 or 2. A gaze direction estimation method for estimating a gaze direction of a subject to be photographed,
A photographing step of photographing the face of the subject to be photographed using a camera that obtains a two-dimensional image of the subject and a distance image including distance information to the subject;
The posture information of the head of the subject to be photographed with respect to the first coordinate system of the camera is calculated from the two-dimensional image and the distance image of the face obtained in the photographing step to set a second coordinate system In addition, the third coordinate system of the eyeball, whose origin is the center of the eyeball of the subject to be photographed, which is represented by translational movement of the second coordinate system, and the origin of the third coordinate system is the origin. And setting a fourth coordinate system of the subject's subject iris represented by a first rotation angle that is a horizontal rotation angle with respect to the third coordinate system and a second rotation angle that is a vertical rotation angle. A coordinate system setting process,
Based on the eyeball model, the iris contour when the three-dimensional contour of the iris in the fourth coordinate system is projected onto the image plane in the first coordinate system via the camera, The first and second rotation angles are calculated by fitting the outline of the iris extracted from the two-dimensional image of the face of the subject to be imaged, and the calculated first and second rotation angles And a line of sight that estimates the line-of-sight direction of the subject based on an optical axis calculated from the pupil center of the subject to be photographed determined based on the eyeball model and the origin of the third coordinate system A direction estimation step;
Comprising
Gaze direction estimation method. In the line-of-sight direction estimation step, each edge point when the three-dimensional outline of the iris in the fourth coordinate system associated with the fitting is projected onto the image plane, and the two-dimensional image Calculating the first and second rotation angles by solving an objective function that indicates a corresponding relationship with a corresponding edge point in the contour of the iris and includes the first and second rotation angles as unknown parameters;
The gaze direction estimation method according to claim 4. A calibration step of calibrating the origin of the third coordinate system based on the two-dimensional image of the face of the subject and the distance image;
When performing the calibration step, in the photographing step, the subject to be photographed is photographed while the subject is facing the camera and gazing at the camera.
The gaze direction estimation method according to claim 4 or 5. A gaze direction estimation program for estimating a gaze direction of a subject to be photographed,
On the computer,
The two-dimensional image of the face obtained by photographing the face of the subject to be photographed using a camera that obtains a two-dimensional image of the subject and a distance image including distance information to the subject. Expressed from the distance image is the posture information of the head of the subject to be photographed with respect to the first coordinate system of the camera to set the second coordinate system, and the second coordinate system is translated and moved. A third coordinate system of the eyeball whose origin is the center of the eyeball of the subject to be imaged, a first rotation angle that is a horizontal rotation angle with respect to the third coordinate system, and a first rotation angle that is a vertical rotation angle. A coordinate system setting step of setting a fourth coordinate system of the iris of the subject to be imaged represented by two rotation angles;
Based on the eyeball model, the iris contour when the three-dimensional contour of the iris in the fourth coordinate system is projected onto the image plane in the first coordinate system via the camera, The first and second rotation angles are calculated by fitting the outline of the iris extracted from the two-dimensional image of the face of the subject to be imaged, and the calculated first and second rotation angles And a line of sight that estimates the line-of-sight direction of the subject based on an optical axis calculated from the pupil center of the subject to be photographed determined based on the eyeball model and the origin of the third coordinate system A direction estimation step;
A gaze direction estimation program to be executed. In the line-of-sight direction estimation step, each edge point when the three-dimensional outline of the iris in the fourth coordinate system associated with the fitting is projected onto the image plane, and the two-dimensional image Calculating the first and second rotation angles by solving an objective function that indicates a corresponding relationship with a corresponding edge point in the contour of the iris and includes the first and second rotation angles as unknown parameters;
The gaze direction estimation program according to claim 7. In the computer,
The two-dimensional image of the face of the subject to be photographed obtained by photographing the subject to be photographed while the subject is facing the camera and gazing at the camera; Further executing a calibration step of calibrating the origin of the third coordinate system based on the distance image;
The gaze direction estimation program according to claim 7 or 8.