JP5873362B2 - Gaze error correction apparatus, program thereof, and method thereof - Google Patents

Description

  The present invention relates to a line-of-sight error correction apparatus that corrects a line-of-sight error measured by a pupil-corneal reflection method using a spherical eyeball model, a program thereof, and a method thereof.

  For example, when it is desired to examine an eye-catching object among objects displayed on the display screen, the line of sight is measured. Further, in a PC (personal computer), the line of sight is measured as a pointing device instead of a mouse. At this time, in order to accurately measure the line of sight in a natural state where the user's head is not restrained, the pupil-corneal reflection method is widely used (Non-Patent Document 1).

Hereinafter, the configuration of the line-of-sight measurement device 10 described in Non-Patent Document 1 will be described with reference to FIG.
As shown in FIG. 16, the line-of-sight measurement apparatus 10 measures the line of sight by the pupil-corneal reflection method, and includes a pupil / Purkinje image detection unit 11, a line-of-sight vector / reference point calculation unit 12, and a personal calibration unit. 13 and line-of-sight coordinate conversion / output means 14.
The line-of-sight measurement apparatus 10 requires an infrared light source 90, a photographing camera 91, and display means 92, which will be described later, in order to measure the line of sight.

The infrared light source 90 is, for example, an infrared LED (Light Emitting Diode) that irradiates the user's eyeball 900 (FIG. 17) with infrared rays.
The photographing camera 91 is, for example, an infrared CCD (Charge Coupled Device) that photographs the user's eyeball 900 in a state where the infrared light source 90 emits infrared rays.
Here, the photographic camera 91 uses internal parameters (for example, focal length, lens distortion, optical axis center position of the image, aspect ratio of CCD pixels) and external parameters (for example, by Tsai camera calibration (Non-Patent Document 2)). For example, the position and rotation angle) are obtained in advance.
The display unit 92 is, for example, a display that displays the position of the line of sight measured by the line-of-sight measurement device 10 on a display screen.

The pupil / Purkinje image detecting means 11 detects a Purkinje image (corneal reflection image) u and a pupil 902 from an eyeball image 93 obtained by photographing the eyeball 900 with the photographing camera 91 as shown in FIG. .
For example, the pupil 902 appears as a darker image in the eyeball image 93 than the iris 903. Accordingly, the pupil / Purkinje image detection means 11 segments the eyeball image 93 into regions of similar luminance, and detects pupil regions from the segmented regions. Then, as shown in FIG. 17 (b), the pupil / Purkinje image detecting means 11 performs elliptical approximation of the pupil contour on the detected pupil region by the sum of squared errors, and obtains the region center as the pupil center e. Further, the pupil / Purkinje image detecting means 11 detects the position u of the Purkinje image within a certain distance from the pupil center e. At this time, the Purkinje image is often detected as a point having no area.
In FIG. 17A, only the user's eyeball 900 is shown for the sake of simplicity.
In FIG. 17B, only the pupil 902 portion of the eyeball image 93 is shown.

The line-of-sight vector / reference point calculation means 12 calculates the line of sight (line-of-sight vector v) as shown in FIG. 18 using the position u of the Purkinje image detected by the pupil / Purkinje image detection means 11 and the pupil center e. It is.
That is, the line-of-sight vector / reference point calculation means 12 measures the eyeball position in the three-dimensional space, and based on the eyeball position, the line-of-sight vector v == the vector indicating the line of sight and the reference point (corneal curvature center c). ec is calculated.

Here, in FIG. 18, the corneal curvature center c, the vector d from the corneal curvature center c to the actual pupil contour, the unit vector n of the normal vector of the plane in contact with the corneal surface in the pupil contour image p, and the cornea The pupil contour image p on the surface, the position vector p ₀ of the pupil contour image, the observed pupil contour q, and the corrected pupil contour q ′ are illustrated.
In FIG. 18, the position vector p ₀ of the pupil contour image is shown as a point for easy viewing of the drawing.
In FIG. 18, the pupil contours q and q ′ are linear in order to show a cross section of the pupil contour that is circular when viewed from the front.

FIG. 18 shows a unit vector r of the direction vector from the photographing camera 91 to the position vector p ₀ of the pupil contour image. Here, the upper unit vector r is illustrated to explain that the unit vector r is obtained from the position vector p ₀ of the pupil contour image. The lower unit vector r is illustrated by moving the upper unit vector r so as to point to the position u of the Purkinje image.

FIG. 18 shows a unit vector t of light rays from the camera 91 after refraction in the pupil contour image p, a corneal curvature radius C, a distance D from the corneal curvature center c to the pupil contour, and a pupil contour image. The distance M from the position vector p ₀ to the position u of the Purkinje image and the radius R of the pupil are illustrated. These items will be described later if necessary.

  Here, in the pupil-corneal reflection method, the line of sight is estimated from the positional relationship between the pupil center e and the position u of the Purkinje image. At this time, the positional relationship between the infrared light source 90 and the photographing camera 91 is known because it is measured in advance. Therefore, the position of the eyeball 900 can be derived by camera calibration.

  That is, the two-dimensional coordinates (x, y) of the position u of the Purkinje image and the pupil center e can be obtained in the camera coordinate system. Further, the depth coordinate z can be obtained with high accuracy from the focal length of the photographing camera 91 and the parallax of the stereo image. Therefore, as shown in FIG. 18, the position vectors of the corneal curvature center c and the pupil center e can be obtained in the world coordinate system.

As shown in FIG. 19, the personal calibration unit 13 performs correction by personal calibration on the line-of-sight vector v calculated by the line-of-sight vector / reference point calculation unit 12.
The distance L and the corneal curvature radius C in FIG. 19 are constants with individual differences. Therefore, the personal calibration means 13 performs personal calibration to obtain the distance L from the corneal curvature center c to the pupil center e and the corneal curvature radius C.
In FIG. 19, a circle having a corneal curvature radius C with the corneal curvature center c as a center is illustrated by a broken line.

  Here, in the personal calibration, as shown in FIG. 20, a spherical eyeball model 910 that assumes the shape of a human eyeball 900 is used. FIG. 20 shows a spherical eyeball model 910 in which the surface of the cornea 901 is divided at an equal angle in a polar coordinate system with the cornea curvature center c as a reference for a part of the cornea 901 in order to make the drawing easy to see. That is, in the spherical eyeball model 910, the shape of the cornea 901 is spherical.

  In the personal calibration, the distance L and the corneal curvature radius C are obtained using the spherical eyeball model 910. Specifically, in personal calibration, the measurement data when the line of sight is directed to the reference marker on the display screen is approximated by applying the least square method to an eyeball model represented by a quadratic or higher-order curved surface. To do. Here, Non-Patent Document 1 describes that the error increases even when approximation of the third order or higher is performed. Therefore, in personal calibration, a spherical surface model composed of an eyeball and a cornea is created by combining secondary curved surfaces, that is, spherical surfaces, and used for correcting the line-of-sight vector v.

The line-of-sight coordinate conversion / output unit 14 converts the line-of-sight vector v corrected by the personal calibration unit 13 into an arbitrary coordinate system and outputs it.
Here, in the pupil-cornea reflection method, as shown in FIG. 21, the world coordinate system (x ′, y ′, z ′), the camera coordinate system (x, y, z), and the image coordinate system (x _0). , Y ₀ ).
This world coordinate system is a three-dimensional coordinate system fixed with respect to the display means 92. For example, the lower part of the display screen of the display means 92 is the origin.
The camera coordinate system is, for example, a three-dimensional coordinate system having the optical principal point of the photographing camera 91 as the origin.
The image coordinate system is a two-dimensional coordinate system that points to the display screen (that is, the eyeball image 93) of the display unit 92. For example, the image coordinate system is a coordinate system in which the depth position of the world coordinate system is zero.
In FIG. 21, in the display unit 92, the intersection of the extension line of the line-of-sight vector v from the eyeball 900 and the display screen of the display unit 92 is shown as a point 92a.

Then, the position vector p ₀ in the image coordinate system, the position vector p ′ in the world coordinate system, and the position vector p in the camera coordinate system are converted into the following using transformation matrices Rw and Rc obtained by camera calibration of Tsai: Coordinate conversion can be performed as in equation (1) (Non-Patent Document 3).
That is, the line-of-sight coordinate conversion / output unit 14 can freely perform coordinate conversion among the world coordinate system, the camera coordinate system, and the image coordinate system according to the equation (1). In Equation (1), T is a transposed matrix.

"Gaze measurement system that achieves simple calibration by two-point correction", Ohno et al., IPSJ Journal, 2003, Vol.44, No.4 “A versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision”, Tsai, R.Y., IEEE Journal on Robotics and Automation (1987), Volume 3, Issue 4, P323-344 "Gaze measurement system based on eyeball shape model-Toward the realization of gaze input device", Ohno, Jishokenho 2001-HI-93, 2001, pp.47-54

However, the technique described in Non-Patent Document 1 has a problem that the error of the measured line of sight is large.
For example, consider a case where a pointing operation is performed on a PC using a measured line of sight. In this case, if there is even an error in the line of sight, the user will feel uncomfortable when specifying an icon on the display screen. In other words, in the pointing by the line of sight, if there is even a slight error, it is impossible in principle to capture the pointer at the center of the line of sight, and it is difficult to correct this error.
In addition, display screens of televisions and PCs have been widened in view of realism and high operability. When line-of-sight measurement is performed on a display screen with a wide field of view, it is necessary to reduce the error even in the periphery.

  Therefore, in order to reduce the line-of-sight error, the conventional pupil-corneal reflection method often performs personal calibration many times. Therefore, the labor required for the personal calibration work becomes a heavy burden when performing the line-of-sight measurement. Furthermore, it is very difficult to determine how accurately the line of sight can be measured once personal calibration is completed. In other words, when the measured line of sight is analyzed later, the result of personal calibration before measurement must be trusted as unique and correct data. For this reason, the personal calibration work has to be performed carefully, which causes a large work load for eye gaze measurement.

  Here, it is considered that the reason why the personal calibration is not completed once is that the gaze error cannot be made zero at both the central portion and the peripheral portion of the display screen because the spherical eyeball model having the cornea as a spherical surface is used. As shown in FIG. 22A, for example, a cornea 901, a pupil 902, and an iris 903 are photographed in the eyeball image 93. As shown in FIG. 22B, the eyeball 900 includes a cornea 901, a pupil 902, an iris 903, a sclera 904, an aqueous humor 905, a lens 906, a retina 907, and a fovea 908. Consists of. The curvature of the cornea 901 is not uniform, and there are many people whose radius of curvature increases toward the periphery. Further, the curvature of the cornea 901 is not uniform even for the same individual. For this reason, it is extremely difficult to generate an eyeball model that is optimized for all people so that personal calibration is performed only once.

  Therefore, an object of the present invention is to provide a gaze error correction apparatus, a program for the gaze error correction apparatus, and a method for reducing the gaze error with a small work load.

  The present inventor noted that in the above-described personal calibration, an error due to the difference between the sphericality of the spherical eyeball model and the asphericity of the corneal shape occurs, and this error remains even after coordinate conversion. The present inventors have found that the above-described problems can be solved by generating an aspheric eyeball model for correcting a line-of-sight error after coordinate conversion.

  That is, the line-of-sight error correction apparatus according to the first invention of the present application is a line-of-sight error correction apparatus that corrects a line-of-sight error measured by a pupil-corneal reflection method using a spherical eyeball model, including a display control unit and an error information input unit. And a distortion vector calculation means, an aspheric eyeball model generation means, and a line-of-sight error correction means.

  According to this configuration, the line-of-sight error correction device displays, by the display control unit, a marker image in which a marker is arranged at a predetermined position and a position of the marker line of sight with respect to the marker measured by the pupil-corneal reflection method. To display. Here, the marker line of sight is obtained by measuring the line of sight when the user gazes at the marker in the image coordinate system by the pupil-corneal reflection method.

  Also, in the line-of-sight error correction device, error information indicating the positional deviation between the marker displayed on the display unit and the marker line of sight is input by the error information input unit. The error information input to the error information input means indicates a positional deviation between the line of sight measured by the pupil-cornea reflection method and the actual line of sight when the user gazes at the marker.

  Further, the line-of-sight error correction device calculates the position of the marker line of sight from the planar image showing the position of the marker line of sight with the same size as the marker image based on the error information input to the error information input unit by the distortion vector calculation unit. A distortion plane distorted so as to coincide with the marker is generated, and a distortion vector indicating the distortion of the distortion plane is calculated in the image coordinate system. The distortion vector calculated by the distortion vector calculating means indicates the amount of displacement and the direction of displacement of the corresponding point between the planar image and the distortion plane.

  In addition, the gaze error correction device uses the aspheric eyeball model generation means to correct the gaze error caused by the spherical spherical eyeball model based on the distortion vector, so that the aspheric eyeball model showing an aspheric corneal shape is provided. Is generated. Then, the line-of-sight error correction device corrects the line-of-sight error measured by the pupil-corneal reflection method by the rotation based on the aspheric eyeball model generated by the aspheric eyeball model generation unit. Thus, the line-of-sight error correction apparatus corrects the line-of-sight error using the aspheric eyeball model generated after coordinate conversion.

In the gaze error correction device according to the second invention of the present application, the aspheric eyeball model generation means performs coordinate conversion of the distortion vector of the image coordinate system into an eyeball center polar coordinate system with the eyeball center as the origin as an aspheric eyeball model. Thus, a difference angle eyeball model generation unit that generates a difference angle eyeball model indicating a difference angle of the line of sight with respect to the spherical eyeball model, the line of sight error correction unit refers to the difference angle eyeball model, and A difference angle eyeball model line-of-sight error correcting means is provided for correcting a line-of-sight error by rotating the line of sight measured by the pupil-corneal reflection method around the origin by a difference angle.
According to such a configuration, the line-of-sight error correction apparatus can generate a difference angle eyeball model by simple coordinate conversion.

Further, the gaze error correction device according to the third invention of the present application is such that the aspherical eyeball model generation means has a photographing camera position l for measuring the gaze measured by the pupil-corneal reflection method, and the rotation angle θ of the eyeball. The position of the Purkinje image u ′ _θ on the cornea surface, the position of the pupil center (x _θ , y _θ , z _θ ), and the virtual position l ′ of the photographing camera indicated by the distortion vector based on the position l of the photographing camera in pupil center polar coordinate system with the origin of the pupil center, the following (3) normal vectors eyeball model generating unit normal vector n _theta generates normal vector eyeball model defined as a non-spherical eye model equation, The line-of-sight error correction means refers to the normal vector eyeball model, calculates a rotation angle at which the normal vector at the position of the Purkinje image faces the photographing camera, and calculates the normal vector eyeball model by the calculated rotation angle. Rotated The normal vector eyeball model after rotation includes a normal vector eyeball model line-of-sight error correction means for outputting a vector directed from the corneal curvature center to the pupil center as the line of sight in which the error is corrected. .

  Here, the pupil-corneal reflection method measures the line of sight with reference to the center of the pupil. The normal vector eyeball model is a pupil center polar coordinate system with the pupil center as a reference (origin). In this way, the gaze error correction device can make the reference positions of the pupil-corneal reflection method and the normal vector eyeball model the same, and therefore can correct gaze errors even when the head moves during gaze measurement. it can.

The line-of-sight error correction apparatus according to the fourth invention of the present application generates a lookup table by associating the distortion vector calculated by the distortion vector calculation means with a region having a size set in advance in the image coordinate system. A table generation means, wherein the aspheric eyeball model generation means generates an aspheric eyeball model with reference to the lookup table generated by the lookup table generation means.
According to such a configuration, the line-of-sight error correction device does not need to perform interpolation calculation each time by referring to the lookup table.

  The line-of-sight error correction program according to the fifth invention of the present application is a hardware such as a CPU, memory, and hard disk provided in a general computer in order to correct the line-of-sight error measured by the pupil-corneal reflection method using a spherical eyeball model. The hardware resource functions as display control means, error information input means, distortion vector calculation means, aspheric eyeball model generation means, and line-of-sight error correction means. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

  A line-of-sight error correction method according to the sixth invention of the present application is a line-of-sight error correction method for a line-of-sight error correction apparatus that corrects a line-of-sight error measured by a pupil-corneal reflection method using a spherical eyeball model. A marker line-of-sight input step, a marker line-of-sight display step, an error information input step, a distortion vector calculation step, an aspheric eyeball model generation step, and a line-of-sight error correction step.

According to the present invention, the following excellent effects can be obtained.
According to the first, fifth, and sixth inventions of the present application, it is not necessary to repeatedly perform personal calibration in order to correct the error of the line of sight measured by the pupil-corneal reflection method using the aspheric eyeball model generated after the coordinate conversion. , Work load can be reduced. Furthermore, according to the first, fifth, and sixth inventions of the present application, since an aspheric eyeball model can be used to correct errors caused by the difference between the sphericality of the spherical eyeball model and the asphericity of the corneal shape, Can be kept small

According to the second invention of the present application, a difference angle eyeball model can be generated by simple coordinate conversion.
According to the third invention of the present application, since the reference positions of the pupil-corneal reflection method and the normal vector eyeball model are the same, even when the head moves during the gaze measurement, the gaze error can be corrected.
According to the fourth invention of the present application, by referring to the lookup table, it is not necessary to perform the interpolation calculation each time, and the calculation process can be speeded up.

It is a block diagram which shows the structure of the gaze error correction apparatus which concerns on embodiment of this invention. It is a figure explaining the display of a marker and a pointer in the gaze error correction device of FIG. FIG. 3 is a diagram for explaining error information input in the line-of-sight error correction apparatus of FIG. 1. FIGS. 2A and 2B are diagrams for explaining generation of a distortion plane in the line-of-sight error correction apparatus of FIG. 1, in which FIG. 1A shows before the plane is distorted, and FIG. It is a figure explaining the coordinate conversion to a polar coordinate system in the gaze error correction device of FIG. FIG. 2 is a diagram for explaining generation of a differential angle eyeball model in the gaze error correction device of FIG. 1. FIG. 3 is a diagram for explaining correction by a differential angle eyeball model in the gaze error correction device of FIG. 1. FIG. 2 is a first diagram illustrating generation of a normal vector eyeball model in the gaze error correction device of FIG. 1. FIG. 3 is a second diagram illustrating generation of a normal vector eyeball model in the gaze error correction device of FIG. 1. FIG. 4 is a third diagram illustrating generation of a normal vector eyeball model in the gaze error correction device of FIG. 1. FIG. 2 is a first diagram illustrating correction by a normal vector eyeball model in the gaze error correction apparatus of FIG. 1. FIG. 5 is a second diagram illustrating correction by a normal vector eyeball model in the gaze error correction apparatus of FIG. 1. FIG. 4 is a third diagram for explaining correction by a normal vector eyeball model in the gaze error correction apparatus of FIG. 1. It is a flowchart which shows operation | movement of the gaze error correction apparatus of FIG. It is a block diagram which shows the structure of the gaze error correction apparatus which concerns on the modification of this invention. It is a block diagram which shows the structure of the conventional gaze measuring apparatus. It is a figure explaining the conventional pupil-corneal reflection method, (a) shows arrangement | positioning of an imaging camera and an infrared light source, (b) shows the positional relationship of a pupil center and a Purkinje image. It is a figure explaining calculation of a look vector in the conventional pupil-corneal reflection method. It is a figure explaining the conventional personal calibration. It is a figure explaining the spherical eyeball model used with the conventional gaze measuring device. It is a figure explaining the coordinate transformation in the conventional gaze measuring device. It is a figure explaining the structure of an eyeball, (a) is an example of the eyeball image which image | photographed the eyeball, (b) shows the cross section of an eyeball.

(Embodiment)
[Configuration of eye gaze measurement and error correction device]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In addition, the same code | symbol was attached | subjected to the means which has the same function, and description was abbreviate | omitted.
With reference to FIG. 1, the configuration of a line-of-sight measurement / error correction apparatus 1 according to an embodiment of the present invention will be described.
As shown in FIG. 1, the line-of-sight measurement / error correction apparatus 1 measures a line of sight by a pupil-corneal reflection method using a spherical eyeball model, and corrects an error of the measured line of sight. And a line-of-sight error correction device 20.

  The line-of-sight measurement device 10A measures a line of sight (line-of-sight vector) of a user (not shown) by a pupil-corneal reflection method, and includes a pupil / Purkinje image detection unit 11, a line-of-sight vector / reference point calculation unit 12, a line of sight Coordinate conversion / output means 14.

  Here, when an aspheric eyeball model is generated by the line-of-sight error correction apparatus 20 (hereinafter referred to as “model generation”), the line-of-sight measurement apparatus 10A uses the line of sight (marker) when the user is gazing at the marker Mk (FIG. 2). Gaze) is measured. Then, the line-of-sight measurement device 10A converts the marker line of sight into the image coordinate system, and outputs it to the marker line-of-sight input means 21a.

  On the other hand, when correcting the line-of-sight error by the line-of-sight error correction apparatus 20 using the aspheric eyeball model (hereinafter referred to as “line-of-sight error correction”), the line-of-sight measurement apparatus 10A allows the user to select an arbitrary object (for example, in the display screen). Measure the line of sight while gazing at the icon. Then, the line-of-sight measurement device 10 </ b> A performs coordinate conversion of the line of sight to a polar coordinate system to be described later, and outputs it to the line-of-sight error correction means 24.

The line-of-sight measurement / error correction apparatus 1 can be manually switched by the user to perform either model generation or line-of-sight error correction.
The line-of-sight measurement apparatus 10A is the same as the line-of-sight measurement apparatus 10 of FIG. 16 except that it does not include the personal calibration unit 13 of FIG.

[Configuration of eye gaze error correction device]
The line-of-sight error correction apparatus 20 corrects the line-of-sight error measured by the line-of-sight measurement apparatus 10A. The line-of-sight error correction apparatus 20 corrects the line-of-sight error, input / output control means 21, distortion vector calculation means 22, aspheric eyeball model generation means 23, And correction means 24.
The input / output control unit 21, the strain vector calculation unit 22, and the aspheric eyeball model generation unit 23 are units used for model generation. On the other hand, the line-of-sight error correction unit 24 is a unit used for line-of-sight error correction.

  The input / output control unit 21 inputs a marker line of sight and error information and outputs each image such as a marker image at the time of model generation. The marker line of sight input unit 21a, the display control unit 21b, and error information Input means 21c.

The marker line-of-sight input unit 21a receives a marker line of sight from the line-of-sight measurement device 10A and outputs the input marker line of sight to the display control unit 21b.
The display control unit 21b causes the display unit 92 to display a marker image to be described later and a pointer Pt (FIG. 2) indicating the position of the marker line of sight input from the marker line-of-sight input unit 21a.
The error information input means 21 c is for inputting error information from the outside (for example, a user) and outputting the input error information to the distortion vector calculation means 22.

<Display of marker and pointer, input of error information>
With reference to FIGS. 2 and 3, the display of markers and pointers and the input of error information will be described in detail (see FIG. 1 as appropriate).
As shown in FIG. 2, at the time of model generation, the display control unit 21 b displays the marker image 94 on the display unit 92. The marker image 94 is an image in which a marker Mk is arranged at a predetermined position. In the example of FIG. 2, the marker image 94 has nine markers Mk _{1 to} Mk ₉ arranged in a matrix. The marker image 94 has the same size as the display screen of the display unit 92 or a smaller size for the purpose of generating a distortion plane described later.

Here, the user gazes at any of the markers Mk of the marker image 94 displayed on the display unit 92. Then, the marker line-of-sight input unit 21a receives a marker line of sight that is a line of sight with respect to the marker Mk from the line-of-sight measurement device 10A, and outputs the input marker line of sight to the display control unit 21b. Then, the display control unit 21b causes the display unit 92 to display a pointer Pt indicating the position of the marker line of sight. In the example of FIG. 2, the pointer Pt for the marker Mk ₅ is displayed on the display means 92.

Further, the user confirms the positional deviation between the marker Mk and the pointer Pt displayed on the display unit 92, operates the mouse (not shown), and inputs this positional deviation (error information) to the error information input unit 21c. To do. In the example of FIG. 3, the pointer Pt when the user gazes the marker Mk ₅ is located at the lower left of the marker Mk _5. Thus, the user, the center of the pointer Pt is to match the marker Mk _5, drag the pointer Pt in mice. That is, the pointer Pt moves to the position of the pointer Pt ′ by the user's operation. In this example, the error information indicates a positional deviation of the pointer Pt with respect to the marker Mk ₅ and can be represented by coordinate information of the marker Mk, the pointer Pt, and the pointer Pt ′.

The display control unit 21b displays the pointer Pt on the display unit 92 in real time when the line-of-sight error is corrected. Further, the display control unit 21b may display the pointer Pt on the display unit 92 in order to confirm whether or not the error correction has been accurately performed even after the line-of-sight error correction.
Further, the user gazes at all the markers Mk as in the case of the marker Mk ₅ and inputs error information for all the markers Mk.

<Calculation of distortion vector>
Hereinafter, the distortion vector calculation means 22 of FIG. 1 will be described in detail with reference to FIG. 4 (see FIG. 1 as appropriate).
At the time of model generation, the distortion vector calculation unit 22 uses the error information input from the error information input unit 21c, the pointer Pt to the marker Mk from the planar image 95 showing the position of the pointer Pt with the same size as the marker image 94. A distortion plane 95 ′ matched with the above is generated, and a distortion vector is calculated.

First, the generation of the distortion plane 95 ′ by the distortion vector calculation means 22 will be described in detail.
As shown in FIG. 4A, the distortion vector calculation means 22 generates a flat image 95 based on the coordinate information of the error information pointer Pt. In the example of FIG. 4A, the planar image 95 is the same size as the marker image 94 of FIGS. 2 and 3, and indicates the position of the pointer Pt for each of the nine markers Mk. Further, in FIG. 4A, the marker image 94 and the planar image 95 are shown in an overlapping manner for easy understanding.

  Next, as shown in FIG. 4B, the distortion vector calculation means 22 uses the error information marker Mk and the coordinate information of the pointer Pt so that the nine pointers Pt coincide with the corresponding markers Mk. The plane image 95 is distorted to generate a distorted plane 95 ′. This distortion plane 95 'can be represented as a curved surface composed of a plurality of curves. For example, the distortion vector calculation means 22 generates the distortion plane 95 'by a curved surface constituted by a Bezier curve or a B-spline curve. Further, the strain vector calculation means 22 is a NURBS (Non-Uniform Rational B-Spline) curved surface, a curved surface obtained by distorting a UV (XY coordinate on the curved surface) plane, or a pseudo curve using a polygon (polygon). A strain plane 95 ′ may be generated.

In the present embodiment, the distortion vector calculation means 22 generates a distortion plane 95 ′ by a NURBS curved surface. Further, the strain vector calculation unit 22 may display the generated strain plane 95 ′ on the display unit 92 via the display control unit 21b. Further, the strain vector calculation unit 22 may allow each user to deform the strain plane 95 ′ displayed on the display unit 92.
The marker Mk is fixed with respect to the display screen. On the other hand, the pointer Pt is distorted as shown in FIG. 4B because it is displayed by texture mapping on the distortion plane 95 ′.

  This NURBS curved surface is a curved surface having a plurality of control points (CV control points). This control point can be arranged at an arbitrary position in the display screen. In addition, the NURBS curved surface is determined by what order equation the curved surface is represented by the value of the degree.

In the NURBS curved surface, each marker Mk is linked to 1 to 4 control points. The control point linked to the marker Mk closer than the fixed distance from the pointer Pt has the control right. This control right is added only to control points that can be controlled (moved) by the user. That is, the user can control only the control points linked to one marker Mk. The control point having the control right can be moved on the distortion plane 95 ′ by a drag-and-drop operation. The NURBS curved surface is deformed in accordance with the movement of the control point. Further, it is assumed that the control right is not added to other control points from the control point being dragged. Furthermore, if the number of control points is increased, when the pointer Pt located at the periphery of the display screen is moved, the situation where the other pointers Pt do not coincide with the marker Mk can be solved.
The NURBS curved surface is described in detail, for example, in a reference document ““ Maya Textbook 1—Basics of Modeling & Texture Setting ”, Kawakami, 2009”.

Next, the calculation of the distortion vector by the distortion vector calculation means 22 will be described in detail.
The distortion vector calculation means 22 calculates a distortion vector indicating the distortion of the distortion plane 95 ′ in the image coordinate system. Here, the distortion vector calculation means 22 obtains the positional deviation of the corresponding point, which is the corresponding point (pixel), between the plane image 95 and the distortion plane 95 ′. And the distortion vector calculation means 22 calculates the vector which shows the positional offset amount and positional offset direction of a corresponding point as a distortion vector in an image coordinate system.

  At this time, if the distortion vector calculation means 22 calculates a distortion vector from arbitrary corresponding points, such as calculating a distortion vector from all corresponding points, or calculating a distortion vector from corresponding points at the position of the marker Mk. Good. In the present embodiment, the distortion vector is calculated from the corresponding point at the position of the marker Mk. Thereafter, the strain vector calculation unit 22 outputs the calculated strain vector to the aspheric eyeball model generation unit 23.

Returning to FIG. 1, the description of the configuration of the line-of-sight error correction device 20 will be continued.
The aspheric eyeball model generation unit 23 generates an aspheric eyeball model corresponding to each user based on the strain vector input from the strain vector calculation unit 22 during model generation.
This aspheric eyeball model is an eyeball model showing an aspherical corneal shape, and there are two types, a differential angle eyeball model and a normal vector eyeball model. Therefore, the aspheric eyeball model generation means 23 includes a difference angle eyeball model generation means 23a and a normal vector eyeball model generation means 23b in order to generate each aspheric eyeball model.
Details of the difference angle eyeball model generation unit 23a and the normal vector eyeball model generation unit 23b will be described later.

The line-of-sight error correction means 24 corrects the line-of-sight error by rotation based on the aspheric eyeball model generated by the aspheric eyeball model generation means 23 at the time of line-of-sight error correction.
The line-of-sight error correction unit 24 includes a difference angle eyeball model line-of-sight error correction unit 24a and a normal vector eyeball model line-of-sight error correction unit 24b because the line-of-sight correction method differs between the two types of aspheric eyeball models. .

  First, after explaining the coordinate transformation to the polar coordinate system that is the premise of the aspheric eyeball model, the generation of the difference angle eyeball model, the correction by the difference angle eyeball model, the generation of the normal vector eyeball model, and the normal vector eyeball The correction by the model will be described in order.

<Coordinate conversion to polar coordinate system>
Hereinafter, coordinate conversion to the polar coordinate system will be described in detail with reference to FIG.
The difference angle eyeball model is represented by an eyeball center polar coordinate system with the eyeball center as the origin. Therefore, the line-of-sight error correction device 20 needs to perform coordinate conversion to the eyeball central polar coordinate system.

As shown in FIG. 5, it is assumed that the positions of the points α and β are known in the space. For example, the point α is the position (optical principal point) of the photographing camera 91 and is located at (x ′, y ′) in the world coordinate system. Further, the point β is the eyeball center, that is, the origin (x ₀ , y ₀ ) of the eyeball center polar coordinate system. Further, the distance between the points α and β is z ′.
In FIG. 5, the rotation about two axes is shown in a single drawing in order to avoid duplication of drawings.

At this time, for example, the x-axis rotation angle θ and the y-axis rotation angle φ necessary for the eyeball 900 facing the point α to face the target γ are defined by the following equation (2). In the equation (2), f _i-w can be obtained by Tsai camera calibration. That is, if the points α and β are known in the space, using the equations (1) and (2), the world coordinate system, the camera coordinate system, the image coordinate system, and the eyeball central polar coordinate system Coordinate conversion can be performed freely between the two. Similarly, if the point β is the pupil center, the equations (1) and (2) can be applied to the pupil center polar coordinate system.

<Generation of differential angle eyeball model>
Hereinafter, the generation of the difference angle eyeball model by the difference angle eyeball model generation unit 23a will be described in detail (refer to FIG. 1 as appropriate).
As shown in FIG. 6, the difference angle eyeball model generation unit 23 a converts the distortion vector of the image coordinate system input from the strain vector calculation unit 22 into the eyeball central polar coordinate system, thereby converting the difference angle eyeball model 96. Is to be generated.

  Here, the difference angle eyeball model generation unit 23a generates a difference angle eyeball model from the strain vector at the position of the marker Mk (FIG. 2). In FIG. 6, it is assumed that there are five markers Mk (distortion vectors) for the sake of simplicity. Further, in FIG. 6, the difference angle eyeball model 96 is illustrated for a part of the cornea 901 in order to make the drawing easy to see.

  First, the difference angle eyeball model generation unit 23a divides the spherical surface into regions with the center of the corneal surface as a pole, similarly to the spherical eyeball model 910 (FIG. 20). The size of this area is arbitrarily set in advance. The accuracy of the differential angle eyeball model 96 is proportional to the size of the region, and improves as the region becomes smaller.

  Further, the difference angle eyeball model generation unit 23a obtains an area corresponding to the position of the marker Mk. That is, the difference angle eyeball model generation unit 23a performs coordinate conversion of the coordinates of the marker Mk represented in the image coordinate system into the eyeball central polar coordinate system, and obtains five regions including the marker Mk after coordinate conversion. Then, the difference angle eyeball model generation unit 23a performs mapping for associating the difference angle vector described later to these five regions.

  Further, the difference angle eyeball model generation means 23a obtains the difference angle vector 96a by converting the distortion vector at the position of the marker Mk into the eyeball central polar coordinate system. That is, the difference angle eyeball model generation unit 23a performs coordinate conversion from the image coordinate system to the eyeball central polar coordinate system for the five distortion vectors. Then, the difference angle eyeball model generation unit 23a obtains a vector connecting the tip position and the end position after coordinate conversion as a difference angle vector 96a in the eyeball central polar coordinate system. Further, as shown in FIG. 6, the difference angle eyeball model generation unit 23a performs mapping that associates the five difference angle vectors 96a with the five regions.

  Thereafter, the difference angle eyeball model generation unit 23a interpolates the difference angle vector 96a by an interpolation method such as linear interpolation for an empty region where the difference angle vector 96a does not exist (not shown). In this way, the difference angle eyeball model generation unit 23a generates the difference angle eyeball model 96 in which the difference angle vector 96a is associated with all regions.

The difference angle vector 96a indicates the difference angle (corrected rotation angle) of the line of sight with respect to the spherical eyeball model 910 (FIG. 20).
In other words, the difference angle eyeball model 96 represents an error caused by a difference between the sphericality of the spherical eyeball model 910 and the asphericity of the cornea 901 by a difference angle vector 96a for each region.

<Correction by differential angle eyeball model>
Hereinafter, with reference to FIG. 7, the correction by the difference angle eyeball model performed by the difference angle eyeball model line-of-sight error correction unit 24a will be described in detail (see FIG. 1 as appropriate).
As shown in FIG. 7, the difference angle eyeball model line-of-sight error correction means 24a refers to the difference angle eyeball model 96 (FIG. 6), and the pupil-cornea with the origin (eyeball center b) of the eyeball center polar coordinate system as the center. The line-of-sight error is corrected by rotating the line-of-sight measured by the reflection method by the difference angle.

In FIG. 7, for ease of explanation, the difference angle (Δθ, Δφ) in the two-dimensional direction is illustrated as the difference angle Δφ in the one-dimensional direction, and the infrared light source 90 and the photographing camera 91 are at the same position. did.
In FIG. 7, the point (pixel) P of the planar image 95 and the point (pixel) P ′ of the distortion plane 95 ′ are corresponding points (corresponding pixels). That is, the distortion vector is PP ′.

  Here, the differential angle eyeball model line-of-sight error correction means 24a receives the line of sight (line-of-sight vector) when the user is gazing at an arbitrary object during line-of-sight error correction. Further, the difference angle eyeball model line-of-sight error correction means 24a obtains which region (FIG. 6) the line-of-sight vector v of coordinate transformation is included in the difference angle eyeball model 96. Then, the difference angle eyeball model line-of-sight error correction unit 24 a acquires the difference angle Δφ of the region including the line-of-sight vector v from the difference angle eyeball model 96.

  Then, the difference angle eyeball model line-of-sight error correction unit 24a rotates the line-of-sight vector v by the difference angle Δφ around the eyeball center b to obtain a line-of-sight vector v ′ in which the error is corrected. In other words, the differential angle eyeball model line-of-sight error correction unit 24a rotates the cornea 901 to the cornea 901 ′, and uses the line-of-sight vector v ′ obtained at the position u ′ of the Purkinje image as the line-of-sight vector v ′ obtained at the position u ′ of the Purkinje image. To correct. That is, in FIG. 7, as the cornea 901 rotates, the positions of the infrared light source 90, the photographing camera 91, and the corneal curvature center c move to the positions of the infrared light source 90 ′, the photographing camera 91 ′, and the corneal curvature center c ′. become.

Thereafter, the differential angle eyeball model line-of-sight error correction unit 24a outputs a line of sight (line-of-sight vector v ′) with the error corrected. In the present embodiment, the differential angle eyeball model line-of-sight error correction unit 24 a outputs the line of sight with the corrected error (shown as “corrected line of sight”) to the display control unit 21 b and displays the line of sight on the display unit 92. Let
The difference angle eyeball model line-of-sight error correction unit 24a can output the line of sight in an arbitrary coordinate system to various output destinations according to the method of using the line of sight in which the error is corrected.

<Generation of normal vector eyeball model>
Hereinafter, generation of a normal vector eyeball model by the normal vector eyeball model generation unit 23b will be described with reference to FIGS. 8 to 10 (see FIG. 1 as appropriate).
As shown in FIG. 8, the normal vectors eyeball model generating unit 23b, in the pupil center polar coordinate system, the following (3) normal normal vector n _theta is defined by the formula vector eyeball model 97 (FIG. 9) Is to be generated.

In this equation (3), the position 1 of the photographing camera 91 (FIG. 1), the rotation angle θ of the eyeball, the position u ′ _θ of the Purkinje image on the cornea surface, and the position of the pupil center e (x _θ , y _θ , Z _θ ) and the virtual camera position l ′. In the expression (3), '|| ||' is a norm.
Further, the virtual position l ′ of the photographing camera 91 is a position indicated by a distortion vector with the position l of the photographing camera 91 as a base point when the cornea is an ideal spherical surface. In other words, the distortion vector is a vector connecting the position 1 of the photographing camera 91 to the virtual position l ′.
Further, the normal vector n _theta, a vector connecting to the position l of the imaging camera 91 from the position u _'theta of Purkinje image.

In FIG. 8, for ease of explanation, the rotation angle (θ, φ) of the eyeball in the two-dimensional direction is illustrated as the rotation angle θ in the one-dimensional direction. Here, the rotation angle θ of the eyeball and the position (x _θ , y _θ , z _θ ) of the pupil center e are obtained in advance by Tsai camera calibration.
FIG. 8 shows a circle having a radius from the pupil center e to the position u ′ _θ of the Purkinje image centered on the pupil center e. At this time, the normal vector n _θ is not perpendicular to the tangent to the spherical surface at the position u ′ _θ of the Purkinje image because there is an error in the spherical eyeball model with _respect to the aspherical cornea shape. Other words, the normal vector n _theta, indicating the normal to the surface of the aspheric cornea.

In other words, the Purkinje image to be photographed at the position l ′ of the virtual camera has a photographing camera 91 (real camera) due to an error caused by the difference between the sphericality of the spherical eyeball model and the asphericity of the corneal shape. That is, the image is taken at position l. In order to correct this error, a normal vector n _θ may be defined at the position u ′ _θ as shown in FIG.

As shown in FIG. 8, the normal vector n _θ = l−u ′ _θ . Further, the position l of the photographing camera is obtained by obtaining the position in the world coordinate system by Tsai camera calibration, and coordinate-converted into the pupil center polar coordinate system. Accordingly, the position u ′ _θ of the Purkinje image may be obtained in order to calculate the normal vector n _θ . Further, the position u ′ _θ of the Purkinje image is on a straight line connecting the position l ′ of the virtual camera and the position (x _θ , y _θ , z _θ ) of the pupil center e in the pupil center polar coordinate system. Further, the position u ′ _θ of the Purkinje image is set to 1 because the distance from the position (x _θ , y _θ , z _θ ) of the pupil center e is constant. Then, the position u ′ _θ of the Purkinje image can be defined by the above equation (3).

Here, as shown in FIG. 9, the normal vector eyeball model generation unit 23b divides the spherical surface into regions, with the center of the cornea surface serving as a pole, like the difference angle eyeball model generation unit 23a. Then, the normal vector eyeball model generation unit 23b obtains the normal vector n (n _θ ) while changing the rotation angle θ of the eyeball, and performs mapping corresponding to each region. In the example of FIG. 9, the normal vector eyeball model generation unit 23b obtains five normal vectors n for the five regions corresponding to the positions of the markers Mk (FIG. 2).

  Thereafter, the normal vector eyeball model generation unit 23b interpolates the normal vector n by an interpolation method such as linear interpolation for an empty area where the normal vector n does not exist (not shown). In this way, the normal vector eyeball model generation unit 23b generates the normal vector eyeball model 97 in which the normal vector n is associated with all regions.

This normal vector n indicates the normal of the surface of the cornea 901 for each region.
The normal vector eyeball model 97 shows a difference in optical reflection characteristics on the corneal surface as a difference in shape (tilt) on the corneal surface. That is, in the normal vector eyeball model 97, the inclination of each region represents the corneal asphericity.

  In other words, as shown in FIG. 10, the normal vector eyeball model 97 rotates the eyeball 900 to the eyeball 900 ′, and obtains the line-of-sight vector v obtained at the position of the Purkinje image u at the position u ′ of the Purkinje image. The eyeball model is corrected to the line-of-sight vector v ′. That is, in FIG. 10, as the eyeball 900 rotates, the positions of the infrared light source 90, the photographing camera 91, and the corneal curvature center c move to the positions of the infrared light source 90 ′, the photographing camera 91 ′, and the corneal curvature center c ′. become.

<Correction by normal vector eyeball model>
Hereinafter, the normal vector eyeball model correction performed by the normal vector eyeball model line-of-sight error correction unit 24b will be described in detail with reference to FIGS. 11 to 13 (see FIG. 1 as appropriate).

  The normal vector eyeball model line-of-sight error correction means 24b refers to the normal vector eyeball model 97 to calculate the rotation angle at which the normal vector n at the position u of the Purkinje image faces the photographing camera 91, and the calculated rotation angle Only the normal vector eyeball model 97 is rotated, and in the rotated normal vector eyeball model 97, a vector directed from the corneal curvature center c ′ (FIG. 13) to the pupil center e is set as a line-of-sight vector v ′ in which the error is corrected. Output.

As shown in FIG. 11, the normal vector eyeball model line-of-sight error correction means 24b receives the line of sight (line-of-sight vector v) when the user is gazing at an arbitrary object during line-of-sight error correction. Here, the pupil center e and the corneal curvature center c are obtained from the positional relationship between the pupil center e and the position u of the Purkinje image in the pupil-corneal reflection method. The line-of-sight vector v is the difference between the corneal curvature center c and the pupil center e in the world coordinate system (v = ec).
In addition, in FIG. 11, the eyeball center b and the pupil center polar coordinate system (one-dot chain line) are illustrated (the same applies to FIGS. 12 and 13).

  Further, as shown in FIG. 12, the normal vector n at the position u of the Purkinje image does not face the direction of the photographing camera 91 before correcting the line-of-sight error. Accordingly, the normal vector eyeball model line-of-sight error correction unit 24 b calculates the rotation angle at which the normal vector n at the position u of the Purkinje image faces the photographing camera 91.

Then, as shown in FIG. 13, the normal vector eyeball model line-of-sight error correction means 24b rotates the normal vector eyeball model 97 by the calculated rotation angle and sees it from the position u of the Purkinje image before correction. The normal vector n at the position u of the Purkinje image is set to face the photographing camera 91. Further, the normal vector eyeball model line-of-sight error correction means 24b obtains a vector that faces the pupil center e from the corneal curvature center c ′ as the line-of-sight vector v ′ with corrected error in the normal vector eyeball model 97 of FIG. . In this way, the line-of-sight vector v is corrected to the line-of-sight vector v ′.
In FIG. 13, by rotating the eyeball 900 to the eyeball 900 ′, the position of the corneal curvature center c moves to the position of the corneal curvature center c ′.

Thereafter, the normal vector eyeball model line-of-sight error correction means 24b outputs the line of sight (line-of-sight vector v ′) with the error corrected. In the present embodiment, the normal vector eyeball model line-of-sight error correction unit 24b outputs the line of sight with the corrected error to the display control unit 21b and causes the display unit 92 to display the line of sight.
Note that the normal vector eyeball model line-of-sight error correction unit 24b can output the line of sight in an arbitrary coordinate system to various output destinations according to the method of using the line of sight in which the error is corrected.

[Operation of line-of-sight error correction device]
Hereinafter, the operation of the line-of-sight error correction apparatus 20 of FIG. 1 will be described with reference to FIG. 14 (see FIG. 1 as appropriate).
The line-of-sight error correction device 20 causes the display control unit 21b to display the marker image on the display unit 92 (step S1: marker display step).
In the line-of-sight error correction device 20, the marker line-of-sight is input from the line-of-sight measurement device 10A by the marker line-of-sight input means 21a (step S2: marker line-of-sight input step).

The line-of-sight error correction device 20 causes the display control unit 21b to display the marker image and the pointer on the display unit 92 (step S3: marker line-of-sight display step).
The line-of-sight error correction device 20 receives error information from the outside by the error information input means 21c (step S4: error information input step).

  Based on the error information, the line-of-sight error correction device 20 generates a distortion plane in which the pointer coincides with the marker from the plane image showing the position of the pointer with the same size as the marker image, based on the error information. A vector is calculated (step S5: distortion vector calculation step).

The line-of-sight error correction device 20 generates an aspheric eyeball model corresponding to each user based on the distortion vector by the aspheric eyeball model generation means 23.
That is, the difference angle eyeball model generation unit 23a generates a difference angle eyeball model by performing coordinate conversion of the distortion vector of the image coordinate system into the eyeball center polar coordinate system.
Further, the normal vector eyeball model generation means 23b generates a normal vector eyeball model in which the normal vector is defined by the above-described equation (3) in the pupil center polar coordinate system (step S6: aspheric eyeball model generation step). ).

The line-of-sight error correction device 20 corrects the line-of-sight error based on the aspheric eyeball model by the line-of-sight error correction means 24.
That is, the difference angle eyeball model line-of-sight error correction unit 24a refers to the difference angle eyeball model and corrects the line-of-sight error by rotating the line of sight by the difference angle around the origin of the eyeball central polar coordinate system.
Further, the normal vector eyeball model line-of-sight error correcting means 24b refers to the normal vector eyeball model, rotates the normal vector eyeball model so that the normal vector n at the position of the Purkinje image faces the photographing camera, In the rotated normal vector eyeball model, a vector directed from the corneal curvature center to the pupil center is obtained and output as a line of sight with corrected error (step S7: line of sight error correction step).
Steps S1 to S6 are processes at the time of model generation, and step S7 is an operation at the time of line-of-sight error correction.

  As described above, the line-of-sight error correction device 20 according to the embodiment of the present invention generates an aspheric eyeball model using the marker line of sight coordinate-converted by the line-of-sight coordinate conversion / output unit 14. Since the line-of-sight error correction device 20 uses this aspherical eyeball model for line-of-sight error correction, it is not necessary to repeatedly perform personal calibration, and the work burden can be reduced. Furthermore, the line-of-sight error correction device 20 can correct errors caused by the difference between the sphericality of the spherical eyeball model and the asphericity of the corneal shape by using the aspheric eyeball model, and thus suppresses the line-of-sight error to be small. Can

Furthermore, the line-of-sight error correction device 20 can generate a difference angle eyeball model by simple coordinate conversion.
Furthermore, since the reference position of the pupil-corneal reflection method and the normal vector eyeball model is the same, the gaze error correction device 20 can correct gaze errors even when the head moves during gaze error correction. .

  For example, in the conventional pupil-corneal reflection method, even when personal calibration is repeatedly performed and the best data is used, an error of 1 ° or more occurs in the line of sight at the periphery of the display screen. On the other hand, the line-of-sight error correction device 20 can ignore the line-of-sight error even at the periphery of the display screen. As described above, the line-of-sight error correction device 20 can suppress the line-of-sight error to be smaller than that of the conventional pupil-corneal reflection method.

  Here, the normal vector eyeball model can also be said to be an absolute aspheric eyeball model in which the non-uniform shape of the corneal surface is shown by a map of normal vectors. On the other hand, the differential angle eyeball model can also be referred to as an aspherical eyeball model indicating a relative difference angle (corrected rotation angle) with respect to the spherical eyeball model of the pupil-corneal reflection method. Regardless of which aspheric eyeball model is used, the line-of-sight error correction device 20 can extremely reduce the line-of-sight error. In consideration of the above points, the gaze error correction device 20 can be manually switched by the user to use either the differential angle eyeball model or the normal vector eyeball model.

  In the present embodiment, the line-of-sight error correction device 20 has been described as using both the differential angle eyeball model and the normal vector eyeball model, but only one of them may be used. That is, the line-of-sight error correction device 20 is a combination of the difference angle eyeball model generation unit 23a and the difference angle eyeball model line-of-sight error correction unit 24a, or the normal vector eyeball model generation unit 23b and the normal vector eyeball model line-of-sight error correction unit. Only one of the combinations with 24b may be provided. Furthermore, the line-of-sight error correction device 20 is an aspheric eyeball model that combines both the difference angle eyeball model and the normal vector eyeball model, for example, an intermediate aspheric surface that holds the difference angle with polar coordinates based on the pupil center. An eyeball model may be used.

  In the present embodiment, the line-of-sight error correction apparatus 20 has been described as performing the interpolation of the difference angle vector and the normal vector in the aspheric eyeball model generation unit 23, but the present invention is not limited to this. That is, the line-of-sight error correction device 20 may use the distortion vector calculation unit 22 to interpolate a distortion vector at a position where there is no marker by an interpolation method such as linear interpolation using the distortion vector calculated at the marker position. .

In the present embodiment, the line-of-sight error correction apparatus 20 has been described with a series of processes of performing line-of-sight error correction after generating a model. However, it is not necessary to generate an aspheric eyeball model for each line-of-sight error correction. Good.
For example, the line-of-sight error correction device 20 authenticates each user with an authentication function (login function). Then, when there is no aspheric eyeball model corresponding to each user, the line-of-sight error correction device 20 generates an aspheric eyeball model corresponding to the user and stores it in the aspheric eyeball model storage unit (not shown). Thereafter, the line-of-sight error correction device 20 may read the aspheric eyeball model corresponding to the user from the aspheric eyeball model storage means and perform the line-of-sight error correction.

  In the present embodiment, the strain plane is described as a NURBS curved surface, but the present invention is not limited to this. That is, in the present invention, the distortion plane may be a parametric curved surface that can be deformed by a control point, such as a curved surface constituted by a Bezier curve or a B-spline curve. Regardless of which curved surface is used, the line-of-sight error correction device 20 can express an arbitrary distorted curved surface by increasing the number of control points.

  In the present embodiment, the pointer has been described as being displayed as texture mapping. At this time, when the planar pointer is distorted, the circular pointer is distorted accordingly. When it is desired to avoid this, the line-of-sight error correction device 20 may guide the UV coordinates of the pointer and place a circular pointer at the UV coordinates.

  In the present embodiment, it is described that the personal calibration based on the conventional spherical eyeball model is not performed, but the present invention may perform the above-described personal calibration. That is, the line-of-sight measurement device 10 </ b> A may include the personal calibration unit 13.

  In the present embodiment, the line-of-sight measurement / error correction apparatus 1 has been described as an independent apparatus, but the present invention is not limited to this. That is, the present invention can also be realized by a line-of-sight error correction program that causes a general computer to operate as each unit of the line-of-sight measurement / error correction apparatus 1.

  In the present invention, a line-of-sight error correction program may be installed in a smartphone or tablet terminal. In this case, the user can input error information by performing an operation of tracing the display screen (display unit 92) of the smartphone or tablet terminal with a finger so that the center of the pointer matches the marker. And when this invention is implement | achieved with the smart phone or the tablet terminal, even when the positional relationship of a display screen and a user's head changes a lot because a user moves a smart phone or a tablet terminal largely, an exact eyes | visual_axis is measured. be able to.

(Modification)
Hereinafter, with reference to FIG. 15, a difference from the first embodiment regarding the line-of-sight error correction device 20 </ b> B according to the modification of the present invention will be described.
The line-of-sight error correction device 20 in FIG. 1 needs to perform interpolation calculation when the user sees a position different from the marker. Therefore, the line-of-sight error correction device 20B performs an interpolation calculation in advance and stores the calculation result in a lookup table.

The look-up table generation unit 25 receives the distortion vector from the distortion vector calculation unit 22, and generates a lookup table by associating the distortion vector with a small area having a size set in advance in the image coordinate system. .
Here, the look-up table generating means 25 divides the image coordinate system into small areas composed of one or more pixels. Then, the look-up table generating means 25 generates a look-up table by associating the distortion vector calculated for each pixel with each small region. At this time, the look-up table generating means 25 may interpolate the distortion vectors of the small areas for each small area that could not be associated with the distortion vectors by an interpolation method such as linear interpolation.

In the line-of-sight error correction device 20B, the input / output control unit 21, the distortion vector calculation unit 22, and the line-of-sight error correction unit 24 are the same as the respective units in FIG.
Further, the aspheric eyeball model generation unit 23B is the same as FIG. 1 except that the distortion vector of the lookup table generated by the lookup table generation unit 25 is used, and thus detailed description thereof is omitted.

As described above, the line-of-sight error correction apparatus 20B according to the modified example of the present invention does not need to perform interpolation calculation each time by referring to the lookup table, and can speed up the arithmetic processing.
Since the line-of-sight error correction device 20B performs only two-dimensional image processing, an error occurs when the head moves.

1,1B Gaze measurement / error correction device 10, 10A Gaze measurement device 11 Pupil / Purkinje image detection means 12 Gaze vector / reference point calculation means 13 Personal calibration means 14 Gaze coordinate conversion / output means 20, 20B Gaze error correction apparatus 21 Input / output control means 22 Strain vector calculation means 23, 23B Aspheric eyeball model generation means 24 Gaze error correction means 25 Look-up table generation means

Claims (6)

A gaze error correction device that corrects a gaze error measured by a pupil-corneal reflection method using a spherical eyeball model,
Display control means for displaying on the display means a marker image in which a marker is arranged at a predetermined position and the position of the marker line of sight with respect to the marker measured by the pupil-corneal reflection method;
Error information input means for inputting error information indicating a positional deviation between the marker displayed on the display means and the marker line of sight;
Based on the error information input to the error information input means, from the plane image showing the position of the marker line of sight with the same size as the marker image, the position of the marker line of sight is distorted to match the marker A strain vector calculating means for generating a strain plane and calculating a strain vector indicating the strain of the strain plane in an image coordinate system;
An aspheric eyeball model generating means for generating an aspheric eyeball model indicating an aspheric cornea shape based on the strain vector;
Gaze error correction means for correcting a gaze error measured by the pupil-corneal reflection method by rotation based on the aspheric eyeball model generated by the aspheric eyeball model generation means;
A line-of-sight error correction device comprising: The aspheric eyeball model generation means includes
As the aspheric eyeball model, a distortion angle vector of the image coordinate system is transformed to an eyeball center polar coordinate system with the eyeball center as an origin, thereby generating a difference angle eyeball model indicating a difference angle of the line of sight with respect to the spherical eyeball model. Differential angle eyeball model generating means for
The line-of-sight error correcting means includes
Referring to the difference angle eyeball model, a difference for correcting an error of the line of sight by rotating the line of sight measured by the pupil-corneal reflection method by the difference angle around the origin of the eyeball central polar coordinate system Angle eyeball model gaze error correction means,
The line-of-sight error correction apparatus according to claim 1, comprising: The aspheric eyeball model generation means includes
The position 1 of the photographing camera for measuring the line of sight measured by the pupil-corneal reflection method, the rotation angle θ of the eyeball, the position u ′ _θ of the Purkinje image on the cornea surface, and the position of the pupil center (x _θ , Y _θ , z _θ ) and a virtual position l ′ of the photographing camera indicated by the distortion vector with the position l of the photographing camera as a base point, in the pupil center polar coordinate system with the pupil center as the origin, the following (3 A normal vector eyeball model generating means for generating, as the aspheric eyeball model, a normal vector eyeball model in which a normal vector n _θ is defined by
The line-of-sight error correcting means includes
Referring to the normal vector eyeball model, calculate a rotation angle at which the normal vector at the position of the Purkinje image faces the imaging camera, rotate the normal vector eyeball model by the calculated rotation angle, and rotate In the normal vector eyeball model later, a normal vector eyeball model line-of-sight error correction means for outputting a vector directed from the corneal curvature center to the pupil center as the line of sight in which the error is corrected,
The line-of-sight error correction apparatus according to claim 1, comprising:

Lookup table generation means for generating a lookup table by associating the distortion vector calculated by the distortion vector calculation means with a small area having a size set in advance in the image coordinate system,
4. The aspheric eyeball model generation unit generates the aspheric eyeball model with reference to a lookup table generated by the table generation unit. The line-of-sight error correction device described. In order to correct gaze errors measured by pupil-corneal reflection method using a spherical eyeball model,
Display control means for displaying on the display means a marker image in which a marker is arranged at a predetermined position and a position of the marker line of sight with respect to the marker measured by the pupil-corneal reflection method;
Error information input means for inputting error information indicating a positional deviation between the marker displayed on the display means and the marker line of sight;
Based on the error information input to the error information input means, from the plane image showing the position of the marker line of sight with the same size as the marker image, the position of the marker line of sight is distorted to match the marker A strain vector calculating means for generating a strain plane and calculating a strain vector indicating the strain of the strain plane in an image coordinate system;
An aspheric eyeball model generating means for generating an aspheric eyeball model showing an aspheric cornea shape based on the strain vector;
A line-of-sight error correction unit that corrects a line-of-sight error measured by the pupil-corneal reflection method by rotation based on the aspherical eyeball model generated by the aspherical eyeball model generation unit;
Line-of-sight error correction program to function as A line-of-sight error correction method of a line-of-sight error correction apparatus for correcting line-of-sight error measured by a pupil-corneal reflection method using a spherical eyeball model,
A marker display step for causing the display means to display a marker image in which the marker is arranged at a predetermined position on the display means;
A marker line-of-sight input step in which a marker line-of-sight input to the marker measured by the pupil-corneal reflection method is input;
A marker visual line display step in which the display control means causes the display means to display the marker image and the position of the marker visual line;
An error information input step in which error information input means inputs error information indicating a positional deviation between the marker displayed on the display means and the marker line of sight;
A distortion plane in which a distortion vector calculation unit is distorted so as to match the position of the marker line of sight with the marker from a plane image showing the position of the marker line of sight with the same size as the marker image based on the error information And a distortion vector calculation step of calculating a distortion vector indicating distortion of the distortion plane in the image coordinate system;
An aspheric eyeball model generating means for generating an aspheric eyeball model showing an aspheric cornea shape based on the strain vector;
A line-of-sight error correcting step, wherein the line-of-sight error correcting means corrects a line-of-sight error measured by the pupil-corneal reflection method by rotation based on the aspheric eyeball model;
A line-of-sight error correction method characterized by sequentially executing.

Images