JP2011115460A - Visual axis controlling device, method of controlling visual axis, and program of the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a visual axis detecting device or the like, which removes a burden on a user and detects an accurate visual axis direction from the center of a pupil and the center of a corneal curvature. <P>SOLUTION: A visual axis controlling device includes: a camera 11 for imaging a direction of eye balls; a plurality of light sources for emitting light to the direction of the eye balls; a corneal curvature radius-calculating part 23 for calculating a corneal curvature radius based on a plurality of Purkinje images on a corneal surface acquired by the light emitted from the light source; a part 24 for calculating the center of a corneal curvature, which calculates the center of the corneal curvature based on the corneal curvature radius, a position of the camera 11, and the position of the light source 12; a part 25 for calculating the center of a pupil, which extracts a pupil region based on the presence and absence of the reflection of the light emitted from the light source and calculates the center of the pupil by an ellipse approximation; and a visual axis direction-determining part 26 for determining a vector passing through the center of the pupil and the center of the corneal curvature as the visual axis direction. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a line-of-sight control device that controls the driving of a controlled object by a line of sight.

A technique for controlling a control object (for example, a wheelchair or the like) using gaze direction estimation is disclosed (see Patent Document 1 and Non-Patent Document 1).
Various methods are known as methods for detecting human eye movements, and can be roughly classified as follows according to the gaze estimation principle and the eye movement detection method. (1) Limbus tracking method: a method using the difference in reflectance between black eyes (iris), cornea and white eyes, and sclera (see Non-Patent Document 2). (2) Corneal reflection method: a method using the brightest image of the near-infrared wavelength region reflection image (Purkinje image) from the front surface of the cornea, the rear surface, the front surface of the lens, and the rear surface having different refractive indexes (see Non-Patent Document 3) . (3) EOG (Electrooculography) method: A method in which an electrode is attached to the vicinity of the eyeball and a potential difference between a positive potential in the cornea and a negative potential in the retina is used (see Non-Patent Document 4). (4) Search coil method: A method in which a user wearing a contact lens with a built-in coil is placed in a magnetic field, and a potential difference generated by the angle of the coil and the magnetic field by eye movement is used (see Non-Patent Document 5). (5) Fundus Haploscope method: a method of directly observing the fundus using a fundus photographic imaging device in the infrared wavelength region (see Non-Patent Document 6). (6) Image analysis method: A method of imaging an eyeball and detecting the pupil center by an image processing method.

  Although the methods (1), (3), (4), and (5) have high gaze estimation accuracy, they are of a contact type and therefore impose a physical and psychological burden on the user. In addition, the system becomes expensive. Although (6) is relatively inexpensive and has been developed with the advancement of image analysis techniques, there is a problem that the estimation system is low. According to the eye movement detection method using the Purkinje image of (2), it is possible to detect the eye movement stably and accurately with a non-contact type.

  In many of these prior arts, a typical eyeball model is assumed, and eyeball motion estimation is performed using all parameters such as eyeball size as constants. Therefore, an error occurs due to individual differences in eyeball shape, and high eye movement measurement estimation accuracy cannot be expected. In order to solve this problem, the user is forced to perform correction work for removing individual differences by calibration. However, the calibration work requires a great deal of skill and time, which is a heavy burden on the user.

  In addition, a technique for performing line-of-sight measurement using a line connecting the eyeball rotation center coordinates and the pupil center as the line of sight is disclosed (see Non-Patent Document 7). This is obtained by moving the eyeball in various directions before the eye-gaze measurement. By doing so, it is not necessary to present an index and gaze measurement is possible everywhere, but there is a problem that preparation takes time and head movement cannot be allowed.

  Furthermore, a technique is disclosed in which line-of-sight measurement is performed using a line connecting the coordinates of the corneal curvature center (or corneal sphere center) and the pupil center as a line of sight (see Non-Patent Document 8 and Patent Document 2). The coordinates of the corneal curvature center (or corneal sphere center) are obtained by using a general corneal curvature radius (corneal sphere radius) value by installing a single point light source and a camera. However, since these methods assume a model such as a typical human eyeball shape, there is a considerable error in the gaze direction estimation, and the gaze must be corrected by calibration.

  Furthermore, a method for simultaneously estimating the eyes of both eyes by using a stereo camera and two light sources (see Patent Document 3), and a method for obtaining the center of corneal curvature of the user by using two light sources and estimating the center of eyeball rotation (See Non-Patent Document 9). The former estimates not only the line-of-sight direction but also the viewpoint ahead of the line of sight obtained from line-of-sight estimation (three-dimensional measurement) of both eyes, and the pupil center and the corneal reflection center are used for line-of-sight estimation. The latter is a method of estimating the center of corneal curvature using two Purkinje images, having the user gaze at the three indices, estimating the center of eyeball rotation, and estimating the line of sight connecting them. The accuracy is verified by the experiment used.

JP-T-2004-504684 JP 2005-253778 A JP 2005-198743 A

Yoshio Matsumoto, Tomoyuki Ino, Tsukasa Ogasawara: “Electric wheelchair driving support system based on face and line-of-sight information”, IEEJ System and Control Study Group, 2001 K.Abe, S.Ohiamd M.Ohyama: "An Eye-gaze Input Sys-tem based on the Limbus Tracking Method by Image Analysis for Seriously Physically Handicapped People", 7th ERCIMWorkshop "User Interface for All" Adjunct Proc., 185- 186, 2002. Mitsuo Ikeda, qualified psychophysics, Morikita Publishing, 1975. Yukiaki Kuno, Toru Yagi, Kazuyuki Fujii, Kazuo Koga, Yoshiki Uchikawa: “Development of eye-gaze input interface using EOG”, Transactions of Information Processing Society of Japan, 39, 5, 1455-1462, 1998. D.A. Robinson: "A method of measuring eye movement using a sclera search coil in a magnetic field", IEEE Trans. On Biomedical Electronics, 10, 137-145, 1963. http://webvision.med.utah.edu/ Junji Matsuda, Takeshi Nagami, Shigeru Yamane, "Development of Gaze Position Measurement System", IEICE Technical TL2002-2, 2000. Takehiko Ohno, Naoki Takekawa, Atsushi Yoshikawa: “Gaze estimation method based on eyeball shape model”, 8th Symposium on Image Sensing, pp.307-312, 2002. Hiroaki Tanaka, Shinichi Hamada, Ken Kasai, Takashi Takeda, “Gaze detection method by measuring the central position of corneal curvature using two light sources”, IEICE Journal, 108,479, MBE2008-128,117-180,2009.

  The technique shown in Non-Patent Document 1 is a technique for controlling a wheelchair based on facial movement and line of sight, but for example, moving the line of sight as in an ALS (amyotrophic lateral sclerosis) patient. Although it can, it has the subject that it cannot apply to the person who cannot move a head.

  Although it is suggested that the technique shown in Patent Document 1 uses the head posture and the line-of-sight direction for steering a wheelchair, no specific means are described, and the technique is insufficient with respect to wheelchair control. Has the problem. In addition, since it is necessary to acquire images of the face and head from at least two viewpoints, two cameras are required, which may increase the size of the apparatus. Furthermore, since it is necessary to perform image processing and analysis processing on at least two images, there is a problem that processing is slow and complicated.

  The techniques shown in Non-Patent Documents 2 to 8 and Patent Document 2 have problems as described above. Moreover, since the technique shown in Patent Document 3 estimates the line of sight of both eyes with a stereo camera, there is a problem that the apparatus becomes large. Furthermore, since it is necessary to estimate the posture of the head, control on a wheelchair that makes the head unstable is difficult. Furthermore, the technique shown in Non-Patent Document 9 has a problem that it is necessary for the user to pay attention to the three indices, which places a burden on the user.

  Therefore, the present invention provides a line-of-sight control device that controls the operation of a control target (for example, a wheelchair) by detecting the line-of-sight direction. In addition, the provided gaze control device can remove a burden imposed on the user and can detect an accurate gaze direction from the pupil center and the corneal curvature center.

  (1) A line-of-sight control device disclosed in the present application is a line-of-sight control device that controls a control object based on a line-of-sight direction, and includes a line-of-sight direction detection unit that detects the line-of-sight direction, the line-of-sight direction, and the control object. A correspondence information storage means for storing correspondence information for associating the motion command, a correspondence information stored in the correspondence information storage means, and a line-of-sight direction detected by the line-of-sight direction detection means. An operation command analysis means for analyzing and a drive means for driving the control object based on the result of analysis by the operation command analysis means are provided.

  As described above, the gaze control device disclosed in the present application drives the control object based on the correspondence information between the detected gaze direction and the operation command of the control object. For example, even a person who can move only his / her line of sight, such as an ALS patient, can achieve the effect that the operation of the control target intended by the user can be realized with only his / her line of sight.

  (2) In the line-of-sight control device disclosed in the present application, the control target is an electric wheelchair, and the correspondence information storage means sets the correspondence information as a forward acceleration as an operation command corresponding to the case where the line-of-sight direction is downward. It is memorized.

  As described above, the line-of-sight control device disclosed in the present application is an electric wheelchair because the controlled object is an electric wheelchair, and even a person who can move only the line of sight, such as an ALS patient, can move only by the line of sight. The effect is that you can control and move by yourself. In addition, since the motion command corresponding to the case where the line-of-sight direction is downward is forward acceleration, it is possible to accelerate forward while confirming the state of the passage and safety that require special attention during forward acceleration with the line of sight downward. As a result, it is possible to realize safe and easy-to-use line-of-sight control.

  (3) In the line-of-sight control device disclosed in the present application, the correspondence information storage means turns the operation command corresponding to the case where the line-of-sight direction is the right direction and the operation command corresponding to the case where the line-of-sight direction is the left direction. , The operation command corresponding to the case where the line-of-sight direction is the front direction is set to advance, the operation command corresponding to the case where the line-of-sight direction is other is stopped, and the correspondence information is stored. It is.

  Thus, the gaze control device disclosed in the present application turns right when the gaze direction is the right direction, turns left when the gaze direction is the left direction, moves forward when the gaze direction is the front direction, and gaze direction The wheelchair is stopped in other cases, so the direction of movement and the line-of-sight direction match, making it easier to control the wheelchair, and by setting the line-of-sight direction of the stop in many areas, the wheelchair can be controlled in consideration of safety. There is an effect that can be performed.

  (4) The line-of-sight control device disclosed in the present application includes an imaging unit that images at least the front of the controlled object, and an obstacle detection unit that detects an obstacle based on imaging information captured by the imaging unit, The drive means stops movement of the control object when the obstacle detection means detects an obstacle.

  In this way, the line-of-sight control device disclosed in the present application images at least the front of the control object, and stops the movement of the control object when the obstacle is detected from the imaging information. It is possible to prevent the contact of the control object, prevent the control object from being broken, and ensure the safety of the person who operates the control object.

  (5) The line-of-sight control device disclosed in the present application includes an ultrasonic sensor that oscillates an ultrasonic wave at least in front of the control target and measures a distance from the obstacle, and the driving unit includes the obstacle sensor. When the distance to the detected obstacle is equal to or shorter than a predetermined length, the movement of the controlled object is stopped.

  As described above, the line-of-sight control device disclosed in the present application detects at least an obstacle ahead of the controlled object by the ultrasonic sensor and stops the movement of the controlled object. For example, when the obstacle is transparent Even in such a case, there is an effect that the obstacle can be reliably detected, and the safety of the person who operates the control object or the failure of the control object can be ensured.

  (6) A line-of-sight control device disclosed in the present application includes a pupil size detection unit that detects the size of the pupil, and a pupil that analyzes a change in the size of the pupil based on the pupil size detected by the pupil size detection unit. Change analysis means, and the drive means stops the movement of the control object based on the change in the size of the pupil analyzed by the pupil change analysis means.

  As described above, the line-of-sight control device disclosed in the present application stops the movement of the control target based on the change in the size of the pupil.For example, when the obstacle suddenly appears and the pupil opens, The control object is stopped from the change in the pupil, and the contact with the obstacle can be prevented.

  (7) A gaze control device disclosed in the present application includes a pupil size detection unit that detects the size of a pupil, and a pupil that analyzes a change in the size of the pupil based on the pupil size detected by the pupil size detection unit. Change analysis means, and based on the change in the size of the pupil analyzed by the pupil change analysis means, the correspondence information for correcting the correspondence information between the line-of-sight direction stored in the correspondence information storage means and the operation command of the control object An information correction means is provided.

  As described above, the gaze control device disclosed in the present application corrects the correspondence information between the gaze direction and the operation command of the controlled object based on the change in the pupil size. By obtaining and performing correction such as increasing the stop area according to the degree of risk, it is possible to obtain correspondence information according to the environment.

  (8) In the line-of-sight control device disclosed in the present application, the line-of-sight detection means includes a camera that images at least the eyeball direction, a plurality of light sources that emit light at least in the eyeball direction, and a cornea obtained by the light emitted from the light source. Based on a plurality of Purkinje images on the surface, a corneal curvature radius calculation means for calculating a corneal curvature radius, a corneal curvature radius calculated by the corneal curvature radius calculation means, a position of the camera, and a position of the light source A corneal curvature center calculating means for calculating a corneal curvature center, a pupil center calculating means for extracting a pupil region based on the presence or absence of reflection of light irradiated by the light source, and calculating a center of the pupil by elliptic approximation, and the pupil A pupil center calculated by the center calculating means, and a gaze direction determining means for determining a vector passing through the corneal curvature center calculated by the corneal curvature center calculating means as a gaze direction. It is characterized in that to obtain.

  As described above, the gaze control device disclosed in the present application accurately calculates the corneal curvature radius using the Purkinje image obtained from the corneal surface by a plurality of light sources, and calculates the corneal curvature center from the corneal curvature radius. The center of curvature of the cornea can be obtained, and the detection of the gaze direction can be performed with a high accuracy that is not possible in the past, and the gaze detection can be performed while reducing the burden on the user by eliminating the trouble of calibration and the like. There is an effect.

(9) The line-of-sight control device disclosed in the present application is characterized in that the light source emits at least near-infrared light and the camera has at least near-infrared sensitivity distribution.
Thus, the line-of-sight control device disclosed in the present application is a camera that emits at least near-infrared light and the camera has at least near-infrared sensitivity distribution. There is an effect that the line-of-sight direction can be accurately detected.

(10) The line-of-sight control device disclosed in the present application is characterized in that the camera is fixed to a part of a person who operates the control object.
As described above, the gaze control device disclosed in the present application is a case where the position of the face is unstable with a moving body such as a wheelchair because the camera is fixed to a part of the person who operates the control target. However, there is an effect that an accurate line-of-sight direction can be detected by stably capturing the eyes.

  (11) A line-of-sight control device disclosed in the present application includes a facial feature extraction unit that extracts a plurality of facial features from a facial image, and a feature point determination unit that determines feature points in each facial feature extracted by the facial feature extraction unit. Normal direction determining means for determining a normal direction of a plane formed by the feature points, and based on the line-of-sight direction detected by the line-of-sight detection means and the normal direction determined by the normal line determination means The line-of-sight direction is determined.

As described above, the gaze control device disclosed in the present application obtains the head posture with the normal direction of the plane formed by the facial feature points as the face direction, and determines the gaze direction based on the gaze direction and the head posture. Therefore, even if the movement of the camera and the head is not integrated, and the user moves the head, there is an effect that the direction of the line of sight can be detected accurately and stably.
Although the present invention has been shown as an apparatus so far, as will be apparent to those skilled in the art, the present invention can also be understood as a method and a program.

It is a hardware block diagram of the gaze control apparatus which concerns on 1st Embodiment. It is a functional block diagram of a gaze control device concerning a 1st embodiment. It is a figure which shows an example of the corresponding information in the gaze control apparatus which concerns on 1st Embodiment. It is a flowchart which shows operation | movement of the gaze control apparatus which concerns on 1st Embodiment. It is a hardware block diagram of the gaze control apparatus which concerns on 2nd Embodiment. It is a functional block diagram of a gaze control device concerning a 2nd embodiment. It is a figure which shows the geometric relationship of the light source, camera, and eyes in the gaze control apparatus which concerns on 2nd Embodiment. It is a figure which shows an example of the Purkinje image corresponding to two light sources in the visual line control apparatus which concerns on 2nd Embodiment. It is a figure which shows the process of a dark pupil method and a bright pupil method. It is a figure which shows the characteristic parameter of an ellipse. It is a figure which shows the definition of an ellipse parameter. It is a figure which shows the method of the ellipse approximation based on the image of the eye in the gaze control apparatus which concerns on 2nd Embodiment. It is a figure which shows the positional relationship of the light source, eyes, and cornea in the gaze control apparatus which concerns on 2nd Embodiment. It is a flowchart which shows operation | movement of the gaze control apparatus which concerns on 2nd Embodiment. It is an installation drawing of experimental equipment using the line-of-sight control device according to the second embodiment. It is a graph which shows the measurement result of a corneal radius in the visual line control device concerning a 2nd embodiment. It is a figure which shows the display used in the eyeball angle detection accuracy confirmation experiment in the gaze control apparatus which concerns on 2nd Embodiment. It is a table | surface which shows an experimental result in the gaze control apparatus which concerns on 2nd Embodiment. It is a functional block diagram of a gaze control device concerning a 3rd embodiment. It is a figure which shows a three-dimensional head posture model. It is a figure which shows head rotation of each plane in a three-dimensional head posture model. It is a figure which shows the bird's-eye view used for the eyes | visual_axis control apparatus which concerns on 3rd Embodiment. It is a figure which shows the motion model of eyes. It is a figure which shows the relationship between the coordinate which shows the head attitude | position in the gaze control apparatus which concerns on 3rd Embodiment, and the coordinate which shows a gaze direction. It is a figure which shows determination and tracking of eyes in the gaze control apparatus which concerns on 3rd Embodiment. It is a figure which shows the detection of the corner of an eye and an opening | mouth with a Gabor filter in the gaze control apparatus which concerns on 3rd Embodiment. It is a 1st figure which shows the experimental result of the gaze control apparatus which concerns on 3rd Embodiment. It is a 2nd figure which shows the experimental result of the gaze control apparatus which concerns on 3rd Embodiment.

  Embodiments of the present invention will be described below. The present invention can be implemented in many different forms. Therefore, the present invention should not be construed based only on the description of this embodiment. Also, the same reference numerals are given to the same elements throughout the present embodiment.

  In the following embodiments, the apparatus will be mainly described. However, as is apparent to those skilled in the art, the present invention can also be implemented as a method and a program for operating a computer. In addition, the present invention can be implemented in hardware, software, or hardware and software embodiments. The program can be recorded on any computer-readable medium such as a hard disk, CD-ROM, DVD-ROM, optical storage device, or magnetic storage device. Furthermore, the program can be recorded on another computer via a network.

(First embodiment of the present invention)
A line-of-sight control apparatus according to the present embodiment will be described with reference to FIGS. The line-of-sight control device according to the present embodiment detects a line-of-sight direction and drives and controls a control object in the detected line-of-sight direction. In addition, although an electric wheelchair is demonstrated as a control target object here, not only this but control can be implemented if it is control of a mobile body (including remote operation, such as a robot).

  FIG. 1 is a hardware configuration diagram of the line-of-sight control device according to the present embodiment. As shown in FIG. 1 (A), the line-of-sight control device 1 includes at least a first camera 11 that images one eye of the user, a second camera 13 that images at least the front of the electric wheelchair 17, and at least An ultrasonic sensor 14 that irradiates an ultrasonic wave in front of the electric wheelchair 17, a PC 10 that determines a line-of-sight direction and outputs a drive control signal of the electric wheelchair 17 to the wheelchair control device 16 via the USB interface 15, and wheelchair control And an electric wheelchair 17 that is driven according to a drive signal of the device 16.

The PC 10 detects the line-of-sight direction, outputs a drive command signal corresponding to the detected line-of-sight direction, and detects an obstacle based on the information captured by the second camera 13 and the information of the ultrasonic sensor 14. The drive command signal is output. The detection of the line-of-sight direction is performed based on the imaging information captured by the first camera 11. Here, for example, the methods described in the respective documents shown in the above prior art documents and the following second and third embodiments are used. Various methods such as the method shown in FIG.
In FIG. 1, the configuration indicated by the dotted line may or may not be provided as necessary.

  FIG. 1B shows how each hardware is used. Since the first camera 11 and the second camera 13 are fixed to the user's glasses and capture images in directions opposite to each other, one of the first camera 11 and the second camera 13 always stably captures the user's eyes at a fixed location. The other can image the forward direction from the user's line of sight. Each camera does not necessarily have to be fixed to the glasses, but the first camera 11 is a part where the eyes can be stably imaged corresponding to the movement of the user's head, that is, a part of the head. The second camera 13 may be anywhere as long as the front of the electric wheelchair 17 can be imaged. Further, although not shown, each hardware is connected in a state where it can communicate with a connection line or wirelessly.

  FIG. 2 is a functional block diagram of the visual line control device according to the present embodiment. The function of the line-of-sight control unit 40 is realized by a CPU, a memory, a hard disk, various interfaces, and the like included in the PC 10 in FIG. The line-of-sight control unit 40 includes a line-of-sight detection unit 30, an operation command analysis unit 31, a correspondence information unit 32, a drive unit 33, an obstacle detection unit 34, a pupil change analysis unit 35, and a correspondence information correction unit 36.

  As described above, the line-of-sight detection unit 30 detects the line-of-sight direction of the user of the electric wheelchair 17 by various methods. The operation command analysis unit 31 analyzes the wheelchair driving operation corresponding to the gaze direction detected by the gaze detection unit 30 based on the correspondence information unit 32. Here, the correspondence information in which the line-of-sight direction stored in the correspondence information unit 32 is associated with the operation command of the controlled object will be described.

  FIG. 3 is a diagram illustrating an example of correspondence information in the visual line control device according to the present embodiment. The line-of-sight direction in FIG. 3A corresponds to the operation command command in FIG. Here, a forward acceleration operation command is executed when the line of sight is directed downward. When starting to move forward in a wheelchair, the user inevitably tends to pay attention to the downward direction (such as the feet of the wheelchair) to ensure safety. It becomes possible. Similarly, the right and left directions correspond to right and left turns, respectively, and the front corresponds to forward movement. In the case of other areas, it corresponds to a stop. By increasing the stop area, the operation can be easily stopped and the safety can be improved. A predetermined size is set in the region of the operation command command corresponding to each line-of-sight direction, and when an instruction to determine the operation command is input with the line-of-sight direction detected within the region, The electric wheelchair is driven accordingly.

  Note that the operation command can be determined in two modes. One is a mode in which it is determined to watch the direction. When a certain direction is being watched for a predetermined time, an operation command in the line-of-sight direction is determined. The other is a mode in which blink is detected and determined. Usually, blinking naturally performed to prevent dry eyes is about 0.5 seconds. Therefore, when a blink of about 0.7 seconds that can be determined to blink intentionally is detected, an operation command in the line-of-sight direction is determined.

  Returning to FIG. 2, the drive unit 33 outputs a drive signal for executing the operation command based on the result analyzed by the operation command analysis unit 33. The obstacle detection unit 34 analyzes the imaging information captured by the second camera 13 to detect an obstacle in the traveling direction of the electric wheelchair 17, and based on the signal sent from the ultrasonic sensor 14, the electric wheelchair 17 obstacles in the direction of travel are detected. In particular, an obstacle that is difficult to detect by image processing (for example, a transparent body such as glass) is detected by the ultrasonic sensor 14. Specifically, the distance from the obstacle is measured by a signal from the ultrasonic sensor 14, and it is determined that the obstacle is detected when the distance is equal to or less than a predetermined value (approached a certain distance or more). it can.

  The pupil change analysis unit 35 analyzes the change in the size of the pupil in the user's eye captured by the first camera 11. For example, when the human eye is surprised, the pupil changes greatly. The pupil change analysis unit 35 analyzes the degree of the change and estimates the situation around the user (for example, the presence or absence of a sudden obstacle). There are two modes for the response based on the analysis result of the pupil change analysis unit 35. One mode outputs a stop driving signal directly to the driving unit 33 and the electric wheelchair 17 is stopped. The other is that the correspondence information correction unit 36 corrects the correspondence information stored in the correspondence information unit 32. In the correction, for example, in the correspondence information of FIG. 3, each defined line-of-sight area is reduced and an area indicating a stop operation command is increased. By doing so, the operation becomes easy for the electric wheelchair 17 to stop, and safety can be ensured.

Note that only one of the two modes described above may be selected in advance, or the mode may be set based on the result analyzed by the pupil change analysis unit 35. For example, if the change in the pupil is large, it can be determined that the degree of surprise is also large, so the electric wheelchair 17 is unconditionally stopped, but if the change in the pupil is small, it is determined that there is a little surprise and the unconditional Instead of stopping the electric wheelchair 17, the correspondence information may be corrected and changed to correspondence information that is easy to stop.
In FIG. 2, the processing unit indicated by the dotted line may or may not be provided as necessary.

Next, the operation of the line-of-sight control device according to this embodiment will be described. FIG. 4 is a flowchart showing the operation of the visual line control device according to the present embodiment. First, the imaging information of the user's eyes captured by the first camera 11 is input (S41), and the line-of-sight direction is detected (S42). The operation command is analyzed with reference to the correspondence information (S43). A wheelchair drive signal corresponding to the analyzed operation command is output (S44). The presence / absence of abnormality is determined from the imaging information captured by the second camera 13 (S45). If there is an abnormality, a drive signal for stopping the drive of the wheelchair is output (S50), and the process is terminated. If there is no abnormality, the presence / absence of abnormality is determined by a signal from the ultrasonic sensor (S46). If there is an abnormality, a drive signal for stopping the drive of the wheelchair is output (S50), and the process is terminated. If there is no abnormality, it is determined whether or not there is a pupil change (S47). If there is no pupil change, the process is terminated. If there is a pupil change, the mode is determined (S48). If the pupil mode is the stop mode, a drive signal for stopping the wheelchair is output (S50), and the process is terminated. If it is in the correction mode, the correspondence information is corrected to information that allows the wheelchair to easily stop (S49), and the process is terminated.
In addition, the said process is always performed repeatedly during control of a wheelchair, and a process is complete | finished when it quits control of a wheelchair.

(Second embodiment of the present invention)
With respect to the line-of-sight detection process of the line-of-sight detection unit 30 in the first embodiment, a method for performing accurate line-of-sight detection without calibration will be described. There are four possible uses that require calibration: (1) The shape of the eyeball and cornea is complicated (the corneal curvature radius increases gently on the ear side and steeply on the nose side), so that the corneal surface cannot be reproduced only by measuring the curvature diameter of the cornea. (2) Camera Distortion of the optical system, (3) refraction at the corneal surface, (4) deviation of the fovea from the center of the eyeball. In the line-of-sight detection processing according to the present embodiment, the line-of-sight detection accuracy is obtained with an error of about 0.57 deg in the x direction and 0.98 deg in the y direction without calibration.

  The line of sight is a straight line connecting the fovea, the node of the lens (lens), and the target point, but it is difficult to specify the position of these, so in this embodiment, the direction through the corneal curvature center and the pupil center is defined as the line of sight . The gaze direction according to this definition is essentially different from the gaze direction that is the original definition. The difference depends on individual differences such as eyeball radius and corneal curvature. In this embodiment, it is possible to detect the direction of the line of sight with high relative accuracy without depending on individual differences by accurately obtaining and calculating parameters without performing calibration.

  Processing of the line-of-sight detection unit in the line-of-sight control device according to the present embodiment will be described with reference to FIGS. FIG. 5 is a hardware configuration diagram of the line-of-sight control device according to the present embodiment. As shown in FIG. 5A, the line-of-sight control device 1 calculates the line-of-sight direction based on at least the camera 11 that captures one eye of the user, the plurality of light sources 12, the camera 11, and the light source 12. PC 10 to be determined. The camera 11 and the light source 12 may be integrated as shown in FIG. In FIG. 5B, a plurality of NIR (Near InfraRed) LEDs 11b are provided centering on a lens 11a, and a Purkinje image is created by the irradiation light of the LEDs 11b. In the PC 10, the corneal curvature center is calculated from the Purkinje image, and a straight line connecting the calculated corneal curvature center and the pupil center is determined as the viewing direction.

  Although only the minimum necessary configuration is described here, for example, when used for controlling the electric wheelchair shown in the first embodiment, the light source 12 is added to the hardware configuration of FIG. It becomes. Further, the line-of-sight control device 1 shown in FIG. 5 may be regarded as a line-of-sight detection device having only a function of detecting the direction of the line of sight.

  FIG. 6 is a functional block diagram of a line-of-sight detection unit in the line-of-sight control device according to the present embodiment. The function of the line-of-sight detection unit 30 is realized by a CPU, a memory, a hard disk, various interfaces, and the like included in the PC 10 in FIG. The line-of-sight detection unit 30 includes an input unit 21, a Purkinje image acquisition unit 22, a corneal curvature radius calculation unit 23, a corneal curvature center calculation unit 24, a pupil center calculation unit 25, and a line-of-sight direction determination unit 26.

  The input unit 21 inputs an eye image captured by the camera 11. The Purkinje image acquisition unit 22 acquires a Purkinje image from the input eye image. The corneal curvature radius calculation unit 23 calculates the corneal curvature radius using the acquired Purkinje image. Here, processing of the Purkinje image acquisition unit 22 and the corneal curvature radius calculation unit 12 will be described.

FIG. 7 is a diagram illustrating a geometric relationship between the light source, the camera, and the eyes in the visual line control device according to the present embodiment. There are first to fourth Purkinje images used for corneal curvature estimation, depending on where they appear. The first Purkinje image is a reflection on the front surface of the cornea, and the reflection on the rear surface of the cornea, the front surface of the lens, and the rear surface of the lens is hereinafter referred to as second to fourth Purkinje images. Among these, the Purkinje image having the largest amount of reflected light is the first Purkinje image. Therefore, in the present embodiment, two first Purkinje images having the largest amount of reflected light from the two light sources are used. As shown in FIG. 7, a reflection image on the cornea surface from two light sources (hereinafter simply referred to as a Purkinje image) bisects the angle formed by the light source position, the corneal curvature center, and the camera optical system aperture center. Located on a straight corneal surface. Therefore, the corneal curvature radius r _E can be expressed by the following equation.

L is the distance between the light sources, H is the distance between the camera and the eyeball, and g is the distance between the Purkinje images corresponding to the two light sources.

  The distance between the light sources and the distance between the camera and the eyeball are set values, and the distance between the two Purkinje images is measured from an eyeball peripheral image acquired by the camera. Therefore, the corneal curvature radius can be estimated. Further, the center position of the corneal curvature can be estimated from the settings of L, H, and g and the measured values. At this time, L and H are set values, and the estimation accuracy of the corneal curvature radius and the center position depends on the deviation from the set values and the measurement error of g. Although L is slightly different from the set value, H is highly likely to change due to the back and forth movement of the user's head. From equation (1), it can be seen that the corneal curvature radius changes in proportion to the back-and-forth movement of the user's head.

  Next, the estimation error factor of g is examined. FIG. 8 is a diagram illustrating an example of a Purkinje image corresponding to two light sources in the visual line control device according to the present embodiment. Since a plurality of Purkinje images are formed on the corneal surface, only the first Purkinje images corresponding to two light sources are extracted from these. First, since the Purkinje image has a higher reflection luminance than the surrounding area (see FIG. 8A), pixels indicating high luminance in the acquired image are extracted and set as candidates for the first Purkinje image (FIG. 8). (See (B)). In the example of FIG. 8, four candidates are extracted. Since the first Purkinje image exists in the vicinity of the pupil, two Purkinje candidates existing at the position closest to the pupil are extracted and set as two first Purkinje images corresponding to the two light sources (see FIG. 8C). ). The pixel centroids of the two first Purkinje images are obtained, and the distance between the centroids is set as the distance between the two first Purkinje images g (see FIG. 8D). In FIG. 8D, the + mark indicates the position of the center of gravity of the two estimated first Purkinje images.

Returning to FIG. 6, the pupil center calculation unit 25 calculates the pupil center. The calculation method will be described below. First, assume that the pupil is an ellipse. By approximating the pupil region with an ellipse to obtain the center coordinates of the pupil, the center of the ellipse is set as the center coordinate of the pupil. The procedure for approximating the pupil region with an ellipse is shown below.
(1) A temporary pupil region is extracted by the dark pupil method shown below.
(2) For the temporary pupil region, k-means clustering (Sadaaki Miyamoto, “Theory and Application of Fuzzy Clustering for Cluster Analysis”, Morikita Publishing Co., Ltd., 1999) is used to remove extra parts (clusters with a small number of individuals). Remove.
(3) An edge is extracted from the area after the excess portion is removed.
(4) A pupil contour sample point is extracted from the edge by a method using the property of an ellipse.
(5) Ellipse approximation is performed on the pupil contour sample points by the method of least squares.
(6) The center coordinates of the ellipse are the pupil center coordinates.

  Here, the dark pupil method will be described. FIG. 9 is a diagram illustrating processing of the dark pupil method and the bright pupil method. The dark pupil shown in FIG. 9A means that when an illuminator is installed at a position away from the lens optical axis of the camera to illuminate the eyeball, the pupil part of the illuminated eyeball loses light reflection and is captured by the camera. The pupil part of the obtained eyeball image utilizes the property that it becomes dark, and this dark part is extracted by image processing. Conversely, the bright pupil method shown in FIG. 9B is that when the eyeball is illuminated with the lens optical axis of the camera and the optical axis of the illumination being coaxial, the pupil portion of the eyeball captured by the camera is reflected by reflection of light on the retina. Will always be brighter. This is a method of extracting this bright part by image processing using this. In this embodiment, since the lens optical axis of the camera is away from the position of the illuminator, the pupil region is extracted using the dark pupil method.

  The pupil region obtained by the dark pupil method includes extra items such as eyelashes, hair, and eyebrows in addition to the pupil. Therefore, it is necessary to remove extraneous matters other than the pupil region. This method uses expansion / reduction processing of the extraction region. The expansion / reduction process uses a method based on morphology. The purpose of the morphology is to extract a feature by deforming a figure by applying a set-theoretic operation to a given binary image or grayscale image. Since it is defined by a set operation between the processing target image and the structural element, the operation result changes depending on how the structural element is selected. By performing dilation / reduction processing, the pupil portion is extracted and the extra portion is deleted.

Next, a labeling process is performed on the temporary pupil region. The size of each area extracted by labeling can be compared. Noise is removed by removing regions below a certain size. The labeling process is to attach the same label (number) to each connected pixel, and to classify several groups of areas as a group. Here, a labeling process based on four neighborhoods is selected. Find an unlabeled pixel (x; y) on the image and add a new label. Next, the same label is attached to the pixels connected in the four directions of the corresponding pixel (x; y). Thereafter, similar processing is performed on the newly labeled pixels. By doing this for the entire image, all independent regions are labeled.
There are various contour shapes of the extraction region after the labeling process, and the extraction region is selected from these according to the circularity defined by the following equation.

Here, C is the circularity, A is the area of the extraction region, and L is the perimeter.

  Since it is assumed that the pupil is an ellipse, the pupil region has a characteristic of high circularity. Therefore, an extraction region having a certain circularity or less is removed. In this way, a region corresponding to the pupil is left (this is called a temporary pupil region). Next, the elliptical sample points of the temporary pupil region are extracted. That is, in the extracted pupil region, there is a possibility that the pupil cannot be extracted accurately because the illumination is reflected or the upper and lower portions of the pupil are hidden and missing. Therefore, an optimum ellipse is approximated in the sense of least squares by detecting an edge from the provisional pupil region. At this time, if calculation is performed using points other than the points on the pupil contour, an approximate ellipse is formed by a point that is not an accurate ellipse contour. In order to avoid this, only the points existing on the pupil ellipse are extracted using the ellipse properties shown below.

  FIG. 10 is a diagram showing the feature parameters of the ellipse. The straight lines l, m, and n are parallel and l and n are equidistant from m. Intersections between the ellipse and the straight line l are a and b, and intersections between the ellipse and the straight line n are c and d. Let o be the midpoint of the intersection of the ellipse and the straight line m. If the middle point of the line connecting the middle points of a and b and the middle points of c and d is defined as o ', o' has a property that it overlaps o. For the temporary pupil region, a point corresponding to oi ′ (i = 1 / N) of a set of N parallel lines equidistant from the straight line m drawn at the center thereof is obtained. The obtained points are distributed on the straight line m. When the noise is sufficiently low, the position where the most points o 'are collected corresponds to the position o. Points that are away from the position include points that are not on the locus of the ellipse, so they are excluded. bi; ci; di are points on the locus of the ellipse, and therefore, using them, an accurate elliptical approximation of the pupil contour can be performed. For the edge detection, a Canny filter (see J. Canny: A Computational Approach to Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, .8, 6, .679-698, 1986.) is used.

  In order to approximate an ellipse from pupil ellipse sample points, the ellipse parameters in FIG. 11 are defined. These parameters can be defined by the following equations (3)-(7). These are equations when the observed noise can be ignored in the acquired data, but in reality the noise is superimposed. However, the pupil contour is derived by applying the least square method using a large number of observation pupil ellipse sample point data (Xi, Yi).

Here, in order to apply the least squares method,

A, B, C, D, and E are obtained by applying the Moore Penrose general inverse matrix (see Kohei Arai, Applied Linear Algebra, Modern Science Co., 2006.).

  FIG. 12 shows the flow of the above series of processing. FIG. 12 is a diagram illustrating an ellipse approximation method based on an eye image in the visual line detection device according to the present embodiment. 12A is an acquired eye image, FIG. 12B is a binarized image, FIG. 12C is a noise-removed image, FIG. 12D is an edge-detected image, and FIG. (E) is an image obtained by extracting sample points, and FIG. 12 (F) is an image of an approximated ellipsoid.

  Returning to FIG. 6, the corneal curvature center calculator 24 calculates the center of the corneal curvature based on the corneal curvature radius calculated by the corneal curvature radius calculator 23. Here, the process of the corneal curvature center calculation unit 24 will be described.

  FIG. 13 is a diagram illustrating a positional relationship between the light source, the eyes, and the cornea in the visual line detection device according to the present embodiment. Using the corneal curvature radius obtained by the corneal curvature radius calculation unit 23, the corneal curvature center is obtained from the positional relationship of the light source, eyeball, and camera shown in FIG. Since the corneal curvature center exists on the bisector connecting the camera, the Purkinje image, and the light source, it is obtained from the following equation. Since two light sources are used in this embodiment, two Purkinje images can be obtained.

  Returning to FIG. 6, the gaze direction determination unit 26 determines the gaze direction based on the corneal curvature center calculated by the corneal curvature center calculation unit 24 and the pupil center calculated by the pupil center calculation unit 25. The processing of the line-of-sight direction determination unit 26 will be described.

  Here, a vector passing through two points of the corneal curvature center and the pupil center is set as the line-of-sight vector, and the gaze point on the display is calculated. The camera position is (0, 0, 0), and the center coordinates of the image captured by the camera are (0, 0, z). Assuming that the line-of-sight vector is v = (xv; yv; zv), the camera, illumination, and display are fixed at this time, and the distance in the z-axis direction is known. Assuming that the distance between the eyeball and the display is zh, the gaze point coordinate t = (xt; yt) on the display can be obtained by the following equation.

  Next, the operation of the line-of-sight detection unit in the line-of-sight control device according to the present embodiment will be described. FIG. 14 is a flowchart showing the operation of the line-of-sight detection unit in the line-of-sight control device according to the present embodiment. First, the imaging information of the eyes imaged by the camera 11 is input (S141), and a Purkinje image is acquired (S142). A corneal curvature radius is calculated from the acquired Purkinje image (S143). The center of the pupil is calculated (S144), and the corneal curvature center is calculated from the corneal curvature radius (S145). The line-of-sight direction is calculated from the pupil center and the corneal curvature center (S146), the gaze point is obtained from the line-of-sight direction (S147), and the process of the line-of-sight detection unit is terminated.

Hereinafter, an experiment using the visual line control device according to the present embodiment will be described. Use the following measuring equipment.
Camera: 1.3 million pixel infrared camera PC: CPU 1.6 GHz, memory 1 GB
Infrared projector: 48 infrared LEDs, wavelength 850nm
These equipments are installed as shown in FIG.

  In this experiment, L = 1000 mm, HC = 670 mm, HT = 150 mm, and HD = 380 mm. Moreover, the image acquired from the camera at the time of gaze measurement was saved as a moving image, and the accuracy was verified using this moving image.

(1) Corneal Radius Measurement Experiment FIG. 16 is a graph showing a measurement result of the corneal radius in the line-of-sight control device according to the present embodiment. The spikes in the figure are due to false detection of Purkinje images, and these can be excluded from the measurement of the corneal radius. In addition, two lines in the figure are maximum errors in the estimated value of the corneal curvature radius that can be considered when the head moves ± 1.0 cm in the display direction. It shows that there is a true value of the corneal curvature radius between these lines. The average value of the corneal curvature radius excluding the edge portion is used as the estimated value of the corneal curvature radius used for the actual eyeball rotation angle estimation. From this result, the value of the corneal curvature radius is set to R = 7.92 mm.

(2) Eyeball rotation angle measurement experiment In the eyeball angle detection accuracy confirmation experiment, the estimated accuracy of the eyeball angle detection accuracy is confirmed by sequentially looking at the five indices displayed on the display shown in FIG. Confirmed. FIG. 18A shows estimation results for eyeball angle detection with different corneal curvature radii. At this time, when the radius of curvature of the cornea is R = 7.92 mm, Δxy = 1.128. Near the measured corneal curvature radius, the eyeball rotation angle detection accuracy is the highest. Therefore, it can be confirmed that the use of the measured corneal curvature radius improves (stabilizes) the eyeball rotation angle detection rather than using the standard value of the corneal curvature radius.

  In addition, changes in the gaze estimation result of actual measurement due to translation and rotation of the head were examined. FIG. 18B to FIG. 18F show the results. Here, movement in each direction of x, y, and z and rotation (around the y and z axes) were examined. In FIG. 18B, it can be seen that the error caused by the head movement in the x direction affects only the x direction. Similarly, in FIGS. 18C and 18D, it can be seen that an error caused by head movement in the y and z directions affects only the y and z directions. This is because the depth is given as a constant. To solve this, the distance between the head and the camera need only be measured in real time. As shown in FIG. 18E, the head rotation in the roll direction shows a large decrease in estimation accuracy. Further, in FIG. 18F, the head rotation in the yaw direction does not show the effect of the head rotation on the estimation accuracy within a certain range, and if the range exceeds a certain range, the estimation accuracy greatly decreases. It was. This is considered to be due to the human cornea shape in which the radius of curvature of the cornea gradually increases on the nose side and sharply on the nose side. In the measurement of the radius of curvature of the cornea, since measurement is performed within a range of 3.0 to 5.0 mm from the central portion of the cornea curvature, it is considered that a large accuracy drop occurs when the radius exceeds this.

  The effect of the head rotation in the pitch direction on the estimation accuracy is that the downward rotation of the head increases the effect of eyelashes, making it difficult to detect pupils (measurement is impossible). Since the influence of eyelashes is reduced, the result that the pupil detection accuracy is improved is obtained. For this reason, it is desirable that the camera position at the time of measurement is installed so as to look up at the head. As a result, in the eye gaze detection apparatus according to the present embodiment, the eyeball rotation angle estimation accuracy is obtained with an error of about 0.57 deg in the x direction and 0.98 deg in the y direction without personal calibration.

(Third embodiment of the present invention)
Although the camera is fixed to the user's glasses in FIG. 1B in the first embodiment, it is not always necessary to fix the camera to a part of the user (particularly the head). In this case, in order to prevent a detection error in the gaze direction due to the movement of the user's head, it is possible to specify the posture of the head and correct the gaze direction error. In the present embodiment, a detection method for accurately detecting the gaze direction even when the user's head has moved will be described.

  For example, both ends of both eyes and both ends of the mouth are extracted from the acquired image in order to detect the shape of the head, the eyes, and the cornea. The center positions of the eyes and the mouth are determined from both ends of the eyes and the mouth, and the posture of the head is defined by the normal of the triangle defined from the center positions of both eyes and the mouth. At this time, the base point of the normal line is set at the midpoint of the center position of both eyes. From the relationship between the rotation center position of the head and the camera position and the geometric relationship between the head rotation center and the defined triangle, it is possible to correct the error in the line-of-sight direction.

  The hardware configuration of the visual line control device according to the present embodiment is the same as that in FIG. However, the position of the first camera 11 does not need to be fixed to the glasses or the head, and may be fixed to the electric wheelchair 17, for example.

  FIG. 19 is a functional block diagram of the visual line control device according to the present embodiment. The difference from FIG. 2 is that a head posture detection unit 37 is newly provided. The head posture detection unit 37 uses the face image captured by the first camera 11 to obtain a normal line of a plane formed by arbitrary points of a plurality of facial features, and obtains the normal direction as the head posture. . The gaze detection unit 30 detects the gaze direction in consideration of the head posture detected by the head gaze detection unit 37. Other processing units are the same as those in FIG.

  In the visual line control device according to the present embodiment, a three-dimensional head posture model is used to detect the head posture. The three-dimensional head posture model is used to convert a three-dimensional head posture from a two-dimensional face feature. Here, facial features such as eyes, eyebrows, nose and mouth are used. The head is made of XY, XZ and YZ planes. The head posture is defined by the rotation angle θ in each plane. By using this model, the three-dimensional head posture can be expressed only by the two-dimensional image.

  The head posture is represented by θ (x, y, z). FIG. 20A shows the head posture on the XY plane viewed from the front. A central axis is assumed at the bottom of the mouth. Based on the rotation of the head, the position of the facial feature moves relative to the central axis. The position of the moved facial feature is determined by the facial feature radius R and the rotation angle θ. FIG. 20B shows the head posture of the YZ plane viewed from the side. When the head rotates up and down like a nodding motion, the facial features move based on their rotation about the central axis. Similar calculations are performed for the XZ plane shown in FIG. And according to each, two-dimensional information is converted into three-dimensional information.

The head rotation in each plane is shown in FIG. Position when one facial feature is located at P (x, y), radius R, initial angle θ ₀ , central axis O (0,0), and P (x, y) rotates with respect to the central axis P ′ (x ′, y ′) is calculated from the acquired two-dimensional image and three-dimensional head posture model. The position is calculated at the rotation angle of the following equation (10).

This is the same for the YZ plane and the XZ plane as shown by the following formulas (11) and (12).

  All position information is defined by the pixel coordinate system. Therefore, the central axis is represented by O (x, y) coordinates, and the position of the facial feature is mapped onto Pi (x, y). As shown in equations (11) and (12), the pixel coordinates can be converted into rotation angles.

  A three-dimensional head posture can be detected by combining the above three-dimensional head posture model with head posture detection using a bird's-eye view shown below. The bird's-eye view is a view as if the bird is an observer. Bird's eye views are often used for blueprints, floor designs, maps, and the like. These views can acquire images of different sizes depending on the distance between the object and the camera. This bird's eye view is used for estimating the head posture.

  FIG. 22 shows a bird's-eye view of head posture measurement. These coordinates are dxU = dxD and dyR = dyL. However, these values change according to changes in the head posture. As shown in FIG. 22A, when the head rotates to the right, dyL becomes larger than dyR. Similarly, as shown in FIG. 22B, when the head rotates to the left, dyR becomes larger than dyL. Similarly, the pitch direction of the head posture can be calculated. By combining the three-dimensional head posture model and the bird's eye view, the head posture can be easily estimated.

  Next, eye movement will be described. Several points are needed as a reference to measure eye position. The ends of both eyes, arches, sclera, iris, etc. are used. The simplest method is to estimate eye movement using the center position of the iris and the edges of both eyes. If trouble occurs and the edge of one or both eyes cannot be detected, another reference point is needed as a backup. The sclera is used for iris position estimation. The iris is located in the center of the cornea and is estimated through the position of the sclera. Another problem is user movement. If one eye is slightly open, the eye orientation is resolved.

The line-of-sight direction is estimated based on the eye movement model shown in FIG. The shape of the eye is formed by an ellipsoid radius R and a viewing angle α. When the eyes are facing the front, the left and right viewing angles α _R and α _L are the same. Therefore, the movement of the eyes on the three-dimensional coordinates is estimated from the model result of the estimated line-of-sight angle. The upper and lower lines of sight α _U and α _D are estimated by the same method. Then, an eye movement output α (R, L, U, D) is estimated.

  Next, the line-of-sight direction is estimated from the head position and the eye movement determined above. FIG. 24 shows a coordinate system for line-of-sight estimation. Vector H indicates the head posture, and vector E indicates the movement of the eyes. The line-of-sight vector is estimated by adding the head posture and the eye motion vector.

  Image processing will be described for analysis. In order to allow the user's movement, the position and posture of the head are estimated. The eyes, the edge of the eye, the head of the nose, and the edge of the mouth, which are facial features, are detected from the acquired image and tracked. The face itself can be detected based on the well-known Viola-Jones method. This method is relatively fast and robust to changing conditions. The eye position is estimated based on the Viola-Jones method of shape base, appearance base, and hybrid base. The Viola-Jones classification detects eyes and uses AdaBoost. To apply the Viola-Jones classification system, the OpenCV Viola-Jones function is used. Before using this function, an XML file must be generated and the face or eye image collected. They have a negative sample and a positive sample, where the negative sample corresponds to an image without an object and the positive sample corresponds to an image with an object. After acquiring the image, OpenCV searches for the center position of the face and the center position of the eyes.

Once the eye is detected, eye tracking by the Viola-Jones method is not necessary. This is because template matching used with morphological filters is as fast and robust. In the eye tracking process, the iris center and the edge of the eye are used. To find the iris center, the correlation coefficient R _coeff in template matching is used.

T and I indicate pixel positions of the template and the current image in the xy coordinates. The correlation coefficient R _coeff indicates a correlation coefficient between a template normalized for correction due to a difference in illumination state and the current image. FIG. 25 is the iris center mapped to the coordinate system.

  Compared to the detection of the iris center, detection of the edge of the eye and the edge of the mouth is considerably difficult, so a Gabor filter that detects the contour of the eye and the mouth is employed. The Gabor function is defined by a sinusoidal plane modulated with a Gaussian envelope. This filter is useful for detecting specified patterns. A two-dimensional Gabor kernel is represented as follows:

σ is the change in Gaussian function, λ is the wavelength of the sine function, θ is the normal orientation of the Gabor function with respect to the stripe, and γ is the aspect ratio specified by the ellipse supported by the Gabor function.

  Before using this filter, first create a Gabor kernel with dispersion, wavelength, orientation, and spatial aspect ratio. Through a difficult process between the source and the Gabor template image, a Gabor image as shown in FIG. 26 can be obtained. In this way, by using the Gabor filter, it is possible to detect edges in two dimensions, and it is easy to detect edges such as arcs, arcs, and ellipses.

  Note that the line-of-sight direction detection process described in the second embodiment may be used for the above-described line-of-sight direction detection process. That is, the line-of-sight direction may be determined from the head posture determined in the present embodiment and the line-of-sight direction determined in the second embodiment.

  Further, as shown in the first embodiment, when the first camera 11 is fixed to a part of the user's head, the line of sight is detected without detecting the head posture in the present embodiment. Only direction detection may be performed.

  Hereinafter, an experiment using the visual line control device according to the present embodiment will be described. As hardware of the measurement equipment, a 2.66 GHz CPU, a computer having 2 GB memory, and a 640 × 480 pixel IR camera were used.

  FIG. 27 and FIG. 28 show experimental results using the visual line control device according to the present embodiment. FIG. 27A is a graph showing the eye detection processing time. As described above, the method of the line-of-sight control device according to the present embodiment detects eyes once by the Viola-Jones method, and performs template matching using the detected eyes as a template, so that it is markedly compared with other methods. It can be seen that the processing speed is increasing. The processing time is 8.21 ms, which is a 100% success rate.

  FIG. 27B is a graph showing the stability of the cursor. The cursor may not be stable depending on the lighting conditions, tiger movement of the line of sight, and algorithm. From the graph, in the bird's eye view of FIG. 22, the case of 6 points (4 points at the end of both eyes and 2 points at the end of the mouth) is the most unstable. This is because the number of points increases the degree of freedom and multiple surfaces are detected. On the other hand, the case of two points (two points at the end of the eye) is stable, but the two points have a problem in terms of accuracy because the surface cannot be uniquely identified. Therefore, the surface can be uniquely identified, and the cursor is stable at 4 points (2 points at both eyes and 2 points at the mouth) or 3 points (2 points at both eyes and 1 point at the center of the mouth). It can be seen that it is better to detect a bird's eye view.

  FIG. 27C is a graph showing the time until the selection key hits in each of the blink mode (mode in which selection is determined by blinking) and timer mode (mode in which selection is determined by gaze). Here, the hit time when a predetermined key is selected from 20 different keys is shown. From the graph, it is understood that when the predetermined number of keys is small, the hit time is shorter in the blink mode, and when the predetermined number of keys is increased, the hit time is shorter in the timer mode. For this reason, it is preferable that the settings of the timer mode and the blink mode can be changed according to the usage status of the user.

  FIG. 27D illustrates an example of a display screen. On the screen, a face image, an eye periphery image, a screen showing the RUN status of the program, a keyboard, and input software are displayed, and the user gazes at an arbitrary key on the keyboard or directs the line of sight. By blinking, the selected key is input and described in the input software.

  FIG. 28 is a table showing the allowable range of the face posture angle. FIG. 28A shows a roll direction and a yaw direction when a bird's-eye view is detected at two points, FIG. 28B detects a bird's-eye view at four points, and FIG. 28C detects a bird's-eye view at six points. The permissible range of each angle in the pitch direction and the permissible range of the distance from the camera are shown. It can be seen that the roll direction and the yaw direction are almost bilaterally symmetric, and a certain degree of posture is allowed. Regarding the pitch direction, the upward direction is good, but the allowable value in the downward direction is small. This is probably because the camera is attached to the top of the screen.

  Therefore, by adjusting the fixed position of the camera in accordance with the user, the angle of the vertical object can be adjusted as an allowable value in the pitch direction, and it is possible to cope with a change in head posture. Here, as also shown in the second embodiment, if the camera is installed below, the influence of eyelashes becomes large, and detection may not be possible, and fine adjustment is required. As for the distance, for example, in the case of 4 points, 25 cm to 54 cm is allowed, so it can be seen that the line of sight can be accurately detected even when the head posture is considerably changed.

  Although the present invention has been described with the above embodiments, the technical scope of the present invention is not limited to the scope described in the embodiments, and various modifications or improvements can be added to these embodiments. . And embodiment which added such a change or improvement is also contained in the technical scope of the present invention. This is apparent from the claims and the means for solving the problems.

1 Gaze control device 10 PC
11 (first) camera 11a lens 11b LED
DESCRIPTION OF SYMBOLS 12 Light source 13 2nd camera 14 Ultrasonic sensor 15 USB interface 16 Wheelchair control apparatus 17 Electric wheelchair 21 Input part 22 Purkinje image acquisition part 23 Corneal curvature radius calculation part 24 Corneal curvature center calculation part 25 Pupil center calculation part 26 Gaze direction determination Unit 30 line of sight detection unit 31 operation command analysis unit 32 correspondence information unit 33 drive unit 34 obstacle detection unit 35 pupil change analysis unit 36 correspondence information correction unit 40 line of sight control unit

Claims (13)

A line-of-sight control device that controls a control object based on a line-of-sight direction,
Gaze direction detecting means for detecting the gaze direction;
Correspondence information storage means for storing correspondence information that associates the line-of-sight direction with the operation command of the control object;
Action command analysis means for analyzing the action command of the control object based on the correspondence information stored in the correspondence information storage means and the line-of-sight direction detected by the line-of-sight direction detection means;
A line-of-sight control device comprising: drive means for driving the control object based on the result of analysis by the operation command analysis means. The line-of-sight control device according to claim 1.
The control object is an electric wheelchair;
The line-of-sight control device, wherein the correspondence information storage means stores the correspondence information with an operation command corresponding to a case where the line-of-sight direction is a downward direction as forward acceleration. The line-of-sight control device according to claim 2,
The correspondence information storage means sets the operation command corresponding to the case where the line-of-sight direction is the right direction as a right turn, sets the operation command corresponding to the case where the line-of-sight direction is the left direction, and sets the operation direction corresponding to the left direction. A line-of-sight control apparatus, wherein the movement information corresponding to the case is set as forward, the movement command corresponding to the case where the line-of-sight direction is in other cases is stopped, and the corresponding information is stored. The line-of-sight control device according to any one of claims 1 to 3,
Imaging means for imaging at least the front of the control object;
Obstacle detection means for detecting an obstacle based on imaging information captured by the imaging means,
The line-of-sight control device, wherein the driving unit stops the movement of the control target when the obstacle detecting unit detects an obstacle. The line-of-sight control device according to any one of claims 1 to 4,
An ultrasonic sensor for measuring a distance from an obstacle by oscillating an ultrasonic wave at least in front of the control object;
The line-of-sight control device, wherein the driving means stops the movement of the control object when the distance from the obstacle detected by the obstacle sensor is equal to or less than a predetermined length. The line-of-sight control device according to any one of claims 1 to 5,
Pupil size detection means for detecting the size of the pupil;
Based on the size of the pupil detected by the pupil size detecting means, and a pupil change analyzing means for analyzing a change in the size of the pupil,
The line-of-sight control device, wherein the driving unit stops the movement of the control target based on a change in the size of the pupil analyzed by the pupil change analyzing unit. The line-of-sight control device according to any one of claims 1 to 5,
Pupil size detection means for detecting the size of the pupil;
Based on the size of the pupil detected by the pupil size detecting means, and a pupil change analyzing means for analyzing a change in the size of the pupil,
Corresponding information correcting means for correcting correspondence information between the line-of-sight direction stored in the correspondence information storing means and the operation command of the control object based on a change in the size of the pupil analyzed by the pupil change analyzing means. A line-of-sight control device. The line-of-sight control device according to any one of claims 1 to 7,
The gaze direction detection means
A camera that images at least the direction of the eyeball;
A plurality of light sources that emit light at least in the eyeball direction;
Corneal curvature radius calculating means for calculating a corneal curvature radius based on a plurality of Purkinje images on the corneal surface obtained by the light irradiated by the light source;
A corneal curvature center calculating means for calculating a corneal curvature center based on the corneal curvature radius calculated by the corneal curvature radius calculating means, the position of the camera, and the position of the light source;
Pupil center calculating means for extracting a pupil region based on the presence or absence of reflection of light emitted by the light source and calculating the center of the pupil by elliptic approximation;
A line-of-sight control device comprising: a pupil center calculated by the pupil center calculating unit; and a line-of-sight direction determining unit that determines a vector passing through the corneal curvature center calculated by the corneal curvature center calculating unit as a line-of-sight direction. The line-of-sight control device according to claim 8,
The line-of-sight control apparatus, wherein the light source emits at least near-infrared rays, and the camera has at least near-infrared sensitivity distribution. The line-of-sight control device according to any one of claims 1 to 9,
The line-of-sight control device, wherein the camera is fixed to a part of a person who operates the control object. The line-of-sight control device according to any one of claims 1 to 9,
Facial feature extraction means for extracting a plurality of facial features from a facial image;
Feature point determining means for determining a feature point in each face feature extracted by the face feature extracting means;
Normal direction determining means for determining a normal direction of a plane formed by the feature points,
A line-of-sight control apparatus that determines a line-of-sight direction based on a line-of-sight direction detected by the line-of-sight detection unit and a normal direction determined by the normal line determination unit. A computer is a visual line control method for controlling an object to be controlled based on a visual line direction,
A gaze direction detecting step for detecting the gaze direction;
Correspondence information that associates the line-of-sight direction with the operation command of the control object is stored, and the operation command of the control object is analyzed based on the stored correspondence information and the line-of-sight direction detected by the line-of-sight direction detection step. Operation command analysis step to perform,
A line-of-sight control method, comprising: a driving step of driving the control object based on a result of the analysis of the motion command analysis step. A line-of-sight control program that causes a computer to function to control an object to be controlled based on a line-of-sight direction,
Gaze direction detection means for detecting the gaze direction,
Correspondence information storage means for storing correspondence information that associates the line-of-sight direction with the operation command of the control object;
An operation command analysis unit that analyzes an operation command of the control object based on the correspondence information stored in the correspondence information storage unit and the line-of-sight direction detected by the line-of-sight direction detection unit;
A line-of-sight control program for causing a computer to function as drive means for driving the control object based on the result of analysis by the operation command analysis means.