JP5077879B2 - Gaze input device, gaze input method, and gaze input program - Google Patents

Description

  The present invention relates to a line-of-sight input apparatus that performs an input operation by detecting the line-of-sight direction of an input operator, and more particularly to a line-of-sight input apparatus that can perform calibration in a short time.

  When operating a computer, it is common to use a keyboard or mouse. However, there are many people who cannot operate the keyboard and mouse because their limbs are inconvenient, and computers are operated by voice recognition and line-of-sight input. As a technique using line-of-sight input, there is a technique shown in Patent Document 1.

The technique shown in Patent Document 1 allows a subject to open and close eyes from an image that captures the entire face of the subject, obtains a difference image, registers an eye and eyebrow template, and obtains data and an input image obtained during calibration. Is a technique for detecting the direction of the line of sight by comparing the data obtained by the above. Thereby, an intended item on the display screen can be selected, and only an eyeball function by eye movement and eyelid opening / closing operation can be input, and a virtual board can be operated by a switching operation.
Japanese Patent No. 3673835

  However, the technique disclosed in Patent Document 1 has a problem that it takes time and effort to process because it is necessary to prepare a template in which the line of sight is aligned for each item in order to detect the line-of-sight direction.

  In addition, in order to reduce the detection error of the gaze direction, the relative distance between the reference point and the black eye is obtained using the upper left eyebrow with little change with respect to the eye movement as the reference point, and the gaze direction is detected according to the distance. However, if the eyebrows move in accordance with the movement of the eyeball, the reference is shifted and an error occurs. Moreover, although position correction is performed for subtle movements of the head, there is a problem that the processing becomes complicated.

  Further, it is necessary to display as many items as important to the user as possible on the display screen. However, if the number of items displayed on the display screen increases, it is necessary to register more templates, and much time is spent for calibration, which increases the mental and physical burden on the user. . In addition, as the number of items displayed on the display screen increases, the line-of-sight difference between the items decreases, which causes a problem of causing an error.

  Furthermore, when the operating environment changes, an error occurs in the detection of the line-of-sight direction, so that recalibration needs to be performed. In particular, when the number of items displayed on the display screen is large, an error occurs due to a slight positional deviation or environmental difference, and therefore recalibration must be performed frequently. However, since the calibration requires a lot of time as described above, there is a problem that the burden on the user increases.

  The present invention has been made in order to solve the above-described problems, and includes a line-of-sight input device, a line-of-sight input method, and a line-of-sight input program capable of performing line-of-sight input processing with high accuracy while reducing processing necessary for calibration. The purpose is to provide.

(1. Blink input by scanning)
The line-of-sight input device according to the present invention is a line-of-sight input device that acquires an eyeball peripheral image of an input operator, detects a line-of-sight direction from the eyeball peripheral image, and inputs an input element displayed on a display screen. A blink detection means for detecting a blink of a person, a switching display means for switching and displaying the input element displayed on the display screen based on a result detected by the blink detection means, and a switching display means for switching. Among the plurality of displayed input elements, an index movement control means for sequentially moving and controlling an index for specifying any one or a plurality of input elements, and an input element specified by the index whose movement is controlled by the index movement control means And an input element determination unit that determines the blink detected by the blink detection unit by inputting the blink as deterministic information.

  As described above, in the present invention, since the input element displayed on the display screen can be input simply by detecting the blink of the input operator, the line-of-sight input device can recognize only the eyeball of the input operator. Just register. In other words, it is not necessary to prepare a plurality of templates in the line-of-sight direction at various angles according to the input elements, and the processing can be performed easily and efficiently.

Also, even if the blink recognition accuracy is low due to changes in the operating environment (for example, natural light entering the room or movement of the input operator), only blink detection needs to be calibrated. Calibration can be performed easily.
Furthermore, since the input operator can perform various input processes only by blinking, the operation can be performed easily and easily, and the burden on the input operator can be reduced.

  Furthermore, since the indicator movement control means is provided, the movement of the indicator can be controlled. For example, when the screen display is switched by the switching display means, the index may be automatically moved, or the movement of the index is started with the detection of blink (two or more blinks) as a trigger. May be. Further, control may be performed to stop the movement of the index by detecting blinks (two or more blinks). That is, the control of the movement of the index can be freely performed by combining it with the blink detection, and the input operator can also set the control to suit his / her own.

(2. Detecting left and right eyes)
The line-of-sight input device according to the present invention includes line-of-sight detection means for detecting at least one of the left and right lines of sight of the input operator, and display control means for controlling screen display based on a result detected by the line-of-sight detection means. It is characterized by providing.

  As described above, in the present invention, at least one of the left and right gazes of the input operator is detected, and the screen display is controlled based on the result, so at most two gaze directions are registered in the gaze input device. You just have to. That is, it is not necessary to prepare a large number of templates in the line-of-sight direction at various angles according to the input elements, and the processing can be performed easily and efficiently.

  Further, since it is possible to control the screen display that combines the line-of-sight direction and the blink, various operations can be realized. For example, when the line of sight is directed to the right, the index on the screen is moved forward with respect to the input element, and when the line of sight is directed to the left, the index on the screen is moved to the left with respect to the input element. By moving to the reverse direction and performing a definite input by blinking, all operations can be performed with simple control.

Furthermore, since it is sufficient to recognize the line of sight in the left and right directions at most, errors in detection of the line of sight direction can be extremely reduced, and the input operator can perform the line-of-sight input operation with high accuracy and reliability.
Furthermore, even if blinking occurs due to changes in the operating environment as described above, or the recognition accuracy of the line-of-sight direction is low, calibration of blink detection and detection of the line-of-sight direction in at most two directions can be performed. Therefore, calibration can be performed easily in a short time.

  In addition, the movement of the index is to keep the forward (or reverse) movement while the line of sight is facing right (or left), and the movement of the index is when the line of sight is other than right (or left). You may make it control which stops.

In the above case, the moving speed of the indicator may be changed according to the time for staring in a predetermined direction. For example, if the user keeps looking to the right, control is performed so that the moving speed of the index increases according to the time.
Furthermore, the line-of-sight direction may be detected by detecting only the right or left line of sight.

(3. Upper and lower line of sight detected)
A line-of-sight input device according to the present invention includes line-of-sight detection means for detecting at least one line of sight of the input operator, and display control means for controlling screen display based on a result detected by the line-of-sight detection means. It is characterized by providing.

As described above, in the present invention, since it is sufficient to register at most two line-of-sight directions in the line-of-sight input device as described above, a large number of templates of line-of-sight directions with various angles corresponding to input elements are prepared. There is no need, and processing can be performed easily and efficiently.
In addition, since the screen display can be controlled by combining the line-of-sight direction and the blink as described above, various operations can be realized.

Furthermore, since it is sufficient that the line of sight in the upper and lower directions can be recognized as much as the above, the detection error of the line of sight can be extremely reduced, and the input operator can perform the line of sight input operation with high accuracy and reliability. it can.
Furthermore, even when blinking or recognition of the line-of-sight direction cannot be recognized due to a change in the operating environment as described above, it is sufficient to perform calibration for blink detection and detection of the line-of-sight direction in at most two directions. Calibration can be performed easily in a short time.

  As for the movement of the index, the forward (or reverse) movement is maintained while the line of sight is facing up (or down), and the movement of the index is moved when the line of sight is facing other than up (or down). You may make it control which stops.

In the above case, the moving speed of the indicator may be changed according to the time for staring in a predetermined direction.
Furthermore, the line-of-sight direction may be detected by detecting only one of the upper and lower lines of sight.

(4. Detect vertical, horizontal, and gaze)
In the line-of-sight input device according to the present invention, the line-of-sight detection means detects the line of sight of at least four directions of the input operator in the upper, lower, left, and right directions, and the display control means is a screen based on a result detected by the line-of-sight detection means. The display is controlled.

As described above, in the present invention, the display operator detects the line of sight of at least four directions of the input operator in the vertical and horizontal directions, and the display controller controls the screen display based on the result detected by the line of sight detector. Like the cross key, the input operator can freely move up, down, left, and right, and the input operator's mental and physical burden can be reduced by saving the time and labor of the input operation.
In addition, since the line-of-sight directions to be registered are four directions, the time and labor required for registration and calibration can be significantly reduced as compared with the conventional case, and the extreme position of up, down, left and right is detected, so the detection accuracy is high, A line-of-sight input operation can be performed reliably.

(5. Detection of blinking)
The line-of-sight input device according to the present invention includes: a skin color detection unit that detects a skin color part in the eyeball peripheral image; a skin color part detected by the skin color detection unit; and a white eye part and a black eye part in the eyeball peripheral image. A pixel value average calculating means for calculating an average of pixel values of peripheral images, a luminance calculating means for calculating the brightness of the eyeball peripheral image from the eyeball peripheral image, and an average of pixel values calculated by the pixel value average calculating means. Threshold calculating means for calculating an optimum threshold value for determining whether the average of the pixel values is black or not from the luminance calculated by the luminance calculating means, and the blink detecting means sets the threshold value calculated by the threshold calculating means to Based on this, the blink of the input operator is detected.

As described above, in the present invention, the average pixel value of the peripheral image of the eyeball is calculated from the skin color portion, the black eye portion, and the white eye portion of the skin of the input operator, and the average of the pixel value and the environment such as illumination in the operation environment Based on the luminance calculated from the above, it is not necessary to compare the image data as in the conventional case, and it is possible to detect the blink reliably with simple processing. it can.
In addition, since blinking is detected in consideration of the magnitude of luminance, it is possible to cope with changes in the environment such as natural light or illumination inserted through the window.

  Furthermore, since an optimal threshold value is calculated from the average of the pixel values in the eyeball peripheral image, an image captured by visible light is sufficient for the eyeball peripheral image, and the apparatus can be made inexpensive and compact.

(6. Discrimination between intentional blink and natural blink)
In the line-of-sight input device according to the present invention, the blink detection unit distinguishes between blinks naturally performed by the input operator and intentionally performed blinks, and detects only intentionally performed blinks. It is characterized by that.

  Thus, in the present invention, from the blink time found by the inventors, in order to distinguish the intentionally performed blink from the naturally performed blink, the naturally performed blink is ignored. Thus, it is possible to perform an input operation by detecting only the blinks made automatically.

(7. Parameter setting)
The line-of-sight input device according to the present invention is characterized in that the input operator can freely set processing parameters in each processing means.
Thus, in the present invention, since the input operator can set the processing parameters, the line-of-sight input device can be customized corresponding to the input operator, and various input operators can use it comfortably. be able to.
Each parameter setting may be registered in advance in a parameter file, or may be changed at any time while an input operator performs an input operation.

(8. Virtual keyboard input)
The line-of-sight input device according to the present invention is characterized in that the input element has the same configuration as a keyboard of a personal computer.
Thus, in the present invention, since the input elements have the same configuration as the keyboard of a personal computer, the same operation as that using the keyboard can be performed by line-of-sight input.
Note that the operation of simultaneously pressing a plurality of keys can be realized by setting a hold state when blinking is performed twice, for example.

Further, for example, a mouse operation can also be performed by right-clicking only the blink of the right eye and left-clicking of the blink of only the left eye.
Furthermore, all the operations other than the mouse operation described above can be realized if one of the left and right eyeball peripheral images can be acquired.

The line-of-sight input device according to the present invention is a line-of-sight input device that acquires an eyeball peripheral image of an input operator, detects a line-of-sight direction from the eyeball peripheral image, and inputs an input element displayed on a display screen. A blink detection means for detecting a blink of a person, a switching display means for switching and displaying the input element displayed on the display screen based on a result detected by the blink detection means, and a switching display means for switching. An index movement control means for sequentially and sequentially controlling an index for identifying an arbitrary one or a plurality of input elements among the plurality of displayed input elements, to the input elements displayed adjacent to each other; and the index An input element determining means for determining an input element specified by the index controlled to move by the movement control means by inputting the blink detected by the blink detection means as definite information; Wherein the index movement control means is for controlling by changing the moving speed of the indicator according to the number of the detected blink.
The line-of-sight input device according to the present invention is a line-of-sight input device that acquires an eyeball peripheral image of an input operator, detects a line-of-sight direction from the eyeball peripheral image, and inputs an input element displayed on a display screen. A blink detection means for detecting blinks of an input operator, a switching display means for switching and displaying the input elements displayed on the display screen based on a result detected by the blink detection means, and a switching display means Among a plurality of input elements displayed by switching, the index movement control means for sequentially moving and controlling an index for specifying any one or a plurality of input elements, and the index controlled by the index movement control means are specified. Input element determining means for determining the input element by inputting the blink detected by the blink detecting means as confirmation information, and the input element has a confirmation It includes a HOLD button to maintain the information remains in the input state, if you enter the HOLD button as confirmation information, a plurality of the confirmation information is input at the same time.
Furthermore, the line-of-sight input device according to the present invention obtains an eyeball peripheral image of the input operator, detects a line-of-sight direction from the eyeball peripheral image, and inputs an input element displayed on a display screen. A blink detection means for detecting blinks of an input operator, a switching display means for switching and displaying the input elements displayed on the display screen based on a result detected by the blink detection means, and a switching display means Among a plurality of input elements displayed by switching, the index movement control means for sequentially moving and controlling an index for specifying any one or a plurality of input elements, and the index controlled by the index movement control means are specified. Input element determination means for determining an input element by inputting the blink detected by the blink detection means as definite information, and a line of sight for detecting the line of sight of the input operator And when the line-of-sight direction of the input operator is a preset direction, the index is moved and controlled in the line-of-sight direction, and the line-of-sight is detected while the line-of-sight direction is detected. are sequentially controlled to move successively to the input elements displayed arranged in the viewing direction is shall be controlled by changing the moving speed of the indicator according to the time has been detected continuously.
Although the present invention has been described as an apparatus, the present invention can be understood as a system, method, and program as will be apparent to those skilled in the art. These outlines of the invention do not enumerate the features essential to the present invention, and a sub-combination of these features can also be an invention.

  Embodiments of the present invention will be described below. The present invention can be implemented in many different forms. Therefore, the present invention should not be construed based only on the description of the present embodiment. Also, the same reference numerals are given to the same elements throughout the present embodiment.

  In this embodiment, the apparatus will be mainly described. However, as will be apparent to those skilled in the art, the present invention can be implemented as a system, a method, and a program for operating a computer. In addition, the present invention can be implemented in hardware, software, or hardware and software embodiments. The program can be recorded on any computer-readable medium such as a hard disk, CD-ROM, DVD-ROM, optical storage device, or magnetic storage device. Furthermore, the program can be recorded on another computer via a network.

(First embodiment of the present invention)
(1. Configuration)
(1-1 Hardware configuration of line-of-sight input device)
FIG. 1 is a schematic diagram illustrating a hardware configuration of the line-of-sight input device 100 according to the present embodiment.
The line-of-sight input device 100 according to the present embodiment includes a line-of-sight input device main body 110, a camera 106, a display 107, and a speaker 108. The line-of-sight input device main body 110 is connected to a CPU (Central Processing Unit) 101, a RAM 102, a ROM 103, an HDD (Hard Disk Drive) 104, a CD-ROM drive 105 for reading data from a CD-ROM 120, and a network. LAN interface 109.
FIG. 1 is only an example schematically showing the hardware configuration of the line-of-sight input processing apparatus 100 according to the present embodiment, and various other configurations can be adopted as long as the present embodiment is applicable. .

The line-of-sight input device 100 can be constructed by copying the line-of-sight input program to the HDD 111 and performing so-called installation (installation shown here is merely an example) in which the copied line-of-sight input program can be loaded. Then, when the user instructs the OS (Operating System) that controls the computer to start the line-of-sight input device 100, the line-of-sight input program is loaded and started.
The line-of-sight input device program may be provided from a recording medium such as a CD-ROM, or may be provided from another computer connected to the network via the LAN interface 109.

(1-2 Overall module configuration of line-of-sight input device)
2 is a module configuration diagram of the entire line-of-sight input device 100 according to the present embodiment, FIG. 3 is a module configuration diagram of the preprocessing unit 210, and FIG. 4 is a module configuration diagram of the blink registration processing unit 323.

(1-2-1 Configuration and Function of the Line-of-Sight Input Device Main Body 110)
2, the line-of-sight input device main body 110 includes a preprocessing unit 210, a blink detection processing unit 220, a line-of-sight detection processing unit 230, a display control processing unit 240, and a screen display processing unit 250. The display control processing unit 240 includes a switching display processing unit 241, an input element determination processing unit 242, a cursor movement control unit 243, and a cursor speed control unit 244.

The function of each processing unit will be described below.
The preprocessing unit 210 performs a process before performing the line-of-sight input process. Pre-processing at start-up and processing at calibration are performed by the pre-processing unit 210. The detailed configuration of the preprocessing unit 210 will be described later.
The blink detection processing unit 220 detects blinks. Here, processing for determining whether or not the input operator M has blinked from the blink information registered in the preprocessing unit 210 is performed.

  The line-of-sight detection processing unit 230 performs a process of specifying the line-of-sight direction of the input operator from the gazing point of the input operator M and determining whether the line-of-sight direction matches the registered line-of-sight direction. If they match, it is determined that the input operator is looking to the right or left, and if they do not match either, it is determined that the line of sight is pointing in a direction other than right or left.

  The switching display processing unit 241 performs a process of switching the display of the input elements displayed on the screen according to the blink of the input operator M. For example, if a line is displayed on the screen such as “A, K, S, T, ... Is switched to “sa, shi, shi, se, so”.

  The input element determination processing unit 242 performs a process of determining by inputting the selected input element by blinking. For example, in the above example, if blinking is performed with “shi” selected, “shi” is determined, and the input character is “shi”.

  The cursor movement control unit 243 controls the movement of the cursor. For example, when the input operator M looks to the right, the cursor that identifies the input element displayed on the screen is moved forward by one, and when the input operator M looks to the left, it is displayed on the screen. Control is performed such that one cursor that identifies the input element is moved backward.

The cursor speed control unit 244 controls the moving speed of the cursor. For example, while the input operator M is looking to the right, control is performed to increase the moving speed of the cursor, or control to change the moving speed according to the number of blinks.
The screen display processing unit 250 performs processing for displaying input elements and a cursor on the screen.

(1-2-2 Configuration and Function of Preprocessing Unit 210)
In FIG. 3, the preprocessing unit 210 includes an eyeball peripheral image acquisition processing unit 310 and a registration processing unit 320. The registration processing unit 320 includes a right gaze direction registration processing unit 321, a left gaze direction registration processing unit 322, and a blink registration processing unit 323.

The function of each processing unit will be described below.
The eyeball peripheral image acquisition processing unit 310 performs processing for acquiring an image around the eyeball from the face image of the input operator M captured by the camera 106. Based on this image, the line-of-sight direction and blinking are detected.

The right gaze direction registration processing unit 321 detects the right gaze direction from the gazing point of the input operator M, and performs processing for registering the state (position) of the eyeball in that case.
The left gaze direction registration processing unit 322 detects the left gaze direction from the gazing point of the input operator M, and performs processing for registering the state (position) of the eyeball in that case.
The blink registration processing unit 323 calculates and registers a threshold value for determining the pixel value of the image when the eyelid is closed and the pixel value of the image when the eyelid is not closed. The detailed configuration of the blink registration processing unit 323 will be described later.

(1-2-3 Configuration and Function of Blink Registration Processing Unit 323)
In FIG. 4, the blink registration processing unit 323 includes a skin color detection processing unit 410, a pixel value average calculation processing unit 420, a luminance calculation processing unit 430, and a threshold value determination processing unit 440.

Below, the function of each processing unit will be described.
The skin color detection processing unit 410 performs processing for detecting a skin color portion in the eyeball peripheral image m. Here, it is assumed that a pixel indicating a general skin color pixel value from the hue angle (from 6 degrees to 38 degrees) is a skin color portion.

  The pixel value average calculation processing unit 420 performs processing for calculating the average of the pixel values of the eyeball peripheral image m. At this time, the skin color portion detected by the skin color detection processing unit 410 is extracted as black, the eyeball peripheral image m is binarized, and the average of the pixel values is calculated. That is, when the eyelid is opened, the average pixel value is increased due to the presence of the white eye portion. On the other hand, when the eyelid is closed, there is no white eye portion, so the average pixel value is low.

  The luminance calculation processing unit 430 calculates the luminance of the eyeball peripheral image m. The average pixel value varies depending on the luminance of the eyeball peripheral image m. That is, in order to determine whether the heel is closed or open, it is necessary to determine in consideration of luminance.

  The threshold value determination processing unit 440 performs processing for determining an optimum threshold value for determining whether the heel is closed or open. Here, an approximate expression is obtained from the average of the pixel values calculated by the pixel value average calculation processing unit 420 and the luminance calculated by the luminance calculation processing unit 430, and the threshold value is determined.

  In the above configuration, the processing related to the operation by detecting the line-of-sight direction may be omitted. That is, for example, when the cursor movement start signal is blinked, the cursor is sequentially moved for each input element at a predetermined interval, and the input operator M enters the selected input element (cursor When the determination is made by blinking, it is possible to operate the apparatus without detecting the gaze direction.

(2 Operation of line-of-sight input device)
FIG. 5 is a flowchart showing processing of the line-of-sight input device 100 according to the present embodiment. First, when the apparatus is activated, preprocessing is performed (step S501). The preprocessing will be described in detail. A detailed flowchart regarding the preprocessing is shown in FIG. First, a face image of the input operator M is captured by the camera 106 from the front side of the input operator M (step S601).

  The imaging of the face image of the input operator M in step S601 will be described in detail below. FIG. 8 is a face image of the input operator M. Here, photographing is performed with a camera (640 × 480) in the visible wavelength region, which can be obtained at low cost. By using an inexpensive camera, the cost of the entire apparatus can be suppressed. The captured image is taken into the line-of-sight input device main body 110 using existing software (DirectShow).

The camera 106 is not limited to a camera in the visible wavelength region, and may be anything that can capture the face of the input operator, such as a camera using infrared rays.
Also, the software for taking the captured face image into the line-of-sight input device main body 110 is not limited to the DirectShow, and any software may be used.

  Returning to FIG. 6, when the face image is captured, an image around the eyeball is extracted and acquired from the face image (step S602). To obtain the image around the eyeball, Paul Viola and Rainer Lienhart's object detection algorithm (Paul Viola and Michael J. Jones. “Rapid Object Detection using a Boosted Cascade of Simple Features”. IEEE CVPR, 2001) (Rainer Lienhart and Jochen Maydt. “An Extended Set of Haar-like Features for Rapid Object Detection”. IEEE ICIP 2002, Vol. 1, pp. 900-903, Sep, 2002). Use OpenCV to implement this program. OpenCV (Intel Open Computer Vision Library) is an open source library developed and released by Intel Research Laboratories. It is optimized for Intel CPU and can perform various image processing. The official release version was released in November 2006 and can be used on computers such as Windows (registered trademark), Linux, and MacOS. From the position of the black eye in the eyeball peripheral image m acquired in this way, the position of the black eye in the right gaze direction and the left gaze direction is registered (steps S603 and S604).

  The line-of-sight direction registration processing in steps S603 and S604 will be described in detail below. FIG. 9 is a schematic diagram showing a state in which the line-of-sight direction is registered. If the coordinates of the point that the input operator M is gazing at are (p, q), the values of p and q can be obtained by the following equations.

Here, d is the distance between the eyeball and the display 107, re is the eyeball radius, (x, y) is a coordinate indicating the position of the eyeball, ex is the moving distance in the x coordinate direction of the black eye, and ey is the y coordinate direction of the black eye. Indicates travel distance.

  If the coordinates (p, q) of the point being watched by the input operator M are in a predetermined area on the right side of the display 107, it is determined that the input operator M is looking right and the input operator M is watching. When the coordinates (p, q) of the current point are in a predetermined area on the left side of the display 107, it is determined that the input operator M is looking left. In the preprocessing, the position of the black eye when the right side is viewed and the left side is specified, and the state is registered. When the left and right gaze directions are detected, the gaze direction is detected using the position of the black eye registered here.

FIG. 9B shows an actual eyeball peripheral image m when the right is viewed and an actual eyeball peripheral image m when the left is viewed. In the present embodiment, only the left and right are detected as the line-of-sight directions, and the other line-of-sight directions are not necessary. For this reason, since only two line-of-sight directions need be registered in the preprocessing stage, the processing becomes much simpler and processing efficiency can be improved as compared with the prior art.
Here, the two left and right gaze directions are registered here, but only one of the left and right directions may be registered. By doing so, the processing can be further simplified and the efficiency can be increased.

Returning to FIG. 6, when the registration of the line-of-sight direction is completed, blink registration is performed (step S605).
The blink registration process in step S605 will be described in detail below. FIG. 7 is a detailed flowchart regarding blink registration processing. First, the skin color portion of the eyeball peripheral image m is detected (step S701).

  The detection of the skin color portion will be described with reference to FIG. FIG. 10 is a graph showing the hue distribution of the skin color. As can be seen from the graph, the hue angle of the flesh color is approximately between 6 degrees and 38 degrees. According to Jamie Sherrah and Shaogang Gong's Skin Color Analysis from the University of Edinburgh's Department of Informatics, the results of investigating the skin color of various races showed that most people's skin color hues were between 6 ° and 38 °. I know it fits in. That is, in the eyeball peripheral image m, it can be determined that a pixel having a pixel value between 6 degrees and 38 degrees corresponds to a skin color portion (an eyelid or surrounding skin). By extracting the skin color portion, it is possible to binarize the eyeball peripheral image m with white and black by setting the skin color to black.

  Returning to FIG. 7, when the skin color portion is detected, the average of the pixel values is calculated based on the detected skin color portion (step S702). Since the brightness of the eyeball peripheral image m varies depending on the use environment of the apparatus, the brightness of the eyeball peripheral image m is calculated (step S703). Based on the average of the pixel values calculated in step S702 and the luminance calculated in step S703, a threshold for determining whether or not to blink is determined (step S704), and the blink registration process is terminated.

  Hereinafter, the blink registration process will be described in more detail. FIG. 11 is a diagram illustrating a state in which blink registration is performed. FIG. 11A shows the result of calculating the average of the pixel values of the peripheral image m of the eyeball when there is no blinking at various values of luminance (value indicated by L in the figure). As can be seen from the figure, the pixel value of the white eye portion is high when there is no blink, so the average pixel value of the eyeball peripheral image m is high, and when there is no blink, there is no white eye portion, The average pixel value of the image m is lowered. In the graph of FIG. 11B, the portion where the pixel value is low is the moment when blinking occurs. Since the average of the pixel values suddenly decreases at the moment of blinking, the blink of the input operator M can be reliably detected by setting and registering a threshold for the average of the pixel values. FIG. 11C is a graph showing the relationship between the luminance of the eyeball peripheral image and the threshold value. From this graph, it can be derived that blinking can be determined when the pixel value of the eyeball peripheral image m is luminance × 0.3 + 4.7 or less. Therefore, luminance × 0.3 + 4.7 is registered as a threshold value, and this value is used when blink detection is performed.

Returning to FIG. 6, when the blink registration is completed, the preprocessing is terminated.
Returning to FIG. 5, when the preprocessing is completed, a face image of the input operator M is captured by the camera 106 (step S502), and an eyeball peripheral image m is acquired from the captured face image (step S503). The line-of-sight direction of the input operator M is detected from the acquired eyeball peripheral image m (step S504). At the time of detection, the gaze direction is detected from the information on the right gaze direction and the information on the left gaze direction registered in the preprocessing. It is determined whether the right gaze direction or the left gaze direction is detected (step S505). If not detected, the process of step S504 is repeated until it is detected. If it is detected, the cursor is moved (step S506). It is determined whether or not the selected input element is an element that the input operator wants to input (step S507). If the input element is not an input element, the process of step S506 is repeated until the element to be input is selected. Done. When the element to be input is selected, the skin color portion of the eyeball peripheral image m is detected (step S508). The detected skin color portion is “black” and the entire image is binarized, and the average of the pixel values is calculated (step S509). It is determined whether or not the calculated average of pixel values is equal to or less than a threshold value for determining blinking registered in the preprocessing (step S510). If the average of pixel values is greater than the threshold value, it is not determined that blinking is performed, and step S508 is performed. The process is repeated until the blink is detected. If the average pixel value is less than or equal to the threshold value, it is determined that blinking has occurred, and the process proceeds to step S511. In step S511, whether the blink is natural or intentional is determined from the number of frames whose pixel values are equal to or less than the threshold. If natural blink, the process returns to step S508. If it is an intentional blink, the selected input element is determined (step S512), the input content is displayed on the display 107 (step S513), and the process is terminated.

Whether the blink is intentional can be determined by the number of frames of the camera. An intentional blink (slow blink) and a natural blink (fast blink) can be discriminated by the number of frames in which the average value of pixels is equal to or less than the threshold value.
Note that the number of frames for determining blinking can be set for each input operator.

In addition, when detecting the left and right gaze directions or detecting blinks, if nothing can be detected even after a predetermined time has passed, the information registered in the preprocessing may become old and calibration may be necessary. For this reason, calibration may be prompted when a predetermined time elapses in a state of no input in this way.
Furthermore, detection of the left and right gaze directions is not always necessary. That is, as described above, the signal for starting the movement of the cursor is blinked, the cursor is sequentially moved for each input element at a predetermined interval, and the input operator M enters the selected input element in the selected state. When it becomes (when the cursor hits), the determination by blinking makes it possible to operate the apparatus without detecting the line-of-sight direction.

(3 Details of operation of the line-of-sight input device)
The operation of the line-of-sight input device 100 will be described in more detail. FIG. 12 is an example of a display screen displayed on the display 107 of the line-of-sight input device 100. On the top and bottom of the screen, the character key of the first line of 50 sounds (Ah, Ka, Sa, Ta, Na, Hama, La, Wa) is displayed, and in the middle there is a key for executing the process. It is displayed. The “right” character is displayed at the right end of the screen, and the “left” character is displayed at the left end.

  First, since it is necessary to perform preprocessing, the input operator M is prompted by a message to see the “right” character, and the state in which the “right” character is seen is registered. Next, a message prompts the user to see the “left” character, and registers the state in which the “left” character is seen. Then, it prompts with a message to blink, and registers the blinked state. This is the end of the preprocessing.

  When the preprocessing is completed, the screen enters the input mode, and the input operator M can freely input. Here, for example, it is assumed that the cursor is currently on the “ta” key. Further, it is assumed that the input operator M wants to input the character “delicious”. In this case, in order to input “U”, it is necessary to move the cursor to the “A” key. In order to move the cursor to the “A” key, the input operator M looks at the character “Left”. Then, the gaze detection processing unit 230 detects the left gaze direction, and the cursor movement control unit 243 moves the cursor to the left “sa” key based on the detected gaze direction. By repeating this process twice, the cursor can be moved to the “A” key. When the input operator M blinks while the cursor is on the “A” key, the blink detection processing unit 220 detects blinking, and the switching display processing unit 241 switches the input elements displayed on the screen. FIG. 13 shows a screen display of the switched input element.

  In FIG. 13, A row (A, I, U, E, O) is displayed at the top of the screen, and a roar of that row is displayed at the bottom of the screen. Has been. An execution key for “return” processing is displayed in the middle. Now, since the display is switched by the switching display processing unit 241, the cursor selects the first “A” key. In order to input the “u” character, the cursor can be moved to the “u” key and blinked there. In order to move the cursor to the “u” key, the input operator M looks at the character “right”. Then, the gaze detection processing unit 230 detects the right gaze direction, and based on that, the cursor movement control unit 243 moves the cursor to the right “I” key. By repeating this process once more, the cursor can be moved to the “U” key. When the input operator M blinks while the cursor is hitting the “u” key, the blink detection processing unit 220 detects the blink, the input element determination processing unit 242 determines the input element, and the screen display processing unit By 250, the input character is displayed on the display 107. Similarly, by inputting “MA” and “I” characters, the character “delicious” can be displayed on the display 107.

  In this example, the cursor is moved by one when the right or left is viewed once. However, the movement may be maintained while the line of sight of the input operator M is looking at the right or left. By doing so, it is not necessary to look right or left many times, and the burden on the input operator M can be reduced.

  Next, an operation when the line-of-sight direction is not detected in the above process will be described. Since the gaze direction is not detected, only blinking is registered in the preprocessing. In this case, when the screen of FIG. 12 is displayed, the cursor is moved at a predetermined time interval (for example, 1 second) such as “Ah, Sat, Ta ...”. If the input operator M wants to input the “delicious” character in the same manner as described above, the input operator M blinks when the cursor selects the “A” key. Then, the blink detection processing unit 220 detects blinks, and the switching display processing unit 241 switches the input elements on the screen as shown in FIG. And here again, the cursor moves like “Oh, U, U, E ...” at a predetermined time interval (for example, 1 second), so the cursor selects the “U” key. Blink at the moment you do. Then, the blink detection processing unit 220 detects blinks, the input element determination processing unit 242 determines an input character, and the screen display processing unit 250 displays it on the display 107. The “ma” and “i” characters can be entered by performing the same operation.

  FIG. 14 is a diagram of a keyboard. Instead of FIGS. 12 and 13, the virtual keyboard may be displayed in this way so that the cursor moves on the virtual keyboard. By doing so, it is possible to input characters and commands without switching the screen. Further, although not shown in FIG. 14, by adding a HOLD button, for example, Ctrl + Alt + Delete can be input simultaneously. The input at this time may be Ctrl-> HOLD-> Alt-> HOLD-> Delete-> HOLD-> blink.

  In each process, the moving speed of the cursor can be changed. When detecting the gaze direction, for example, while looking right, the movement speed of the cursor gradually increases, and when the speed becomes appropriate for the input operator M, the movement of the cursor is diverted. Speed acceleration can be stopped. This can be easily realized by the line-of-sight detection processing unit 230 detecting the line of sight, and as a result, the cursor speed control unit 244 controls the moving speed of the cursor. Even when the line-of-sight direction is not detected, the cursor speed can be changed according to the number of blinks. For example, if the speed is increased by one step when blinking twice and the speed is decreased by one step after blinking three times, the moving speed of the cursor can be controlled only by detecting the blink.

  In each process, the process of the switching display processing unit 241 can be omitted. For example, to input the character “U”, blinking is performed three times with the “A” key in FIG. 12 selected. To input the character “MA”, blink once with the “MA” key selected. To input the character “I”, blinking is performed twice with the “A” key selected. In other words, characters are input by blinking the number of line numbers (the third line for "U", the first line for "MA", the second line for "I"). Also good.

  Furthermore, when the line-of-sight direction is not detected, the movement of the cursor can be controlled by blinking. When the screen display is switched by the switching display processing unit 241, the cursor may be automatically moved, or for example, the movement of the cursor may be started with the detection of two or more blinks as a trigger. Moreover, you may make it stop conversely. Cursor movement control and speed control can be freely set by combining with blinking, and can be customized according to the preference of the input operator M.

  Furthermore, in this embodiment, the detection of the line-of-sight direction is the left-right direction, but it may be the vertical direction. In that case, for example, if you look up, you can move the cursor forward from left to right (top to bottom), and if you look down, you can reverse the cursor from right to left (bottom to top) The same function as the direction can be realized.

  Furthermore, it becomes possible to move the cursor in four directions by making it possible to detect four gaze directions in the vertical and horizontal directions. For example, when an upward line of sight is detected, the cursor moves up, and when a downward line of sight is detected, the cursor moves down. If the cursor can be moved up, down, left, and right in this manner, the input operator can easily select an input element and the burden can be reduced. In particular, when input processing is performed using a virtual keyboard as shown in FIG. 14, the input efficiency can be significantly increased by moving the cursor up, down, left, and right because there are many input elements.

Furthermore, any direction other than up, down, left, and right may be used as long as it can clearly detect the difference in the line-of-sight direction. For example, detection in an oblique direction or a combination of upper and right directions may be used.
Furthermore, although not shown in the figure, the input mode and the edit mode can be switched by adding a key for changing the mode. In the input mode, characters are input into text as described above. In the edit mode, edit processing such as addition, deletion, and change of characters can be performed on the input text.
Furthermore, in FIG. 12 and FIG. 13, the characters “right” and “left” are not essential displays. The above operation is possible if only the right and left directions can be detected without looking at the characters.

(4 Parameter setting)
In the present embodiment, the input operator M can freely set processing parameters in order to perform an operation suitable for him. Some examples of parameter settings are shown below.
(1) In determining the input element by blinking, the number of blinks can be set. For example, in the case of determination, blink detection is set to twice.
(2) When calibration is performed periodically, the time can be set. For example, if the environment is likely to change, the accuracy may be easily lowered, so the time is set short and the calibration is increased. In the case of the present invention, processing can be performed only by blinking at the minimum, and only blinking and calibration in the four gaze directions (up, down, left, and right) at the maximum. Therefore, the burden on the input operator does not become so great even if the calibration is performed excessively.
(3) When the non-operating time exceeds a predetermined time, the input operator M is prompted to perform calibration, but the non-operating time can be set.
(4) The size and position of the frame when acquiring the eyeball peripheral image m can be set. For example, the input operator's face is different for each person, and depending on the person, the eye position may be above or below, and an eyeball peripheral image may not be obtained accurately. By adjusting the position, an eyeball peripheral image can be obtained accurately.
(5) When calibration is completed, it is possible to set whether to automatically enter the input mode or to set the mode with the mode change key.
(6) It is possible to set whether or not to automatically start a text file (NOTEPAD or the like) when calibration is completed.
(7) The capture size of the camera can be set. Normally, it is set to 640 × 480, but can be set to 320 × 240 when processing is slow.
(8) The capture interval of the camera can be set.
(9) It is possible to set an application that is frequently started by shot cut.

The above is an example of a part of parameter settings.
The parameter can be set not only for the above but also for various operating environments. That is, the apparatus can be customized according to the input operator and the operating environment, and the burden on the input operator can be greatly reduced.

(Second embodiment of the present invention)
(1 Configuration and function)
FIG. 15 is a module configuration diagram of the entire line-of-sight input device 100 according to the present embodiment. The difference from the first embodiment is that an audio output processing unit 260 is newly provided.
The voice output processing unit 260 performs a process of outputting a character input by the input operator M as a voice. Many patients with severe amyotrophic side sclerosis (referred to as ALS) are not only physically disabled but also unable to speak language. Therefore, communication with a person is performed by inputting characters as described above. However, in the present embodiment, since the characters are output as speech, communication by sound can be achieved. Since communication can be achieved with sound, the input operator M can feel secure, and since it is not necessary to perform an act of decoding in communication, it can be performed easily.

  Thus, according to the present invention, the processing and time required for calibration are reduced by extremely simplifying the processing for detecting the line-of-sight direction, and the physical and mental burden of the input operator is reduced. In addition, since the line-of-sight input process can be performed with high accuracy, the number of calibrations can be reduced.

  Although the present invention has been described with the above embodiments, the technical scope of the present invention is not limited to the scope described in the embodiments, and various modifications or improvements can be added to these embodiments. . And embodiment which added such a change or improvement is also contained in the technical scope of the present invention. This is apparent from the claims and the means for solving the problems.

It is the schematic diagram which showed the hardware constitutions of the visual line input device which concerns on 1st Embodiment. It is a module block diagram of the whole eyes | visual_axis input device which concerns on 1st Embodiment. It is a module block diagram of a pre-processing part. It is a module block diagram of a blink registration process part. It is a flowchart which shows the process of the gaze input apparatus which concerns on 1st Embodiment. It is a detailed flowchart regarding preprocessing. It is a detailed flowchart regarding the registration process of blink. It is a face image of the input operator M. It is the schematic diagram which showed a mode that registration of a gaze direction was performed. It is the graph which showed the hue distribution of skin color. It is a figure which shows a mode that registration of blink is performed. It is an example of the display screen displayed on the display of a visual line input device. It is an example of the display screen of an input element when the display screen of FIG. 12 is switched. It is a figure of a keyboard. It is a module block diagram of the whole visual line input device which concerns on 2nd Embodiment.

Explanation of symbols

101 CPU
102 RAM
103 ROM
104 HDD
105 CD-R drive 106 Camera 107 Display 108 Speaker 109 LAN I / F
100 Gaze input device 110 Gaze input device main body 120 CD-ROM
210 Pre-processing unit 220 Blink detection processing unit 230 Gaze detection processing unit 240 Display control processing unit 241 Switching display processing unit 242 Input element determination processing unit 243 Cursor movement control unit 244 Cursor speed control unit 250 Screen display processing unit 260 Audio output processing unit 265 Speaker 310 Eyeball peripheral image acquisition processing unit 320 Registration processing unit 321 Right gaze direction registration processing unit 322 Left gaze direction registration processing unit 323 Blink registration processing unit 410 Skin color detection processing unit 420 Pixel value average calculation processing unit 430 Luminance calculation processing unit 440 Threshold determination processing unit

Claims (9)

In a line-of-sight input device that acquires an eyeball peripheral image of an input operator, detects a line-of-sight direction from the eyeball peripheral image, and inputs an input element displayed on a display screen.
Blink detection means for detecting blink of the input operator;
Switching display means for switching and displaying the input element displayed on the display screen based on the result detected by the blink detection means;
An index for sequentially and sequentially controlling an index for specifying any one or a plurality of input elements among a plurality of input elements displayed by switching by the switching display means, in succession to the input elements displayed adjacent to each other. Movement control means;
Input element determination means for determining an input element specified by the index movement-controlled by the index movement control means by inputting the blink detected by the blink detection means as deterministic information,
The line-of-sight input device, wherein the index movement control means controls the movement speed of the index by changing the number of blinks detected. In a line-of-sight input device that acquires an eyeball peripheral image of an input operator, detects a line-of-sight direction from the eyeball peripheral image, and inputs an input element displayed on a display screen.
Blink detection means for detecting blink of the input operator;
Switching display means for switching and displaying the input element displayed on the display screen based on the result detected by the blink detection means;
Index movement control means for sequentially moving and controlling an index for specifying any one or a plurality of input elements among the plurality of input elements switched and displayed by the switching display means;
Input element determination means for determining an input element specified by the index movement-controlled by the index movement control means by inputting the blink detected by the blink detection means as definite information;
The input element includes a HOLD button that maintains the confirmation information that has been confirmed immediately before in the input state, and when the HOLD button is input as the confirmation information, a plurality of pieces of the confirmation information are simultaneously input. A line-of-sight input device. In a line-of-sight input device that acquires an eyeball peripheral image of an input operator, detects a line-of-sight direction from the eyeball peripheral image, and inputs an input element displayed on a display screen.
Blink detection means for detecting blink of the input operator;
Switching display means for switching and displaying the input element displayed on the display screen based on the result detected by the blink detection means;
Index movement control means for sequentially moving and controlling an index for specifying any one or a plurality of input elements among the plurality of input elements switched and displayed by the switching display means;
An input element determining means for determining an input element specified by the index movement-controlled by the index movement control means by inputting the blink detected by the blink detection means as definite information;
Gaze detection means for detecting the gaze direction of the input operator,
When the line-of-sight direction of the input operator is a preset direction, the index is moved and controlled in the line-of-sight direction, and the index is displayed adjacently while the line-of-sight direction is detected. said continuously input element is sequentially moved control, characterized that you control by changing the moving speed of the indicator according to the time the viewing direction has been detected continuously, the visual axis input device. In a line-of-sight input method of acquiring an eyeball peripheral image of an input operator, detecting a line-of-sight direction from the eyeball peripheral image, and inputting an input element displayed on a display screen,
Computer
Blink detection step for detecting blink of the input operator;
A switching display step of switching and displaying the input element displayed on the display screen based on the result detected in the blink detection step;
An index for sequentially and sequentially controlling an index for specifying any one or a plurality of input elements among a plurality of input elements displayed by switching in the switching display step, in succession to the input elements displayed adjacently. A movement control step;
An input element determination step for determining an input element identified by the index movement-controlled by the index movement control step by inputting the blink detected in the blink detection step as confirmation information, and
The line-of-sight input method, wherein the index movement control step changes the movement speed of the index according to the number of blinks detected. In a line-of-sight input method of acquiring an eyeball peripheral image of an input operator, detecting a line-of-sight direction from the eyeball peripheral image, and inputting an input element displayed on a display screen,
Computer
Blink detection step for detecting blink of the input operator;
A switching display step of switching and displaying the input element displayed on the display screen based on the result detected in the blink detection step;
Among the plurality of input elements that are displayed by switching by the switching display step, the index movement controlling step for sequential movement control indicators to identify any one or more input elements,
An input element determination step for determining an input element identified by the index movement-controlled by the index movement control step by inputting the blink detected in the blink detection step as confirmation information, and
Includes a HOLD button maintained in the input state of the confirmation information determined immediately before the input element, if you enter the HOLD button as confirmation information, and wherein Rukoto plurality of the confirmation information is input at the same time Gaze input method. In a line-of-sight input method of acquiring an eyeball peripheral image of an input operator, detecting a line-of-sight direction from the eyeball peripheral image, and inputting an input element displayed on a display screen,
Computer
Blink detection step for detecting blink of the input operator;
A switching display step of switching and displaying the input element displayed on the display screen based on the result detected in the blink detection step;
An index movement control step for sequentially moving and controlling an index for identifying any one or a plurality of input elements among the plurality of input elements switched and displayed in the switching display step;
An input element determination step for determining an input element specified by the index movement-controlled by the index movement control step by inputting the blink detected in the blink detection step as definite information;
Performing a gaze detection step of detecting a gaze direction of the input operator;
When the line-of-sight direction of the input operator is a preset direction, the index is moved and controlled in the line-of-sight direction, and the index is displayed adjacently while the line-of-sight direction is detected. sequentially controlled to move, the visual axis input method comprising that you control by changing the moving speed of the indicator according to the time the viewing direction has been detected continuously in succession to the input element. In a line-of-sight input program that allows a computer to function to acquire an eyeball peripheral image of an input operator, detect a line-of-sight direction from the eyeball peripheral image, and input an input element displayed on a display screen.
Blink detection means for detecting a blink of the entering force the operator,
Switching display means for switching and displaying the input element displayed on the display screen based on the result detected by the blink detection means ;
An index for sequentially and sequentially controlling an index for specifying any one or a plurality of input elements among a plurality of input elements displayed by switching by the switching display means , in succession to the input elements displayed adjacent to each other. Movement control means ,
Causing the computer to function as an input element determining means for determining an input element specified by the index movement-controlled by the index movement control means by inputting the blink detected by the blink detecting means as definite information ;
The line-of-sight input program, wherein the index movement control means changes and controls the movement speed of the index according to the number of blinks detected. In a line-of-sight input program that allows a computer to function to acquire an eyeball peripheral image of an input operator, detect a line-of-sight direction from the eyeball peripheral image, and input an input element displayed on a display screen.
Blink detection means for detecting blink of the input operator;
Switching display means for switching and displaying the input element displayed on the display screen based on the result detected by the blink detection means;
The switching among the plurality of input elements that are displayed by switching the display means, any one or more sequential movement control to index movement control means an index for specifying the input element,
Causing the computer to function as an input element determining means for determining an input element specified by the index movement-controlled by the index movement control means by inputting the blink detected by the blink detecting means as definite information;
Includes a HOLD button maintained in the input state of the confirmation information determined immediately before the input element, if you enter the HOLD button as confirmation information, and wherein Rukoto plurality of the confirmation information is input at the same time Line-of-sight input program. In a line-of-sight input program that allows a computer to function to acquire an eyeball peripheral image of an input operator, detect a line-of-sight direction from the eyeball peripheral image, and input an input element displayed on a display screen.
Blink detection means for detecting blink of the input operator;
Switching display means for switching and displaying the input element displayed on the display screen based on the result detected by the blink detection means;
Index movement control means for sequentially moving and controlling an index for specifying any one or a plurality of input elements among the plurality of input elements that are switched and displayed by the switching display means;
Input element determination means for determining an input element specified by the index movement-controlled by the index movement control means by inputting the blink detected by the blink detection means as definite information ,
Causing the computer to function as a line-of-sight detection means for detecting the line-of-sight direction of the input operator ;
When the line-of-sight direction of the input operator is a preset direction, the index is moved and controlled in the line-of-sight direction, and the index is displayed adjacently while the line-of-sight direction is detected. said continuously input element is sequentially moved control, eye control program characterized that you control by changing the moving speed of the indicator according to the time the viewing direction has been detected continuously.