KR20080055622A - Human-Friendly Computer I / O System - Google Patents

Abstract

According to the present invention, in a GUI environment where a computer interface between a computer and a human is a graphic object, a stereo vision sensor recognizes a user's facial features and recognizes a user with permission to use the device. To visualize the user's eyes and points of interest, automatically display graphic objects, adjust the resolution of stereoscopic displays, and drive HRTF based 3D audio systems to provide three-dimensional visual and auditory effects To increase work efficiency.

Description

Ergonomic Human Computer Interface

The present invention relates to an interface between a computer (or a computer-like device) and a human, and the computer recognizes a human face and displays the three-dimensional position and posture of the face, the direction of view and the point of interest, and the position of the ear. By providing the user with the proper audiovisual output and the convenient input method.

In general, a computer passively waits for a user's instruction through a keyboard or a mouse, and then expresses a result of performing the indicated task to stimulate vision and hearing among the five senses of the human body regardless of the posture or position of the user. Provide text, video, and sound.

Therefore, the passive stereoscopic display can see the correct image only when the user sees it at a fixed position.

The computer mouse has become a standard input device along with the keyboard, but since the movement of the left and right arms is asymmetrical, the sitting posture is distorted, tiredness becomes easy, and long-term use causes pain in the hands or shoulders.

If the computer recognizes the user, estimates the user's interests and intentions, prepares the candidate candidates in advance, and allows the user to select or confirm, that is, facilitate the user's instruction, thereby increasing work efficiency and convenience.

The prior art that can be used for this purpose can be summarized as follows.

Monocular visual systems identify pixel groups that are clearly distinguished by intensity or intensity of color in an image and compare them with previously stored shapes. Face recognition usually involves five steps:

1. Image Acquisition

2. Detect shapes that look like faces in the acquired image (for example, oval face shape and shape with two eyes and mouth)

3. Template generation by detecting facial features from facial images

4. Search the Data Base for candidate face templates that are similar to the template you created.

5. Confirmation or recognition

The facial pose refers to the direction in which the user's face is directed and is expressed in pitch, yaw, and roll angle, and ultimately, the user's attention is directed.

There are two methods for estimating the posture of a face in three dimensions with a two-dimensional image.

Analytical Geometry-Find the 2D coordinates of the face components in a 2D projected 3D coordinate system and the solution of the coordinate transformation equation where the 3D coordinate system is projected to the 2D image.

Ex) US PN 6,154,559, US PN 6,937,745, US PN 7,121,946

2. Learning-based method-Predict posture by storing samples of two-dimensional images according to posture and posture and finding samples approximating the acquired images.

Ex) US PN 6,471,756

Recently, a method of using stereo images, which have been used in aerial surveys for a long time, has also been proposed.

Yes) US PN 6,188,777

The above techniques are based on the generic head model to estimate the face posture by learning the feature shape and color of the face elements, so that the system can be used for an unspecified number of people. There are many problems.

On the other hand, there have been many researches on the method of estimating the gaze as the position of the eye, but since the human eye keeps wandering to receive new information, there is a problem that it is too sensitive to use as a mouse.

(Note: Eye Controlled Media: Present and Future State: Arne John Glenstrup, Theo Engell-Nielsen, University of Copenhagen, June 1. 1995)

In the present invention, the computer is to find out who the user is, and to measure the position of the user, the posture of the face, and the position of the pupil, to estimate the intention of the user in a GUI (Graphic User Interface) environment, and to indicate the user. It is a technical task to implement a method and apparatus for preparing and presenting a possible command content in advance and outputting an audiovisual effect suitable for a user's eyes and ears based on the user's confirmation or selection. .

In the present invention, as a peripheral device of a computer (or a computer-like machine), two or more speakers, a stereoscopic display providing a two-dimensional flat panel display or a three-dimensional image, and at least two CCD cameras or video at the corners or the periphery of the display. With the camera, the user is based on wearing an eyeglass frame without eyeglasses.

The frame has at least three visual markers to make the camera easier to detect, and the frame has a scale for camera calibration.

Eyeglass frames without eyeglasses are also used to calibrate stereo cameras, and are called head mouse templates because they act as head mice.

1 shows two CCD cameras 5 and 6 at both corners of a two-dimensional display 7 and the user has a Head Mouse Template 4 with three markers (1), (2) and (3). It is worn.

The Head Mouse Template has a flat surface and when placed on the user's face like glasses, it is considered to be parallel to the face reference plane. The face reference plane is a plane formed by both eyes and is perpendicular to the line of sight. For convenience of development, the center of the markers (1) and (2) passes through the line of sight from the center of both eyes.

FIG. 2 is a view from above of the user's head. The axis connecting the left and right CCD cameras along the display plane is called the X axis, and the axis perpendicular to the display and directed toward the Head Mouse Template is called the Z axis.

For one marker 8, the positions of the images projected on two CCD cameras separated by a distance of b are each x _img1 And x _img2 It is different from and, and the position difference is divided by the pixel unit length a of the CCD element and is called disparity d . The distance z from the display to the marker is the distance baseline b between the focal length f of the CCD camera lens and the CCD camera, and is expressed by the following approximation equation. (Note: RANGE AND VELOCITY ESTIMATION OF OBJECTS AT LONG RANGES USING MULTIOCULAR VIDEO SEQUENCES,

N. Scherer, R. Gabler; http://www.isprs.org/istanbul2004/comm5/papers/649.pdf)

z = (b · f) / (d · a)

Thus, if the Head Mouse Template coordinate system and the display coordinate system share a horizontal plane, the markers (1) and (2) on the eyeglass frame obtain the horizontal position of the gaze point of the user's gaze, and the markers (2) and (3) are perpendicular to the gaze point. You can get the location.

FIG. 3 illustrates in two dimensions how to move the origin of the display coordinate system to the center of sensor 1 (left CCD camera) to find the position where the line of sight touches a point on the display.

이다.The position where the images of the markers M1 (1) and M2 (2) are projected on the left CCD camera is x _p1 And x _p2 , The positions of M1 and M2 are (x _p1) z ₁ / f, z ₁ ) and (x _p2 z ₂ / f, z ₂ ), at the center point M _c of the markers The position of the point (9) where the line of sight perpendicular to the line connecting the markers meets the display is

to be.

4 is a diagram illustrating the relationship between the display coordinate system and the head mouse template coordinate system, and is relatively relatively to facial features such as the facial features of the user behind the head mouse template, that is, the edges of the eyebrows, the edges of the eyes, and the ears. The location of unrelated facial features can be measured. In addition, as a specific feature for each user, the location of the wart or the point 10 may be measured. (For users with dots or warts)

If the coordinates of the facial feature k in the Head Mouse Template coordinate system are (x _T ^k , y _T ^k , z _T ^k ), the pixel coordinates (x _cp ^k _L , y _cp) of the image of the CCD camera on the left and right sides of the display coordinate system ^k _L , -f), (x _cp ^k _R , y _cp ^k _R , -f) is expressed by the following equation.

On the other hand, the coordinate system based on the head mouse template and the display coordinate system transformation relationship, that is, the template translated by X _T , Y _T , Z _T from the display coordinate system, and rotated in order around the X, Y, Z axes by θ, φ, ψ in order. The conversion formula of the coordinate system is as follows.

After memorizing the positional coordinates of the user's face features, the point of interest and the point of interest (POI) on the display plane are calculated by measuring the face feature points without the Head Mouse Template and calculating the transformation relationship between the template coordinate system and the display coordinate system. 9) can be obtained.

The binocular (Binocluar) vision system has the advantage of being independent of illumination, distance, and viewpoint compared to the monocular vision, but unlike the human eye, there is a correlation problem in computer vision. In order to solve this problem, camera calibration should be performed. The known method is "Camera Calibration of Stereo Photogrammetric System with One-Dimensional Optical Reference Bar" QU Xu et al, International Sysmposium on Instrumentation Science and Technology (2006) 1048-1052, "A Flexible New Technique for Camera Calibration" Zhengyou Zhang, December 2, 1998. http://research.microsoft.com. Whenever the computer display setting is changed by the scale of the Head Mouse Template, it is convenient to apply the above method.

When initializing a computer (or a computer-like machine), the camera is calibrated with the Head Mouse Template, the POI position on the display is calculated by the Head Mouse Template, the cursor is represented at the calculated POI position, and the user confirms. (feedback), i.e. changing the position of the face if the cursor is not located at the desired position, the computer repeats the position of the cursor in the new position according to the changed face posture, when the cursor reaches the desired position. The user may take the role of a hand mouse by pressing or releasing a predetermined button on the keyboard to indicate an action such as confirmation or selection. This is a stereo photo measurement of the user's facial features during initialization. The coordinates of the facial feature points irrelevant to the facial expressions are read and registered.In the operating system after the initialization, the user's facial features are measured without the Head Mouse Template installed. A specific user can be recognized by comparing with the database of the user, and at the same time, the position and posture of the face reference plane are calculated to estimate the POI on the display and display the cursor.

FIG. 5 illustrates an example of installation of a stereoscopic visual system and a computer peripheral device for providing stereoscopic sound to create a parallax difference by overlapping the same stereoscopic object or different images in the left eye and the right eye. to be. The configuration includes left and right displays 11 and 12 that provide images for the left and right eyes, and mirrors 13 and 14 to reflect them, and four speakers for stereoscopic effect ( 15), (16), (17), (18) and microphones (19), (20) for three-dimensional sound acquisition. The CCD cameras 5 and 6 are located at the center of the mirrors 13 and 16.

FIG. 6 is a plan view of FIG. 5, and is a part of a vision system. FIG. In FIG. 6, area (21) is a stereo-region in which the images of the left eye and the right eye overlap.

POI (9) is represented. 7-a shows the positions of the pupils when the POI is infinity, and FIG. 7-b is the positions of the pupils when the POIs are finite, and the distance to the POIs, i.e., three-dimensional depth, as these pupil position differences. Depth) can be obtained. Therefore, in the stereoscopic display, only the panum's area around ± 10 ° around the POI can increase the resolution, and the others can reduce the resolution to reduce eye fatigue. (See "Foveation for 3D Visualization and Stereo Imaging" Arzu Coltekin, TKK Institute of Photogammetry and Remote Sensing Publication Espoo 2006, Helsingki University of Technology)

During the initialization of the computer, in the step of obtaining and registering the camera correction and coordinates of the user's facial features, a virtual POI is generated to record a change in the position of both eyes of the user while making a near proximity from the infinite depth of depth to create a table. . That is, the user's pupil position difference (22) and (23) are measured with respect to the pupil position at the infinity point of time of FIG. 7-a and the finite distance point of interest (POI) 9 of FIG. A table of pupil position differences is constructed, and after initialization, the operating system measures pupil position differences (22) and (23), and reads the corresponding stereoscopic depth from the table and expresses them at the corresponding positions.

 As perceived by the difference in position (parallax) between the left eye and the right eye for the same point (POI) in vision, the stereoscopic sense of sound is the Interaural Time Difference (ITD). , Interaural Level Difference (ILT) and Head-Related Transfer Function (HRTF). HRTF refers to the reflection sound effect of each frequency due to diffraction and reflection, with sound below 1 KHz and sound above 4KHz through head, torso and outer ear depending on the direction of sound. Therefore, in order to operate the HRTF based 3D Audio System, the signal information for each sound source, the position of the listener and the face posture, and the position of the sound source must be input. (Note: "Efficeient and Effective Use of Low Cost 3D Audio Systems", Kenneth Wang, Proceedings of the 2002 International Conference on Auditory Display, Kyoto, Japan, July 2-5, 2002.

US PN 6,961,439 Method and Apparatus for Producing Spatialized Audio Signals)

In the present invention, in order to input the position of the listener and the face posture, the left ear and the right ear are recognized and the position is measured and inputted, or when both ears are not caught in the image, other feature points of the user's face or the three-dimensional position of the head mouse template and the like. The positional data of the user's ear can be input by recognizing the posture of stereoscopic photogrammetry.

1 is a conceptual view of a user wearing a head mouse template with three markers and looking at a display

FIG. 2 is a plan view of FIG. 1 showing the concept of depth measurement with disparity appearing in two sensors. FIG.

FIG. 3 is a plan view of FIG. 1, which is a conceptual diagram for obtaining a line of sight and a POI. FIG.

4 is a conceptual diagram of measuring facial features by wearing a Head Mouse Template

5 is an illustration of a system that provides stereoscopic vision and stereophonic sound.

6 is a conceptual diagram representing stereoscopic depth in a stereoscopic display.

7 is a conceptual diagram illustrating a position difference of a user's pupil with respect to an infinity view and a finite distance view

Claims (8)

How to manipulate the position of graphical objects such as cursors and pointers on a display on a computer system in a GUI environment, 1: acquiring images of a user's face and head with at least two image acquisition devices (CCD cameras) 2: Recognize three or more visual markers in the Head Mouse Template from the user's face image and calculate the three-dimensional position for each marker from the display with the position difference (disparity) of the marker image pixels on each CCD camera screen. Steps to 3: obtaining a straight line (line of sight) perpendicular to the plane formed by the markers and passing through the center between the user's left and right eyes, and calculating an intersection point (POI) where the straight line meets the display plane 4 .: Displaying graphical objects at the location of the computed POI How to include The method of claim 1, wherein in calculating the three-dimensional position for each marker in step 2, calibration of internal and external variables of the CCD camera is performed in a known manner using markers and scales in the Head Mouse Template. The user's facial database is registered as user information with computer usage rights by obtaining three-dimensional coordinates of the user's facial features (both the ends of the eyebrows, the corners of the eyes, the edges of the lips, the ears, the points, the warts, etc.) How to further include the steps The stereoscopic display according to claim 2, wherein when the display is a stereoscopic display, in the step of registering a user face by obtaining three-dimensional coordinates for the user's face feature, a virtual POI is formed on the stereoscopic display to form a nearest stereoscopic image from infinity stereo depth. The method further includes the step of recording the change of the center position of both eyes while changing to the depth, and adding the stereoscopic visual characteristic data of the corresponding user. The method according to claim 1, wherein in the step of recognizing three or more visual markers in the head mouse template from the face image of the user in step 2, if the head mouse template image cannot be extracted, a match is made in the registered user face database. Recognizing a specific user having permission to use the facial image data, and calculating a three-dimensional position and posture of the face reference plane from the positions of the respective facial feature points. In computer systems, where the position of graphical objects such as cursors or pointers on a display or stereoscopic display screen is in the Human Computer Interface, At least two image acquisition devices (CCD cameras) separated from the user's face so that they are located within the stereo-regieon; Head Mouse Template is mounted parallel to the user's face reference surface and has at least three visual markers and a grid spaced at regular intervals. A logic processor that detects the position of the image pixel on the CCD camera with respect to the facial feature point or visual marker of the user, calculates the three-dimensional position as a disparity, estimates the user's POI position, and expresses the graphical object at the estimated position; Storage media for storing data such as user's facial feature data and stereoscopic visual data Containing system The system according to claim 1, wherein in the case of a system having a stereoscopic display, in the step 3: obtaining the visual direction, the left eye is read by measuring the positions of the left and right pupils and reading the corresponding stereo depth from the user's stereoscopic visual data table. The method further comprises the step of expressing a graphical object at a position corresponding to the corresponding screen of the right eye and the corresponding stereoscopic depth corresponding to the corresponding screen of the right eye, increasing the resolution only near the panum's area of the corresponding area and lowering the resolution of the other area. The system of claim 5, further comprising an HRTF based 3D Audio system consisting of at least two speakers or earphones as a peripheral device of the computer system. The method of claim 7, wherein the method is for driving an HRTF based 3D Audio system. 1: acquiring images of a user's face and head with at least two image acquisition devices (CCD cameras) 2: Recognizes three or more facial feature points or three or more visual markers of a Head Mouse Template in a user's face image, and displays a three-dimensional position from the display as a position difference (disparity) of image pixels on each CCD camera screen. Computation of 3D position and posture of face reference plane 3: Step of obtaining the position of the left and right ears from the position and posture of the face reference plane and inputting it to the HRTF based 3D Audio system How to include

Images