KR20100047793A - Apparatus for user interface based on wearable computing environment and method thereof - Google Patents

Abstract

PURPOSE: An apparatus for a user interface based on a wearable computing environment and a method thereof are provided to support a user-friendly input interface by recognizing the motion of gesture in 3D pattern. CONSTITUTION: A location indicator generates an optical signal near a wrist of a user, and a signal measuring unit(11) receives an optical signal. Plural image measurement units(11a,11b,11c) measure the front image of a user through an image sensor. A signal processor(15) calculates 3D coordinates through image analysis, and recognizes the motion pattern of both hands on the 3D coordinates. The signal processing unit outputs a command corresponding to the motion pattern.

Description

Apparatus for user interface based on wearable computing environment and method

The present invention relates to a user interface device based on a wearable computing environment and a method thereof, and more particularly, a wearable computing device suitable for a wearable computing environment, which can be used as an input of a wearable system or a peripheral device in a three-dimensional space in front of the user. The present invention relates to a user interface device based on a computing environment and a method thereof.

The present invention is derived from a study conducted as part of the IT source technology development project of the Ministry of Knowledge Economy [Task management number: 2008-F-048-01, Task name: Development of Wearable Personal Companion technology for u-computing space collaboration] .

Many attempts have been made to detect the user's movement, especially the hand's movement in the limited space equipped with the system and use it as the interaction between human and computer. Existing systems have the disadvantage that the user may only wear input in the form of a well-equipped, well-equipped device.

In addition, a commercially available device such as a three-dimensional space mouse or a pen measures a user's hand movement using a gyro sensor and uses it as a user input. This device is also inconvenient to be held and used by the user and carried by the user when necessary.

Multi-touch, such as Apple's Ipod Touch, Microsoft's Surface, and Jeff's Multi-Touch device, take full advantage of multi-touch by applying touch to the device's display, but do not hold the device in your hand or use it on a limited device. There is discomfort.

In particular, in the case of a user interface for a wearable system in which a device or a computer is attached to or worn on a body, the user interface should be designed in consideration of factors such as mobility required to carry the device and wearability that a user can easily use and carry.

An object of the present invention for solving the above problems is, wearable computing environment-based user that can be used as input to the wearable system or peripheral devices in the three-dimensional space in front of the user in the same movement as the user's hand in a wearable computing environment An interface device and a method thereof are provided.

The wearable computing environment-based user interface device according to the present invention for achieving the above object, receives a position signal generated from a position indicating device that is worn near the user's wrist to generate a position signal indicating the user's hand position And 3D coordinates are calculated from the signal measuring means for measuring the image in front of the user, and the image measured by the signal measuring means, and the user's hand on the 3D coordinates from the position signal received by the signal measuring means. And signal processing means for recognizing a movement pattern and outputting a corresponding command.

The signal measuring means may include a plurality of image measuring units including an image sensor and a body shaking measuring unit for measuring a degree of shaking of a body from a user's movement.

The body shake measurement unit is characterized in that it comprises an inertial sensor.

The image measuring unit may include a filter for distinguishing the image from the position signal generated from the position indicating device.

The signal processing unit may implement a virtual display screen in an area where the angles of view of the image sensors included in the plurality of image measuring units overlap each other, and calculate three-dimensional coordinates from an image of the virtual display screen.

The signal processing means includes a two-dimensional coordinate calculation unit for extracting two-dimensional coordinates from each of the images measured by the plurality of image measuring units, and the two-dimensional coordinates and the position signal extracted by the two-dimensional coordinate calculation unit. It is characterized in that for calculating the three-dimensional coordinates based on the detected position information of the hand.

The signal processing means further comprises a body shake correction unit for correcting the shake of the image measured from the image measuring unit based on the degree of shaking of the body measured by the body shake measuring unit.

The position indicating device includes a position signal generating means for generating a position signal every time the user moves the position of the user's hand and an input signal measuring means for receiving a control command from the user, and based on the user control command inputted to the input signal measuring means. To control the operation of the position signal generating means.

The position signal generating means is characterized in that for generating the position signal in the form of an optical signal.

On the other hand, the wearable computing environment-based user interface method according to the present invention for achieving the above object, receives a position signal generated from a position indicating device that is worn near the user's wrist to generate a position signal, the user front Measuring the image of the user, calculating three-dimensional coordinates from the image measured in the measuring step, determining the position of the user's hand from the position signal received in the measuring step, and checking the user on the calculated three-dimensional coordinates. Recognizing a movement pattern of the hand, and outputting a command corresponding to the movement pattern recognized in the recognition step.

The measuring may include measuring a degree of shaking of the body from the movement of the user.

The method may further include correcting the shaking of the measured image based on the measured shaking degree of the body.

The calculating may include implementing a virtual display screen in an area where the angles of view of the plurality of image sensors measuring the image overlap each other, and calculating three-dimensional coordinates from the image of the virtual display screen. It is done.

The calculating may include extracting two-dimensional coordinates from each image measured by the plurality of image measuring means, wherein the extracted two-dimensional coordinates are detected by the position signal of the measuring step. The three-dimensional coordinates are calculated based on the position information of the hand.

The position signal is generated by the position indicating device whenever the position of the user's hand moves.

The position signal may be generated based on the input signal when a signal indicating the start and end of the hand movement is input from the user to the position indicating device.

The position signal is characterized in that it is generated in the form of an optical signal.

According to the present invention, when a gesture is performed with both hands in a three-dimensional space in front of the user, the movement is tracked, recognized as a predetermined pattern in three dimensions, and processed, thereby allowing the user to move and use a computer in a wearable computing environment. By supporting a multipoint input function, a method of selecting or manipulating an object on a user display has an advantage of supporting a user-friendly input interface that is like handling an object in a space.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 and 2 are diagrams for explaining a wearable computing environment based user interface device according to the present invention.

1 and 2, the wearable computing environment-based user interface device according to the present invention is mounted near a wrist, so that the position indicating device 20 and the position indicating device 20 for detecting a movement of a hand are provided. And a motion processor 10 for recognizing the movement of the hand from the detected signal and processing the corresponding motion.

First, as shown in FIG. 1, the user controls the display screen 1 on the virtual display screen 2, not the actual display screen 1 such as the wall display device of the wearable computer or the head mounted display (HMD). The hand is moved for.

Here, the movement of the hand corresponds to everything that the user can express, such as letters, symbols, and gestures, and corresponds to a complex gesture by both hands as well as one hand.

Therefore, the user inputs the virtual display screen 2 using both hands, thereby controlling the object output in front of the real user's eyes in a three-dimensional space similar to multi-touch. In particular, as shown in FIG. 2, the plurality of image measuring units 11a and 11b attached to both ends of the spectacles may perform input in a predetermined three-dimensional space in front of the user, and the corresponding space may include a plurality of images. It is determined by the angle of view from the image measuring units 11a and 11b.

In this case, the position indicating device 20 may be provided in plurality. Position indicator 20 is implemented in the form of a bracelet, can be worn on one wrist or both wrists of the user. In addition, the position indicator 20 may be implemented in the form of a ring may be worn on the user's finger. Thus, in the embodiment of the present invention, the position indicating device 20 is implemented in the form of a bracelet, the case is worn on both wrists of the user as an example, but is not limited thereto. For a detailed description of the position indicating device 20, refer to FIG.

On the other hand, the motion processing device 10 is implemented in a form that can be worn on the user's body like the position indicator 20, the embodiment may be worn on any part of the body, such as glasses, hats, clothes. In the embodiment of the present invention, the motion processing device 10 is implemented in the form of glasses in order to produce the same effect as seen from the user's perspective.

The motion processor 10 includes a plurality of image measuring units 11a and 11b. In this case, the plurality of image measuring units 11a and 11b are disposed at different positions, respectively, and measure signals generated from the position indicating device 20 at the corresponding positions. A detailed description of the motion processing device 10 will refer to FIG. 3.

3 is a block diagram showing the configuration of the motion processing device 10 according to the present invention.

As shown in FIG. 3, the motion processing device 10 according to the present invention includes a signal measuring means 11, a command input means 13, a signal processing means 15, and a communication means.

In addition, the signal measuring means 11 includes a first image measuring unit 11a, a second image measuring unit 11b, and a body shake measuring unit 11c.

The first image measuring unit 11a and the second image measuring unit 11b may be disposed at different positions, and may be disposed at both ends of the glasses as shown in FIG. 2. Of course, in FIG. 3, the first image measuring unit 11a and the second image measuring unit 11b are provided. However, the third image measuring unit may be further provided.

In this case, the first image measuring unit 11a and the second image measuring unit 11b may be image sensors capable of measuring signals generated from the position indicating device 20. Of course, it may have the form of infrared rays, visible light, laser, etc. generated by the position indicating device 20.

In addition, the first image measuring unit 11a and the second image measuring unit 11b receive a signal generated from the position indicating device 20, and detect a user's hand position from the received signal. In this case, the first image measuring unit 11a and the second image measuring unit 11b include a physical filter for distinguishing the image signal measured by the image sensor from the signal received from the position indicating device 20.

In this case, the physical filter may correspond to an infrared passband filter. The infrared passband filter removes the interference by the visible light, thereby allowing the first image measuring unit 11a and the second image measuring unit 11b to measure the infrared signal more clearly.

On the other hand, the body shake measuring unit 11c measures the degree of body shake of the user. When measuring the movement of the hand, the body shake measurement unit 11c may further include an inertial measurement unit (IMU) to compensate for an error caused by the shaking of the user's body. At this time, the body shake measuring unit 11c transmits the measured signal to the signal processing means.

The command input means 13 is a means for receiving a control command from a user, and includes a communication module for receiving a control command from a moving device.

When a control command is input from the user to the position indicating device 20, the position indicating device 20 transmits the corresponding command to the motion processing device 10. Therefore, the command input means 13 receives the control command from the position indicating device and transmits it to the signal processing means 15.

The signal processing means 15 includes a two-dimensional coordinate calculation section 15a, a three-dimensional coordinate calculation section 15b, a body shake correction section 15c, a pattern recognition section 15d, and a command processing section 15e.

First, the 2D coordinate calculation unit 15a is an area in which the position indicating device 20 is disposed from each image measured by the first image measuring unit 11a and the second image measuring unit 11b, that is, an area where a hand is located. Calculate the two-dimensional coordinates of the hand position at. At this time, the two-dimensional coordinate calculation unit 15a extracts the two-dimensional coordinates of the infrared image appearing in the form of points in each measurement image.

Thereafter, the 3D coordinate calculation unit 15b calculates the 3D coordinates at the corresponding position by using the 2D coordinates extracted by the 2D coordinate calculation unit 15a. See FIG. 6 for a model for calculation of three-dimensional coordinates.

In this case, the first image measuring unit 11a and the second image measuring unit 11b, the two-dimensional coordinate tracking unit, and the three-dimensional coordinate calculating unit 15b may be implemented in a complex manner and processed by one processor.

The body shake correction unit 15c grasps the degree of body shake from the signal measured by the body shake measuring unit 11c. At this time, the body shake correction unit 15c corrects the body shake of the image measured by the first image measuring unit 11a and the second image measuring unit 11b based on the detected information.

The pattern recognizing unit 15d recognizes the movement pattern in the three-dimensional coordinates calculated by the three-dimensional coordinate calculation unit 15b from the image corrected by the body shake correction unit 15c.

Thereafter, the command processor 15e extracts a command corresponding to the movement pattern recognized by the pattern recognition unit 15d and transmits the command to the wearable computer through the communication unit 17.

The command processor 15e may be connected to another device through a wired / wireless communication interface to transmit a command to the corresponding device.

At this time, the three-dimensional coordinate calculation unit 15b, the pattern recognition unit 15d, and the command processing unit 15e may add a communication interface to the two-dimensional coordinate calculation unit 15a and process it in other external devices.

4 is a block diagram showing the configuration of the position indicating device 20 according to the present invention.

Referring to FIG. 4, the position indicating device 20 according to the present invention includes a position signal generating means 21, an input signal measuring means 23, a signal processing means 25, and a communication means 27.

The position signal generating means 21 is means for generating a position signal for notifying the current position of the position indicating device 20. The position signal generating means 21 outputs in the form of a signal that can be measured by the first image measuring unit 11a and the second image measuring unit 11b of the motion processing device 10, such as infrared rays, visible light, and laser. In the embodiment of the present invention, the position signal generating means 21 is an example of generating an infrared signal.

The input signal measuring means 23 is a means for receiving a control command from the user. That is, the user instructs the validity of the hand movement currently performed by the user by an operation such as a click of a mouse which is a computer input device.

At this time, the input signal measuring means 23 recognizes a control command from the user by measuring a button operation in the shape of a ring, a finger or a wrist contact sound, an electromyography and the like.

For example, when a user taps (taps) in the air with a thumb and index finger, the input signal measuring means 23 interprets that the present motion is a valid motion and recognizes it as a signal indicating the start of the motion. . In addition, the input signal measuring means 23 may measure a command for instructing the start and the end of the movement from the action of tapping a finger and the spreading action when there is no user movement or when using EMG.

The input signal measuring means 23 transmits the measured user's control command to the signal processing means 25.

The signal processing means 25 includes an input signal recognition section 25a and a command processing section 25b.

The input signal recognizing section 25a recognizes a user's control command measured by the input signal measuring means 23. At this time, the command processing unit 25b transmits the user's control command recognized by the input signal recognition unit 25a to the command input unit 13 of the motion processing device 10 through the communication unit 27.

On the other hand, the command processor 25b outputs a predetermined control signal to the position signal generating means 21 when the command for notifying the start of user movement is recognized by the input signal recognition unit 25a. Therefore, the position signal generating means 21 generates the position signal in accordance with the control signal from the command processor 25b.

5 is an exemplary view referred to for explaining the operation of the image measuring unit of the motion processing device according to the present invention.

As shown in FIG. 5, the first image measuring unit 11a and the second image measuring unit 11b acquire images of the front surface. In this case, an area in which the virtual display screen 2 is implemented is determined by the angles of view of the first image measuring unit 11a and the second image measuring unit 11b.

In other words, the first image measuring unit 11a and the second image measuring unit 11b have a constant angle of view θ, and are virtual in an area where the angles of view of the first image measuring unit 11a and the second image measuring unit 11b overlap. The display screen 2 is implemented.

Therefore, in the virtual display screen 2 of the three-dimensional space implemented as described above, the user selects an object in the virtual space for the gesture of holding a finger or holding a finger and moves the object by moving a hand in the state. You can control your computer like you do.

6 is an exemplary view referred to for explaining the operation of the coordinate calculation method according to the present invention.

Referring to FIG. 6, when a position signal is generated from the position signal generating means 21 of the position indicating device 20, the first image measuring unit 11a and the second image measuring unit 11b receive the position signal.

At this time, the three-dimensional coordinate calculation unit 15b calculates the three-dimensional coordinates based on the position of the position signal generating means 21, the first image measuring unit 11a, and the second image measuring unit 11b.

A model for calculating three-dimensional coordinates can be obtained by using Equation 1 below.

Here, dr is a distance from the point where the position signal generated from the position signal generating means 21 to the first image measuring unit 11a reaches. dl is a distance from the point where the position signal generated from the position signal generating means 21 to the second image measuring unit 11b reaches.

L is the distance from the center of the first image measuring unit 11a to the center of the second image measuring unit 11b.

f is generated from the position signal generating means 21 in a direction perpendicular to the first image measuring unit 11a and the second image measuring unit 11b at the center of the first image measuring unit 11a and the second image measuring unit 11b. The distance to the point where the position signal meets, that is, the focal length of the image measuring unit.

x is a vertical distance from the first image measuring unit 11a and the second image measuring unit 11b to the position signal generating means 21.

Therefore, the 3D coordinate calculator 15b calculates the 3D coordinates using the 2D coordinates extracted from the 2D coordinate calculator 15a and the x value calculated above.

The operation of the present invention configured as described above is as follows.

7 is a flowchart illustrating an operation flow of a wearable computing environment-based user interface method according to the present invention, and illustrates an operation of the motion processing device 10.

As shown in FIG. 7, the motion processing device 10 measures a user's body motion based on a signal from the position indicating device 20 (100). In addition, the measured motion image is processed using the image sensors of the first image measuring unit 11a and the second image measuring unit 11b of the motion processing apparatus 10 (S110).

Thereafter, the two-dimensional coordinate calculation unit 15a calculates two-dimensional coordinates from the image of step S110 (S120), and the three-dimensional coordinate calculation unit 15b of the two-dimensional coordinates and 'S100' in the step 'S120'. The three-dimensional coordinates are calculated based on the signal received in the process (S130).

On the other hand, the body shake measurement unit 11c measures the degree of body shake from the signal received in step S100 (S140), and the body shake correction unit 15c measures the corresponding image according to the information measured in step S140. Body shaking and the resulting error is corrected (S150).

The pattern recognition unit 15d recognizes a user's movement pattern from the 3D coordinates calculated in step S130 and the image corrected in step S150 (S160), and corresponds to the pattern recognized in step S160. The command data is extracted and transmitted to the wearable computer through the communication module (S170).

The wearable computing environment-based user interface device and method thereof according to the present invention have been described as being applied to a wearable computer as an embodiment, but it is natural that the wearable computer may be used as an interface device of a general computer as well as a wearable computer.

As described above, the wearable computing environment-based user interface device and the method thereof are not limited to the configuration and method of the embodiments described as described above, but the embodiments may be modified in various ways. All or part of each of the embodiments may be configured to be selectively combined to make it possible.

1 and 2 are views referred to for describing the operation of the wearable computing environment-based user interface device according to the present invention.

3 is a block diagram illustrating a configuration of a motion processing device of a user interface device according to the present invention.

4 is a block diagram showing the configuration of the position indicating device of the user interface device according to the present invention.

5 is an exemplary view referred to to explain an image measuring method according to the present invention.

6 is an exemplary view referred to in describing a position coordinate measuring method according to the present invention.

7 is a flowchart illustrating an operation flow for a user interface method according to the present invention.

Claims (14)

A user interface device based on a wearable computing environment, A position indicating device worn near a user's wrist to generate an optical signal; Signal measuring means including a plurality of image measuring units including an image sensor for receiving an optical signal generated from the position indicating device and measuring an image in front of a user; And Analyzes each image measured by the signal measuring means to calculate three-dimensional coordinates, and recognizes a movement pattern of the user's hand on the three-dimensional coordinates from the optical signal received by the signal measuring means, and receives a corresponding command. Signal processing means for outputting; wearable computing environment based user interface device comprising a. The method according to claim 1, The signal processing means, A user based on the wearable computing environment, wherein a virtual display screen is implemented in an area where the angles of view of the image sensors included in the plurality of image measuring units overlap each other, and three-dimensional coordinates are calculated from the images of the virtual display screen. Interface device. The method according to claim 1, The signal processing means, And a two-dimensional coordinate calculator for extracting two-dimensional coordinates from each of the images measured by the plurality of image measuring units. The wearable computing environment-based user interface device, characterized in that for calculating the three-dimensional coordinates based on the extracted two-dimensional coordinates and the position information of the hand detected by the optical signal. The method according to claim 1, The image measuring unit, The wearable computing environment-based user interface device comprising a filter for distinguishing the image from the position signal generated from the position indicating device. The method according to claim 1, The signal measuring means, The wearable computing environment-based user interface device further comprises a; body shake measurement unit for measuring the degree of shaking of the body from the user's movement. The method according to claim 5, The body shake measurement unit, A wearable computing environment based user interface device comprising an inertial sensor. The method according to claim 5, The signal processing means, The wearable computing environment-based user interface device further comprising a; body shake correction unit for correcting the shake of the image measured from the image measuring unit based on the degree of shaking of the body measured by the body shake measuring unit. The method according to claim 1, The position indicating device, Position signal generating means for generating an optical signal each time a user's hand moves; And And an input signal measuring means for receiving a control command from a user. Wearable computing environment based user interface device, characterized in that for controlling the operation of the position signal generating means based on a user control command input to the input signal measuring means. As a user interface method based on wearable computing environment, Generating an optical signal indicating the position of the user's hand from the position indicating device worn near the user's wrist; Receiving an optical signal generated from the position indicating device and measuring a plurality of images of the front of the user; Calculating three-dimensional coordinates by analyzing each image measured in the measuring step; Identifying a position of the user's hand from the optical signal received in the measuring step, and recognizing a movement pattern of the user's hand on the calculated 3D coordinates; And And outputting a command corresponding to the recognized movement pattern in the step of recognizing the wearable computing environment. The method according to claim 9, The calculating step, Wearable computing environment characterized in that to implement a virtual display screen in the area where the angle of view of the plurality of image sensors for measuring the image in the measuring step and to calculate the three-dimensional coordinates from the image of the virtual display screen Based user interface method. The method according to claim 9, The calculating step, Extracting two-dimensional coordinates from each image measured by the plurality of image measuring means; The wearable computing environment-based user interface method, characterized in that for calculating the three-dimensional coordinates based on the extracted two-dimensional coordinates, and the position information of the hand detected by the optical signal of the measuring step. The method according to claim 9, The measuring step, Measuring the degree of shaking of the body from the movement of the user; wearable computing environment based user interface method comprising a. The method according to claim 12, And correcting the shaking of the image measured in the measuring step based on the measured shaking degree of the body. The method according to claim 9, The optical signal is, The user interface method based on the wearable computing environment, characterized in that when a signal indicating the start and end of the movement of the hand from the user to the position indicating device is input, generated based on the signal input from the user.

Images