KR100934389B1 - Display device having virtual interaction interface and its operation method - Google Patents

Abstract

The present invention relates to a display apparatus having a virtual interaction interface and a method of operating the same. To this end, the present invention relates to a display apparatus for displaying a virtual screen in a space by using an image means and a display means with a mark image for position recognition. Use the virtual touch on the screen to interface with the object to operate naturally according to the user's input. Therefore, in the daily life of the user, general mobile devices can be directly controlled without using additional equipment or complicated procedures.

Virtual, interactive, interface, display, mark image

Description

DISPLAY APPARATUS THAT HAS A VIRTUAL INTERACTION INTERFACE AND A METHOD FOR IMPLEMENTING THE INTERFACE

The present invention relates to a display device having a virtual interaction interface and a method of operating the mobile device. More particularly, the present invention relates to a mobile device using a specific portion (eg, hand or foot) of a user so as to dynamically possess an environment of augmented reality. A display device capable of interacting with the device and a method of operating the same.

The present invention is derived from the research conducted as part of the IT new growth engine core technology development project of the Ministry of Information and Communication and the Ministry of Information and Communication Research and Development. [Task Management Number: 2006-S-032-02, Title: Personal Life Log-based Intelligent Service] Technology].

Adding virtual objects to our real life enriches our augmented reality, our lives.

The dual virtual reality system is an example of augmented reality, and by introducing the virtual reality system, the user can experience virtuality with high consistency with the real situation. A typical virtual reality system is composed of a virtual space simulated by a computer, an interface system that smoothly supports the user in the real world and the interaction between the two.

Currently, industrial production systems are adopting small quantity batch production systems in order to meet the shorter product life cycles and diverse user needs. In order to increase efficiency in the product lifecycle system, which consists of planning, design, design, production, marketing, and after-care of a product, a production company may use computer aided design (CAD). By using computer systems to digitize design work, integrate concurrent design and factory automation controls, the company reduces time from product planning to market.

As such, computer systems introduced in the current production process are increasing efficiency by performing computer simulations by digitizing existing data managed by documents and manual processes.

As described above, the conventional background technology to which augmented reality is applied has inherent limitations that can be used only when a specific environment is established, and in order to use augmented reality, a large number of mark images for location recognition are required. In order to apply to mobile devices, there are various limitations because it requires patches that need to be attached to a specific table or a user's body.

In addition, while the interaction between the user and the mobile device requires accuracy, convenience, and naturalness, the hand gesture recognition of the user has a lot of driving devices such as cumbersome gloves because the direct lines are directly connected to other devices. Equipment has helped to improve accuracy, but has the problem of being a big factor in preventing convenient or natural interaction.

Accordingly, the technical problem of the present invention is to solve the above-mentioned problems, and to display a virtual screen in space using an image means and a display means attached with a mark image for position recognition, and virtualize the screen using a hand. A display device and a method of operating the same have a virtual interaction interface that can be interfaced so as to be naturally manipulated according to a user input by touching.

According to an aspect of the present invention, there is provided a display apparatus having a virtual interaction interface including an image means having a mark image for position recognition attached thereto, a display processor for recognizing a mark image and outputting a virtual screen in a space, and a user. And a command interface for detecting a motion of touching a virtual screen through an image means, and a command recognizer for manipulating an object output on the virtual screen according to the detected motion.

According to another aspect of the present invention, there is provided a method of operating a display device having a virtual interaction interface, including: recognizing a mark image and outputting a virtual screen on a space; Detecting the generated movement, and interfacing to manipulate the object output to the virtual screen according to the detected movement.

According to the present invention, a virtual screen is displayed on a space by using an image means 105 and a display device to which a mark image for position recognition is attached and a virtual touch on the screen using a hand to operate an object naturally according to a user's input. By doing so, it is possible to control general mobile devices in a user's daily life without using additional equipment or complicated procedures.

In addition, the present invention has the effect that if the user configures the general mobile device in an environment familiar to the user, the user can control the environment of his computer as it is while doing everyday life.

Hereinafter, with reference to the accompanying drawings will be described in detail the operating principle of the present invention. In the following description of the present invention, when it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Terms to be described later are terms defined in consideration of functions in the present invention, and may be changed according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout the specification.

1 is a block diagram of a display apparatus having a virtual interaction interface according to an exemplary embodiment of the present invention, which includes a display means 101, a virtual interaction interface means 103, and an imaging means 105.

The display means 101 outputs the virtual screen 1011 on the space.

The virtual interaction interface means 103 recognizes the mark image S1 and interfaces to output the virtual screen 1011 in the space, and a user specific portion (for example, a hand or a foot) touches the virtual screen 1011. Is a block that detects a motion generated by the image means 105 and interfaces the object to be manipulated according to the detected hand movement. The display processor 1031, the image interface processor 1033, and the command recognizer 1035 are provided. It is provided.

The display processor 1031 is a block for outputting the virtual screen 1011 by recognizing the mark image S1 for position recognition. As shown in FIG. 2A, the display processor 1031 and the matrix calculator 1031b may be used. And an object visualization unit 1031c.

The mark recognition unit 1031a recognizes the position of the mark image S1 attached to the imaging means 105 and provides it to the matrix calculation unit 1031b.

The matrix calculation unit 1031b calculates (expresses) the position of the virtual screen 1011 in a matrix manner based on the position of the mark image S1 input from the mark recognition unit 1031a and provides it to the object visualization unit 1031c. do.

That is, in the matrix method calculation (expression) of the matrix calculation unit 1031b, for example, the real world position of the mark image S1 is called x, y, z, and the position of the virtual screen 1011 is x ', y'. z ', matrix transformation between two coordinate systems

(Where M _s is a scale transformation matrix, M _r is a rotate transformation matrix, and M _t is a translate transformation matrix.)

In other words, the scale transformation matrix is expressed as

The rotate transform matrix is represented by the Euler transform as follows.

축 회전

Shaft rotation

축 회전

Shaft rotation

축 회전

Shaft rotation

행렬

procession

Next, the translate transformation matrix is expressed as follows.

The object visualization unit 1031c interfaces to output a virtual screen to the calculation position input from the matrix calculation unit 1031b.

The image interface processor 1033 is a block for detecting the movement of the user's hand using the imaging means 105. As shown in FIG. 2B, the image interface processor 1033a, the disparity calculator 1033b, and the distance measurer ( 1033c).

The image input unit 1033a receives a user's hand movement captured by the imaging unit 105 as an image and provides the image to the disparity calculator 1033b.

The disparity calculation unit 1033b calculates a difference between the inconsistencies in the user's hand movement image input from the image input unit 1033a and provides the difference to the distance measuring unit 1033c.

That is, the calculation of the inconsistency of each disparity of the disparity calculator 1033b is as follows.

Referring to FIG. 5, when two imaging means 105 are used, O _l and O _r are left and right imaging means, f is focal length, P is a subject, and C _l and C _r. Is the center of the image introduced into each image means, and P _l and P _r are the positions of the subjects shown in each image.

Calculate the top of each mismatch through.

The distance measuring unit 1033c measures the distance between the image means 105 and the user's hand using the disagreement mismatched from the disparity calculator 1033b.

That is, the distance measurement of the distance measuring unit 1033c is as follows.

In order to restore the position of P, using the mapping values of P _l and P _r , Equation 3

(여기서, 과 은 도 5의 와 동일한 값이다.)

(Where, and are the same values as in FIG. 5)

Becomes

In addition, the two imaging means 105 is represented by Equation 4

Has a correlation such that the distance value Z is represented by Equation 5

In this way, the distance between the imaging means 105 and the subject P can be measured.

The command recognizer 1035 is a block for recognizing movements of a user's hand to recognize various commands. As shown in FIG. 2C, the selection recognizer 1035a, the execution recognizer 1035b, and the drag recognizer 1035c are illustrated. Has

The selection recognizing unit 1035a recognizes the selected area input by the user's hand for a short time and provides it to the execution recognizing unit 1035b.

The execution recognizer 1035b executes a specific object on the selected area input from the selection recognizer 1035a.

The drag recognizing unit 1035c recognizes the selected object input by the user's hand for a long time and instructs to move on the virtual screen.

The image means 105 is attached with a mark image S1 for position recognition, and provides an image captured by a user's hand movement to an image input unit 1033a in the image interface processor 1033.

Therefore, the present invention displays the virtual screen in the space using the image means 105 and the display device attached to the mark image for position recognition, and virtually touches the screen using the hand to manipulate the object naturally according to the user's input. By interfacing, users can control general mobile devices in the user's daily life without using additional equipment or complicated procedures.

4 is a flowchart illustrating in detail a method of operating a display apparatus having a virtual interaction interface according to an exemplary embodiment of the present invention.

First, it is assumed that the mark image S1 for position recognition is attached to the imaging means 105 as shown in FIG.

In the assumed state as described above, the mark recognition unit 1031a in the display processing unit 1031 recognizes the position of the mark image S1 attached to the imaging means 105 (S401) to the matrix calculation unit 1031b. to provide.

Then, the matrix calculation unit 1031b calculates (S403) the position of the virtual screen 1011 based on the position of the mark image S1 input from the mark recognition unit 1031a in the above-described matrix method, and performs the object visualization unit ( 1031c).

The object visualization unit 1031c interfaces to output (S405) the virtual screen into the display means 101 in space at the calculation position input from the matrix calculation unit 1031b.

At this time, in the state in which the virtual screen is output in the display means 101, the user performs various commands using his or her hands (S407), and the image means 105 touches the virtual screen 1011. An image obtained by capturing the motion (S409) is provided to the image input unit 1033a in the image interface processor 1033.

Then, the image input unit 1033a in the image interface processing unit 1033 receives a user's hand motion captured by the imaging unit 105 as an image (S411) and provides it to the disparity calculation unit 1033b.

The disparity calculation unit 1033b calculates the difference between the inconsistencies in the user's hand movement image input from the image input unit 1033a in the above-described manner (S413) and provides it to the distance measuring unit 1033c. The distance measuring unit 1033c measures the distance between the imaging means 105 and the user's hand by using the above-described disparity calculation unit 1033b and the above-described distance measuring method (S415).

The selection recognizing unit 1035a in the command recognizing unit 1035 recognizes the selected area input by the user's hand for a short time (S417) and provides the received recognition unit 1035b to the execution recognizing unit 1035b. A specific object is executed on the selected area input from 1035a (S419).

Then, the drag recognition unit 1035c recognizes the selected object input by the user's hand for a long time (S421) and instructs to move on the virtual screen.

Meanwhile, FIG. 3 is a configuration diagram in which the present invention is applied to a head mounted display (HMD) S2 which is a specific display device, and is performed in the same manner as the operation procedure of the display apparatus shown in FIG. 4. do.

Therefore, the present invention displays the virtual screen in the space using the image means 105 and the display device attached to the mark image for position recognition, and virtually touches the screen using the hand to manipulate the object naturally according to the user's input. By configuring the general mobile device into a user-friendly environment, the user can control the environment of his computer without losing his / her daily life.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the scope of the following claims, but also by those equivalent to the scope of the claims.

1 is a block diagram of a display device having a virtual interaction interface according to a preferred embodiment of the present invention;

FIG. 2A is a block diagram illustrating in detail the display processor illustrated in FIG. 1;

FIG. 2B is a block diagram illustrating in detail the image interface processor shown in FIG. 1; FIG.

FIG. 2C is a block diagram illustrating in detail the command recognition unit shown in FIG. 1; FIG.

3 is a configuration diagram applied to a specific display device HMD according to an embodiment of the present invention,

4 is a flowchart illustrating in detail a method of operating a display apparatus having a virtual interaction interface according to an embodiment of the present invention;

5 is an exemplary diagram for calculating a disparity difference of each disparity calculation unit shown in FIG. 2B according to the present invention.

<Description of the symbols for the main parts of the drawings>

101: display means 1011: virtual screen

103: virtual interaction interface means 1031: display processing unit

1031a: mark recognition unit 1031b: matrix calculation unit

1031c: object visualization unit 1033: image interface processing unit

1033a: Image input unit 1033b: Disparity calculator

1033c: distance measuring unit 1035: command recognition unit

1035a: selection recognition unit 1035b: execution recognition unit

1035c: drag recognition unit 105: video means

S1: mark image

S2: HMD

Claims (9)

An imaging means to which a mark image for position recognition is attached, A mark recognition unit that recognizes the position of the mark image, a matrix calculator that calculates the position of the virtual screen in a matrix manner based on the position of the recognized mark image, and an object visualization interface for outputting the virtual screen to the calculated position A display processing unit having a unit, An image interface processor configured to detect a movement of a user specific region touching the virtual screen through the image means; A command recognition unit for interfacing the object output on the virtual screen according to the detected movement Display device having a virtual interaction interface comprising a. delete The method of claim 1, The video interface processor, An image input unit which inputs a motion image of a user specific portion captured by the imaging means; A disparity calculator configured to calculate an upper difference of each mismatch in the motion image; Distance measuring unit for measuring the distance between the image means and the user specific portion using the calculated upper mismatch Display device having a virtual interaction interface comprising a. The method of claim 1, The command recognition unit, A selection recognizing unit recognizing an input of the user specific portion; An execution recognition unit executing an object according to the input of the recognized user specific part; Drag recognition unit for recognizing the movement of the executed object Display device having a virtual interaction interface comprising a. The method of claim 4, wherein And the user specific part is a hand or a foot of a human. A display method comprising an image means to which a mark image for position recognition is attached, the display method comprising: (a) recognizing the position of the mark image, calculating the position of the virtual screen based on the position of the recognized mark image in a matrix manner, and interfacing to output the virtual screen to the calculated position; (b) detecting a movement generated by the user specific region touching the virtual screen through the imaging means; (c) interface to manipulate an object output on the virtual screen according to the detected movement Display method having a virtual interaction interface comprising a. delete The method of claim 6, In step (b), (b1) inputting a moving image of a user specific portion captured by the imaging means; (b2) calculating an upper difference of each mismatch in the motion image; (b3) measuring the distance between the imaging means and the user specific part using the calculated upper mismatch; Display method having a virtual interaction interface comprising a. The method of claim 6, Step (c) is, (c1) recognizing an input of the user specific portion; (c2) executing the object according to the input of the recognized user specific part; (c3) recognizing to move the executed object Display method having a virtual interaction interface comprising a.

Images