KR101991401B1 - Method and apparatus for displaying augmented reality - Google Patents

Abstract

A method of displaying an augmented reality performed by an augmented reality display device includes the steps of acquiring depth information on a distance from a real image and augmented reality display device to each area of a real image, The method includes the steps of: obtaining a distance from the augmented reality display device to a virtual object; determining an out-focusing area of the real image based on the obtained distance and depth information; And displaying the augmented reality image including the augmented reality image.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for displaying augmented reality,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method and apparatus for displaying augmented reality, and more particularly, to a method and apparatus for displaying an augmented reality image including a real image and a virtual object.

As is well known, augmented reality technology is a technique of providing a special effect by expressing a virtual object together with a background image of a real world, causing an illusion that a virtual object feels as if it exists in the real world.

In such augmented reality technology field, efforts to improve the reality of the augmented reality image are continuing.

Conventionally, research has been conducted mainly on how to display a virtual object on a real image. For example, a light source in a real image is recognized, a virtual object having a shadow is generated based on the position of the recognized light source, and the generated virtual object is rendered in a real image to display an augmented reality image.

However, there is a limit to improve the reality of the augmented reality image only by considering how to display the virtual object in the real image.

Korean Patent Publication No. 10-2011-0107545, published on October 04, 2011.

When looking at the real world, the human eye does not focus on all objects but focuses only on the object in focus and its surroundings, and out-focusing on other objects. However, when the augmented reality display device displays a real image in the augmented reality image, unlike the human eye, the outfocusing process is not performed. Accordingly, when a person looks directly at the real world and sees the reality image of the AR, the user feels a sense of heterogeneity with respect to the AR image, which reduces the reality of the AR image.

According to an embodiment of the present invention, there is provided an augmented reality display device and an augmented reality display method for displaying an augmented reality image by out-focusing on a certain area in a real image.

The problems to be solved by the present invention are not limited to those mentioned above, and another problem to be solved can be clearly understood by those skilled in the art from the following description.

A method of displaying an augmented reality performed by an augmented reality display device according to an aspect of the present invention includes the steps of acquiring depth information about a distance between a real image and each area of the augmented reality display device, The method includes the steps of: arranging a virtual object on an image; acquiring a distance from the augmented reality display to the virtual object; determining an out-focusing area of the real image based on the obtained distance and the depth information And displaying the augmented reality image including the virtual object and the real image reflecting the area to be outfocused.

According to another aspect of the present invention, an augmented reality marking apparatus includes a display unit, an information acquiring unit that acquires depth information about a distance from a real image displayed through the display unit to each area of the real image, And a controller, wherein the controller is configured to arrange a virtual object on the real image, acquire a distance from the augmented reality display to the virtual object, and perform out-focusing on the real image based on the obtained distance, And controls the display unit to display an augmented reality image including the virtual image and the real image reflecting the area to be outfocused.

According to the embodiment of the present invention, the vicinity of the position where the virtual object is disposed in the real image is focused and the remaining area is out-focused to display the augmented reality image.

Accordingly, when a person looks directly at the real world, the user feels no sense of heterogeneity with respect to the augmented reality image when looking at the reality image of the augmented reality image together, thereby enhancing the reality of the augmented reality image.

1 is a block diagram of an augmented reality display apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method of displaying an augmented reality performed by the augmented reality display apparatus according to an embodiment of the present invention.
3 is a diagram illustrating an example of an augmented reality image displayed by the augmented reality display apparatus according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

1 is a block diagram of an augmented reality display apparatus according to an embodiment of the present invention.

1, the augmented reality display apparatus 100 includes an information obtaining unit 110, a control unit 120, a storage unit 130, and a display unit 140. As shown in FIG.

The information obtaining unit 110 obtains depth information on the distance from the augmented reality display device 100 to each region of the real image with respect to the real image to be used for the augmented reality image. For example, the information obtaining unit 110 can photograph a real image using a stereo camera and acquire depth information for each area together. Alternatively, the information acquiring unit 110 may acquire a real image and its depth information using a general camera and a depth sensor. Alternatively, the information obtaining unit 110 may include a camera separately provided from the augmented reality display device 100, and a communication interface capable of receiving information from the depth sensor. In this case, the information obtaining unit 110 can receive the real image and its depth information from the outside through the communication interface.

The storage unit 130 stores a computer-executable instruction for performing a process of controlling the augmented reality display process at the time of execution by the control unit 120. [ Then, the 3D model data corresponding to the virtual object to be displayed on the augmented reality image may be stored. In addition, the image data and the depth information of the real image obtained by the information obtaining unit 110 may be stored. For example, the storage unit 130 may store at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM, and a ROM Readable < / RTI > type of recording medium.

The controller 120 executes the computer executable instructions stored in the storage unit 130 to perform a process of controlling the augmented reality display process as a whole. According to this process, a virtual object is placed on the real image obtained by the information obtaining unit 110, a distance from the augmented reality display device 100 to the virtual object is obtained, and based on the obtained distance and depth information Determines an area to be out-focused in the real image, and controls the display unit 140 to display the virtual object and the augmented reality image including the real image reflecting the area to be out-focused. For example, the control unit 120 may include a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

The display unit 140 displays a virtual object and an augmented reality image including a real image reflecting an area to be out-focused according to a control signal resulting from a process performed by the controller 120. [ For example, the output unit 140 may include a flat panel display device such as a liquid crystal display (LCD) or the like.

2 is a flowchart illustrating a method of displaying an augmented reality performed by the augmented reality display apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the method of displaying an augmented reality according to an exemplary embodiment of the present invention includes acquiring depth information about a distance between a real image and each region of a real image from the augmented reality display device (S210).

In addition, the method of displaying an augmented reality according to an exemplary embodiment further includes a step S220 of arranging a virtual object in the obtained real image.

In addition, the augmented reality display method according to an embodiment further includes a step (S230) of acquiring a distance from the augmented reality display device to the virtual object. Here, the distance from the augmented reality display device to the virtual object may be extracted from the three-dimensional model data in which the depth information has been determined in advance, or may be calculated based on the depth information obtained in step S210.

In addition, the method of displaying an augmented reality according to an exemplary embodiment further includes a step S240 of determining an out-focusing area of a real image based on a distance from the obtained augmented reality display device to a virtual object. Here, an area in which the difference between the distance from the augmented reality display device to the virtual object and the distance from the augmented reality display device to each area of the real image is greater than or equal to a predetermined value may be determined as the area to be outfocused. That is, the virtual object and its surrounding area can be determined as the in-focus area, and the other area can be determined as the out-focusing area. For example, the in-focus region and the out-focusing region can be divided into a plurality of regions.

In addition, the method of displaying an augmented reality according to an exemplary embodiment of the present invention further includes the step of displaying an augmented reality image including a virtual object and a real image reflecting an area to be outfocused (S250). Here, blur processing may be performed on the pixels in the area to be out-focused of the real image to perform out-focusing processing. At this time, the blur processing may use a Gaussian blur kernel. In addition, within the area to be out-focused, the degree of out-focusing can be adjusted by changing the Gaussian blur kernel according to the difference between the distance from the augmented reality display device to the virtual object and the distance to each area of the real image. That is, as the distance from the virtual object in the area to be outfocused increases, the intensity of the blur processing is increased so that it becomes fainter.

Hereinafter, a method of displaying an augmented reality performed by the augmented reality display apparatus 100 according to an embodiment of the present invention will be described in more detail with reference to Figs.

The storage unit 130 of the augmented reality display apparatus 100 includes a computer 120 for performing a process of controlling the augmented reality display process at the time of execution by the control unit 120 of the augmented reality display apparatus 100. [ Executable instructions are stored. Then, the 3D model data corresponding to the virtual object to be displayed on the augmented reality image may be stored.

In this state, the controller 120 executes the computer executable instructions stored in the storage unit 130 to perform a process of controlling the augmented reality display process as a whole. When the corresponding process is performed by the control unit 120, the information obtaining unit 110 of the augmented reality display apparatus 100 obtains a difference between a real image to be used for the augmented reality image and each region of the augmented reality display apparatus 100 Obtain depth information about the distance. For example, the information acquisition unit 110 may acquire depth information by capturing a real image using a stereo camera. Alternatively, the information acquiring unit 110 may acquire a real image and its depth information using a general camera and a depth sensor. Alternatively, if the information obtaining unit 110 includes a camera separately provided from the augmented reality display device 100 and a communication interface capable of receiving information from the depth sensor, the information obtaining unit 110 may obtain information The real image and its depth information can be input from the outside (S210).

The control unit 120 reads the three-dimensional model data stored in the storage unit 130 and confirms the coordinate information of the read three-dimensional model data. Then, in accordance with the coordinate information confirmed in the real image obtained in step S210, The virtual object corresponding to the dimensional model data is arranged (S220).

Then, the controller 120 acquires the distance from the augmented reality display device 100 to the virtual object. Here, the controller 120 may obtain the distance from the augmented reality display device 100 to the virtual object based on the depth information of the 3D model data stored in the storage unit 130. [ If the position of the virtual object is changed by the user, the distance from the augmented reality display device 100 to the virtual object can be calculated based on the depth information of the real image obtained in step S210. For example, the distance from the rendering camera, which is virtually placed at the same position as the augmented reality display device 100, to the virtual object is obtained. Based on the depth information acquired in step S210 for the area in which the virtual object is disposed, The distance from the virtual object 100 to the virtual object can be calculated (S230).

Then, the controller 120 determines an out-focusing area of the real image based on the distance from the augmented reality display device 100 acquired in step S230 to the virtual object. Here, the controller 120 may be configured to out-focus an area having a difference between the distance from the augmented reality display device 100 to the virtual object and the distance from the augmented reality display device 100 to each area of the real image to a predetermined value or more You can decide. For example, when the distance from the augmented reality display apparatus 100 to the virtual object is 10 m, out of the real images, the distance from the augmented reality display apparatus 100 is more than 9.5 m or more than 10.5 m Area can be determined. That is, the area where the virtual object and its surrounding objects exist can be determined as the in-focus area, and the other area can be determined as the out-focusing area (S240).

Next, the control unit 120 controls the display unit 140 to display the virtual object and the augmented reality image including the real image reflecting the area to be out-focused. Here, the controller 120 may perform out-focusing processing by performing blur processing on the pixels in the out-focusing area determined in step S240 among the real images obtained in step S210.

3 is a diagram illustrating an example of an augmented reality image displayed by the augmented reality display apparatus 100 according to an embodiment of the present invention. (B) is an augmented reality image after out-focusing processing for a partial region of the real image. 3B, the controller 120 sets the focusing area 302 in which the periphery of the virtual object 301 is set as the focusing area 302 and the other area is the out-focusing area 303 as the augmented reality image Can be generated.

The controller 120 may use a Gaussian blur kernel when performing blur processing on pixels in an area to be out-focused. In addition, the Gaussian blur kernel is calculated according to the magnitude of the difference between the distance between the augmented reality display device 100 and the virtual object, and the distance to each area of the augmented reality display device 100, The degree of out-focusing can be adjusted. That is, as the distance from the virtual object in the area to be outfocused increases, the intensity of the blur processing is increased so that it becomes fainter.

Then, the display unit 140 displays the virtual object and the augmented reality image including the real image reflecting the area to be out-focused, according to the control signal resulting from the process performed by the control unit 120. [ For example, as shown in (a) of FIG. 3, the periphery of the virtual object 301 is processed in the focusing area 302, and the other areas are displayed in the out-focusing area 303 (S250).

Meanwhile, in a state where the augmented reality image is displayed by the display unit 140 in step S250, the user uses the interface provided by the augmented reality display device 100, such as a graphic user interface, In this case, steps S230 to S250 are performed again. That is, the area to be out-focused in the real image is also changed in response to the change of the position of the virtual object.

As described above, according to the embodiment of the present invention, the periphery of the position where the virtual object is disposed in the real image is focused, and the remaining area is out-focused to display the augmented reality image.

Therefore, when a person looks directly at the real world and sees the reality image of the augmented reality image together, he feels no sense of heterogeneity with the augmented reality image, thereby realizing the reality of the augmented reality image.

Combinations of each step of the flowchart and each block of the block diagrams appended to the present invention may be performed by computer program instructions. These computer program instructions may be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus so that the instructions, which may be executed by a processor of a computer or other programmable data processing apparatus, And means for performing the functions described in each step are created. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

According to the embodiment of the present invention, the vicinity of the position where the virtual object is disposed in the real image is focused and the remaining area is out-focused to display the augmented reality image.

The present invention can be widely used in various augmented reality industrial fields employing a device for augmented reality in which a virtual object is represented together with a background image in a real world to provide a special effect.

100: Augmented reality display
110: Information obtaining unit
120:
130:
140:

Claims (7)

An augmented reality display method performed by an augmented reality display device,
Obtaining depth information on a distance from the augmented reality display device to each region of the real image,
Disposing a virtual object on the real image;
Obtaining a distance from the augmented reality display device to the virtual object;
Determining an out-focusing area of the obtained real image based on the obtained distance and the depth information;
And displaying the augmented reality image including the virtual image and the real image blurred on the pixels in the area to be outfocused among the obtained real images
Augmented reality display method. The method according to claim 1,
Wherein the step of determining an area to be out-focused includes determining an area in which a difference between a distance to the virtual object and a distance to each area of the real image is equal to or greater than a predetermined value,
Augmented reality display method. delete The method according to claim 1,
The blur processing may be performed using a Gaussian blur kernel
Augmented reality display method. 5. The method of claim 4,
The degree of out-focusing is adjusted by changing the Gaussian blur kernel according to the difference between the distance to the virtual object and the distance to each region of the real image within the out-focusing area,
Augmented reality display method. In order to allow the processor to perform the augmented reality display method according to any one of claims 1, 2, 4, and 5
A computer program stored on a computer readable recording medium. 1. An augmented reality display device for displaying an augmented reality image,
A display section,
An information acquiring unit acquiring a photographed reality image with respect to a reality shown through the display unit and acquiring depth information about a distance from the augmented reality display unit to each area of the real image,
And a control unit,
Wherein the control unit disposes a virtual object on the real image, acquires a distance from the augmented reality display to the virtual object, and out-focuses the acquired real image on the basis of the obtained distance and the depth information And controls the display unit to display an augmented reality image including the virtual image and a real image blurred on the pixels in the area to be out-focused, from among the obtained real images
Augmented reality display.

Images