KR101242764B1 - Apparatus and method for creating 3-dimensional augmented reality image using 3d image and gpu - Google Patents

Abstract

The present invention relates to a device and a method for implementing stereoscopic augmented reality using a stereoscopic 3D image and a PPP, and more specifically, to a left eye and a right eye image inputted through a binocular camera or a single view image input through a monocular camera. The present invention relates to a technology for generating a stereoscopic augmented reality image that allows users to feel true stereoscopic augmented reality by generating and overlaying a 3D graphic object of left and right eyes using a GPU while simultaneously generating a stereoscopic 3D image.

Description

Apparatus and method for generating stereoscopic augmented reality image using stereoscopic 3D image and WiFi {APPARATUS AND METHOD FOR CREATING 3-DIMENSIONAL AUGMENTED REALITY IMAGE USING 3D IMAGE AND GPU}

The present invention relates to an apparatus and method for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a PPP, and more specifically, to a left eye and a right eye or a monocular camera input through a binocular camera. The present invention relates to a technology for generating a stereoscopic augmented reality image by generating a 3D image with a right eye image and simultaneously generating a 3D graphic object around a predetermined pattern by recognizing a predetermined pattern and generating an overlay of the left eye and the right eye, respectively.

Augmented reality is a technology that superimposes a virtual object on the real world seen by the user. It is also called Mixed Reality (MR) because it combines the virtual world with additional information in real time into one image. It is a hybrid VR system that converges the real environment and the virtual environment. Since the late 1990s, R & D has been focused on the United States and Japan.

Augmented reality, a concept that complements the real world with the virtual world, uses a virtual environment made of computer graphics, but the main role is the real environment. Computer graphics serve to provide additional information needed for the real world. By overlapping the three-dimensional virtual image on the real image that the user is viewing, it means that the distinction between the real environment and the virtual screen becomes blurred.

Virtual reality technology immerses the user in the virtual environment and cannot see the real environment. However, augmented reality technology, which is a mixture of real environment and virtual objects, allows the user to see the real environment, providing better realism and additional information. For example, when the camera is illuminated with a smartphone camera, information such as the location and phone number of a nearby store is displayed as a stereoscopic image.

However, in the past, the user perceives a stereoscopic feeling by using a non-stereoscopic panel and implementing augmented reality in stereoscopic images of a single camera, that is, displaying a 3D object on a certain pattern card. It was 2D, so the feeling was antagonized.

In addition, in the related art, camera augmentation has always been necessary since augmented reality is supported only for a preview image of a camera.

The present invention proposes a technique for generating a stereoscopic augmented reality image by overlaying a 3D object generated using a 3D image and a GPU, not a 2D image.

In addition, the present invention proposes a technique for generating a stereoscopic 3D image from left and right eye images input through a binocular camera, and generating a stereoscopic augmented reality image by overlaying a 3D object generated using a GPU.

In addition, a stereoscopic 3D image is generated by converting a single viewpoint image input through a monocular camera into a left eye and a right eye image, and a technique of generating a stereoscopic augmented reality image by overlaying a 3D object generated using a GPU.

In addition, the present invention proposes a technique for generating a stereoscopic 3D image using a downloaded stereoscopic image or a single viewpoint image as well as a preview image of a binocular camera or a monocular camera and simultaneously generating a 3D graphic object to generate a stereoscopic augmented reality image.

In addition, a technology capable of generating a 3D object using a GPU through a single view image or the pattern information included in the left and right eye images, or generating a 3D object based on information received from a user, to generate a stereoscopic augmented reality image. To present.

According to an aspect, the apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a PGP receives a left eye and a right eye image, and generates a stereoscopic image generator and a left eye and right eye image for generating a stereoscopic 3D image from the received left and right eye images. And an object generator for generating 3D objects of left and right eyes to be overlayed at predetermined positions of the received left and right eye images, and the generated 3D objects of left and right eyes and the left and right eyes of a stereoscopic 3D image. It includes a stereoscopic augmented reality image generating unit for generating a stereoscopic augmented reality image by overlaying a predetermined position of the right eye image.

According to an additional aspect, the left and right eye images from the binocular camera module may further include an image input unit for transmitting to the stereoscopic image generating unit and the object generating unit.

According to another aspect, an image input unit which receives a single view image from a monocular camera module and transmits the image to the image conversion unit, receives a single view image from the image input unit, converts the image into left and right eyes, and transmits the image to the stereoscopic image generating unit and the object generating unit. The conversion unit may further include.

According to another aspect, the image input unit receives a single view image or a left eye and right eye image from the image storage unit and transmits the image to the image converting unit and the stereoscopic image generating unit and the object generating unit when the image received from the image input unit is left and right eye images. When the received image is a single view image, the transmission unit may further include an image converter for converting the image into the left and right eyes and transmitting the image to the stereoscopic image generator and the object generator.

According to an additional aspect, the object generating unit, a pattern recognition unit and a pattern recognition unit for recognizing at least one of the patterns included in the received left eye and right eye image to extract pattern information for generating 3D objects of the left eye and right eye It may include a graphic processing unit for generating a 3D object of the left eye and the right eye by rendering using the GPU (Graphic Processing Unit) based on the pattern information extracted from.

According to another aspect, the object generator generates a 3D object for left and right eyes by rendering using a graphic processing unit (GPU) based on an object information input unit for receiving 3D object generation information from a user and the received 3D object generation information. It may include a graphics processing unit.

According to one or more exemplary embodiments, a stereoscopic augmented reality image generation method using a stereoscopic 3D image and a PGP includes: receiving a left eye and a right eye image from a stereoscopic image generator, generating a stereoscopic 3D image from the left and right eye images, and an object generating unit from a left eye And receiving a right eye image, generating a 3D object of left and right eyes to be overlayed on a predetermined position of the received left and right eye images, and generating a 3D augmented reality image generator. And overlaying the object on predetermined positions of the left and right eyes of the stereoscopic 3D image to generate a stereoscopic augmented reality image.

According to an additional aspect, the generating of the stereoscopic 3D image may include transmitting the left and right eye images from the binocular camera module to the stereoscopic image generator and the object generator.

According to another aspect, the step of generating a stereoscopic 3D image, the image input unit receives a single view image from the monocular camera module and transmits to the image conversion unit and the image conversion unit receives the single view image and converts to the left eye and right eye image The method may include transmitting the stereoscopic image generator and the object generator.

According to another aspect, the step of generating a stereoscopic 3D image, the image input unit receives the image from the image storage unit and transmits the image to the image conversion unit and the stereoscopic image generation unit when the image received by the image conversion unit is left eye and right eye image And transmitting directly to the object generating unit and converting the received image into a left eye and right eye image to the stereoscopic image generating unit and the object generating unit when the received image is a single view image.

According to an additional aspect, the generating of the 3D object may include: extracting pattern information for generating 3D objects of each of the left and right eyes by recognizing at least one of patterns included in the left and right eye images, and extracting the extracted pattern information. Based on the rendering using the GPU (Graphic Processing Unit) may include the step of generating a 3D object of the left and right eyes.

According to another aspect, the 3D object generation step may include receiving 3D object generation information from a user and generating 3D objects of left and right eyes using a GPU (Graphic Processing Unit) based on the received 3D object generation information. It may include.

It is possible to provide an apparatus and method for generating a stereoscopic augmented reality image that allows a user to feel augmented reality in true stereoscopic by overlaying a 3D object generated using a 3D image and a PGP, not a 2D image.

In addition, it is possible to provide an apparatus and method for generating stereoscopic 3D images and a stereoscopic augmented reality image using a WiFi to contribute to the expansion of the stereoscopic 3D market and 3D graphics market through synergy of stereoscopic 3D generation and 3D graphics.

FIG. 1 is a conceptual diagram of stereoscopic augmented reality image generation using a stereoscopic 3D image and a WiFi.
2 is a block diagram of an apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to an exemplary embodiment.
3 is a block diagram of an apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to another embodiment.
4 is a block diagram of an apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to another embodiment.
5 is a flowchart illustrating a method of generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to an exemplary embodiment.
6 is a flowchart illustrating a method of generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to another embodiment.

Specific details of other embodiments are included in the detailed description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Hereinafter, an apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a GPU and a method thereof will be described in detail with reference to the accompanying drawings.

FIG. 1 is a conceptual diagram of stereoscopic augmented reality image generation using a stereoscopic 3D image and a WiFi. 1 illustrates an apparatus for generating a stereoscopic augmented reality image by mounting a binocular camera module according to an embodiment. An apparatus for generating stereoscopic augmented reality images using stereoscopic 3D images and GPUs does not necessarily mean that the binocular camera module is mounted, as described with reference to various embodiments below, and is equipped with a monocular camera module or a camera module. This may be a device that is not mounted at all.

Referring to FIG. 1, a concept of generating a stereoscopic augmented reality image is described. First, a left eye and a right eye image of a real object are photographed together with a predetermined pattern card through a binocular camera module. The certain pattern card includes pattern information for generating a 3D graphic object. A pattern card is a marker used in augmented reality. A pattern card (marker) refers to an object that can be easily recognized by computer vision technology, for example, a flat pattern written on a black background or a geometric with unusual colors. It can be said to be an object.

The 3D augmented reality image generating device receives 3D graphics of the left and right eyes including the pattern card photographed through the binocular camera module and renders the image using the GPU (Graphic Processing Unit) where the pattern card is located. Create an object. At this time, the 3D augmented reality image generating apparatus recognizes and analyzes the pattern information included in the pattern card and extracts information about the size of the 3D graphic object to be generated and the position to be overlayed. The apparatus for generating a stereoscopic augmented reality image generates a 3D graphic object of left and right eyes using the extracted information, and generates a 3D augmented reality image by overlaying the 3D graphic object on the corresponding left and right eyes.

2 is a block diagram of an apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to an exemplary embodiment. Hereinafter, the stereoscopic augmented reality image generating apparatus 100 using the stereoscopic 3D image and the WiFi according to an embodiment will be described in detail with reference to FIG. 2. 2 illustrates an embodiment of generating a stereoscopic augmented reality image by using the stereoscopic augmented reality image generating apparatus 100 equipped with a binocular camera module.

As shown in FIG. 2, the apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a PPU 100 may include a stereoscopic image generating unit 140, an object generating unit 150, and a stereoscopic augmented reality image generating unit 160. Include.

The stereoscopic image generator 140 receives left and right eye images, and generates a stereoscopic 3D image from the received left and right eye images. In the binocular stereoscopic image, although the two left and right camera modules of the camera system are photographing the same object, the object is photographed with a slight horizontal difference like the human eye. It is called time difference. When the 3D image is captured by the human eye and the 3D image is reproduced in the brain, if the image does not match according to the left and right camera intervals and the size of the parallax of the left and right images toward the object, the eye becomes tired and headaches occur. Cause problems such as causing.

The parallax between two images taken with a binocular camera has a characteristic that the parallax increases from a distance to a near distance. The left and right images are similar to each other, but because the objects in the distance and the objects in the short distance are mixed, a difference occurs in the matching point depending on the state of the background and the subjects. Accordingly, the stereoscopic 3D image is generated by adjusting the convergence by matching the gaze points by moving both images left and right or up and down about a target object or gaze area (a point where depth = 0) as a reference.

According to an additional aspect of the present embodiment, the stereoscopic augmented reality image generating apparatus 100 may further include a binocular camera module and an image input unit 120.

The binocular camera module captures live images of the left and right eyes of the same object at regular intervals, such as a human eye. The image input unit 120 receives the left and right eye images photographed from the binocular camera module and transmits the images to the stereoscopic image generator 140.

Meanwhile, the 3D augmented reality image generating apparatus 100 may further include an image storage unit 110. The image storage unit 110 may be a stereoscopic image downloaded from an external device (PC, other binocular camera, etc.) through a wired or wireless connection (USB, Bluetooth, infrared communication, etc.) or a server providing stereoscopic image content through wireless network communication. You can save the downloaded 3D video. The image input unit 120 may receive stereoscopic images of left and right eyes stored in the image storage unit 110 by a user's UI manipulation.

That is, the image input unit 120 may receive the left and right eye images captured by the binocular camera module or the left and right eye images stored in the image storage unit 110 selectively by a user's UI manipulation. Accordingly, the apparatus 100 for generating a stereoscopic augmented reality image may generate a stereoscopic augmented reality image using not only the preview image captured by the binocular camera module but also the downloaded stereoscopic image stored in the image storage unit 110. .

The object generator 150 receives left and right eye images, and generates 3D object images of left and right eyes to be overlaid on predetermined positions of the received left and right eye images. The object generator 150 may receive at least one of a left eye and a right eye image from the image inputter 120. In this case, the 3D object is generated for each of the left eye and right eye images. The 3D object is a virtual object for generating an augmented reality image by overlaying the photo-realistic image and is generated through a rendering process using a graphic processing unit (GPU).

The object generator 150 generates a 3D graphic object by rendering using the GPU using information for generating the 3D object. In this case, the information for generating a 3D object includes modeling for generating a 3D graphic object such as geometry, viewpoint, texture mapping, lighting, and shading information, and location and size of the 3D object to be overlaid. Information for generating an augmented reality image, such as scaling and torsion information.

Realtime Rendering refers to the process of creating an image called raster graphics image in real time from 3D model information using a computer program. Real-time rendering is used in a variety of fields, including architecture, video games, simulators, special effects, and design visualization. Real-time rendering requires a lot of computation, so we use a graphics accelerator (GPU) to speed up rendering.

According to a more specific aspect, the object generator 150 may include an object information input unit 152 and a graphic processor 153. The user may directly input the 3D object generation information by selectively operating the UI of the 3D augmented reality image generating apparatus 100. The object information input unit 152 receives information necessary for generating a 3D object input by a user through UI manipulation of the 3D augmented reality image generating apparatus 100.

That is, the object information input unit 152 may directly receive 3D object generation information corresponding to the parallax of the stereoscopic image, which is a live image, in order to have the same stereoscopic sense as the stereoscopic image generated by the stereoscopic image generator 140. .

The graphic processor 153 renders 3D object images of the left and right eyes by rendering them to match the parallax of the received left and right eyes using a GPU (Graphic Processing Unit) based on the received 3D object generation information.

According to an additional aspect of the present embodiment, a predetermined pattern may be included in predetermined positions of the left and right eye images input to the image input unit 120, and the object generator 150 may further include a pattern recognition unit 151. have. The predetermined pattern includes various kinds of images such as a chessboard pattern, and the type of the 3D object synthesized with the real world image may be determined according to the type of the pattern. The predetermined pattern included in the left and right eye images may be taken together when the real image is taken from the binocular camera module, and the real pattern image including the pattern is downloaded and stored in the image storage unit 110 through another binocular camera. It may be.

The pattern recognition unit 151 recognizes at least one of the patterns included in the left and right eye images received from the image input unit 120 and extracts pattern information for generating 3D object images of the left and right eyes. That is, the pattern recognition unit 151 searches for one of the left eye and right eye images received from the image input unit 120 to recognize the boundary of the pattern. When the boundary of the pattern is identified in the live image, the 3D position information, which is the position of the pattern, is calculated, and the pattern information is extracted by estimating the size of the pattern and the degree of twist. When the pattern information is extracted, the pattern recognition unit 151 transmits the pattern information and the 3D graphic modeling information to the graphic processing unit 153. In this case, the 3D graphic modeling information corresponding to various patterns may be stored in a separate memory unit (not shown), and when the pattern information extraction is completed, the pattern recognition unit 151 may use the pattern information to determine the corresponding pattern. The 3D modeling information may be read from the memory unit (not shown) and transmitted to the graphic processor 153.

In this case, the object information input unit 152 may further receive parallax information for representing a 3D object of the 3D object in the augmented reality image by the user's UI manipulation and transmit the received parallax information to the graphic processing unit 153. The disparity information input from the user may be disparity information corresponding to the disparity of the stereoscopic image which is the real image in order to have the same stereoscopic sense as the stereoscopic image generated by the stereoscopic image generating unit 140.

The graphic processor 153 generates a 3D object by rendering using a graphic accelerator (GPU) based on the pattern information, 3D object generation modeling information, or parallax information. In general, a rendering process for generating a 3D graphic object using a 3D graphics accelerator may be roughly divided into a geometry processing process and a rasterization process.

First, the geometry processing process mainly refers to a process of transforming an object of a 3D coordinate system according to a viewpoint, performing lighting processing and shading, and projecting the object to a two-dimensional coordinate system. The geometrical process involves a significant amount of matrix and trigonometric operations, resulting in significant computational load. In the conventional 3D graphics processing method, the CPU performs this geometric processing, but recently, by performing the geometric processing in the 3D graphics accelerator, the computational load of the CPU is greatly reduced, thereby improving the performance of the entire system.

Raster processing is the process of determining the color value in the image of the 2D coordinate system and storing it in the frame buffer. The raster process performs rasterization on the 3D graphic model converted to vector graphics by geometry transformation. Rasterization is the process of converting vector graphics into pixel pattern images. That is, the raster processing process stores a generated image in a frame buffer by attaching a polygon of a 3D graphic model, which has never existed, to correspond to pixels on a screen.

The stereoscopic augmented reality image generator 160 overlays the left and right eye 3D objects generated by the object generator 150 at predetermined positions of the left and right eye images of the stereoscopic 3D image generated by the stereoscopic image generator 140. (Overlay) to generate a stereoscopic augmented reality image.

According to an additional aspect, the stereoscopic augmented reality image generating apparatus 100 may further include an image output unit 170. The image output unit 170 is a stereoscopic augmented reality generator (3) to fit the output format of the display (line base, pixel_base, sub-pixel base, side-by-side, top-bottom, red-blue, red-green, etc.) The stereoscopic augmented reality image generated at 160 is formatted and output to the display.

3 is a block diagram of an apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to another embodiment. Hereinafter, an apparatus 300 for generating a stereoscopic augmented reality image using a single viewpoint image will be described with reference to FIG. 3. 3 illustrates an embodiment in which a stereoscopic augmented reality image is generated using a stereoscopic augmented reality image generating apparatus 300 equipped with a monocular camera module.

As shown in FIG. 3, the stereoscopic augmented reality image generating apparatus 300 may include an image converter 330, a stereoscopic image generator 340, an object generator 350, and a stereoscopic augmented reality image generator 360. Include.

The image converter 330 receives a single view image, converts the received single view image into left and right eye images, and transmits the single view image to the stereoscopic image generator 340 and the object generator 350. On the other hand, the image conversion unit 330 may receive a stereoscopic image of the left eye and the right eye, and when the stereoscopic image is received, the stereoscopic image generation unit 340 and the object generation unit 350 do not perform an image conversion process. send. Here, various known methods may be used for the method itself for converting a single view image into a multiview image of left and right eyes.

According to an additional aspect of the present embodiment, the stereoscopic augmented reality image generating apparatus 300 may further include a monocular camera module and an image input unit 320.

The monocular camera module captures live-action images. The image input unit 320 receives a live image captured by the monocular camera module and transmits it to the image converter 330.

On the other hand, the stereoscopic augmented reality image generating apparatus 300 may further include an image storage unit 310. The image storage unit 310 provides an image downloaded from an external device (PC, other monocular camera, binocular camera, etc.) via wired or wireless connection (USB, Bluetooth, infrared communication, etc.) or through wireless network communication. You can connect to the server and save the downloaded video. The image input unit 320 may receive a single view image or a left eye and a right eye image stored in the image storage unit 310 by a user's UI manipulation and transmit the image to the image converter 330.

That is, the image input unit 320 may receive a single viewpoint image captured by the monocular camera module or a single viewpoint image stored in the image storage unit 310 or a stereoscopic image of left and right eyes selectively by a user's UI manipulation. have. Therefore, the stereoscopic augmented reality image generating apparatus 300 not only preview images photographed by the monocular camera module but also stereoscopic augmented reality images using the downloaded single viewpoint / multiview image stored in the image storage unit 310. Can be generated.

The stereoscopic image generator 340 receives the left and right eye images transmitted from the image converter 330, and generates a stereoscopic 3D image from the received left and right eye images.

The object generator 350 receives left and right eye images transmitted from the image converter 330 and generates 3D object images of left and right eyes to be overlaid on predetermined positions of the received left and right eye images. . In this case, the 3D object is generated for each of the left eye and right eye images. The 3D object is a virtual object for generating an augmented reality image by overlaying the photo-realistic image and is generated through a rendering process using a graphic processing unit (GPU).

The object generator 350 generates a 3D graphic object by rendering using the GPU using information for generating the 3D object. In this case, the information for generating a 3D object includes modeling for generating a 3D graphic object such as geometry, viewpoint, texture mapping, lighting, and shading information, and location and size of the 3D object to be overlaid. Information for generating an augmented reality image, such as scaling and torsion information. In this case, the information for generating the 3D object may be selected by a user's UI manipulation.

According to a more specific aspect, the object generator 350 may include an object information input unit 352 and a graphic processor 353. The user may directly input the 3D object generation information by selectively manipulating the UI of the 3D augmented reality image generating apparatus 300. The object information input unit 352 receives information necessary for generating a 3D object input by a user through UI manipulation of the 3D augmented reality image generating apparatus 300.

That is, the object information input unit 352 may directly receive 3D object generation information that matches the parallax of the stereoscopic image, which is a live image, in order to have the same stereoscopic sense as the stereoscopic image generated by the stereoscopic image generator 340. .

The graphic processor 353 generates 3D object images of the left and right eyes by rendering the 3D object generation information to match the parallax of the received left and right eyes using the GPU (Graphic Processing Unit).

According to an additional aspect of the present embodiment, a predetermined pattern may be included in a predetermined position of the single view image input to the image input unit 320, and the object generator 350 may further include a pattern recognition unit 351. . The predetermined pattern includes various kinds of images such as a chessboard pattern, and the type of the 3D graphic image synthesized with the real world image may be determined according to the type of the pattern. The predetermined pattern included in the single view image may be photographed together when the real image is taken from the monocular camera module, and the actual pattern image including the pattern is downloaded and stored in the image storage unit 110 through another monocular camera. Can be.

The pattern recognition unit 351 extracts pattern information for generating 3D object images of left and right eyes by recognizing at least one of patterns included in the left and right eye images received from the image converter 330. That is, the pattern recognition unit 351 searches for one of the left eye and right eye images received from the image converter 330 to recognize the boundary of the pattern. When the boundary of the pattern is identified in the live image, the three-dimensional position information of the pattern is calculated, and the pattern information is extracted by estimating the size of the pattern and the degree of twist. When the pattern information is extracted, the pattern recognition unit 351 transmits the pattern information and the 3D graphic modeling information to the graphic processing unit 353. In this case, the 3D graphic modeling information corresponding to various patterns may be stored in a separate memory unit (not shown), and when the pattern information extraction is completed, the pattern recognition unit 351 may use the pattern information to determine the corresponding pattern. The 3D modeling information may be read from the memory unit (not shown) and transmitted to the graphic processor 353.

In this case, the object information input unit 352 may further receive parallax information for representing a 3D object of the 3D object in the augmented reality image by the user's UI manipulation and transmit the received parallax information to the graphic processor 353. The parallax information input from the user may be parallax information corresponding to the parallax of a stereoscopic image which is a real image in order to have the same stereoscopic sense as the stereoscopic image generated by the stereoscopic image generating unit 340.

The graphic processor 353 generates a 3D object by rendering using a graphic accelerator (GPU) based on pattern information, 3D object generation modeling information, or parallax information.

The stereoscopic augmented reality image generator 360 overlays the left and right eye 3D objects generated by the object generator 350 at predetermined positions of the left and right eye images of the stereoscopic 3D image generated by the stereoscopic image generator 340. (Overlay) to generate a stereoscopic augmented reality image.

In addition, according to an additional aspect, the 3D augmented reality image generating apparatus 300 may further include an image output unit 370. The image output unit 370 is a stereoscopic augmented reality generator (370) to match the output format of the display (line base, pixel_base, sub-pixel base, side-by-side, top-bottom, red-blue, red-green, etc.) The stereoscopic augmented reality image generated at 360 is formatted and output to the display.

4 is a block diagram of an apparatus for generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to another embodiment. Hereinafter, an apparatus 500 for generating a stereoscopic augmented reality image will be described with reference to FIG. 4. 4 illustrates an embodiment in which a stereoscopic augmented reality image is generated by using a stereoscopic augmented reality image generating apparatus 500 in which a camera module is not mounted.

As shown in FIG. 4, the stereoscopic augmented reality image generating apparatus 300 includes an image storage unit 510, an image input unit 520, an image converter 530, a stereoscopic image generator 540, and an object generator ( 550 and a stereoscopic augmented reality image generator 560.

The image storage unit 510 is connected to an external device (PC, etc.) by wired or wirelessly to download an image, or connects to a server providing image content through wireless network communication and stores the downloaded image. The image stored in the image storage unit 510 may include a 2D image of a single viewpoint or a stereoscopic image of left and right eye images.

The image input unit 520 receives a single view image or left and right eye images stored in the image storage unit 510 by a user's UI operation and transmits the image to the image converter 530.

The image converter 530 receives a single view image or a left eye and a right eye image. In this case, when the received image is a left eye and a right eye image, the image is directly transmitted to the stereoscopic image generator 540 and the object generator 550 without any conversion process. If the received image is a single view image, the single view image is converted into a left eye and a right eye image and transmitted to the stereoscopic image generator 540 and the object generator 550. Here, various known methods may be used for the method itself for converting a single view image into a multiview image of left and right eyes.

The stereoscopic image generator 540 receives the left and right eye images transmitted from the image converter 530, and generates a stereoscopic 3D image from the received left and right eye images.

The object generator 550 receives the left and right eye images transmitted from the image converter 530 and generates 3D objects of the left and right eyes to be overlaid on predetermined positions of the received left and right eye images. In this case, the 3D object is generated for each of the left eye and right eye images. The 3D object is a virtual object for generating an augmented reality image by overlaying the photo-realistic image and is generated through a rendering process using a graphic processing unit (GPU).

The object generator 350 generates a 3D graphic object by rendering using the GPU using information for generating the 3D object. In this case, the information for generating a 3D object includes modeling for generating a 3D graphic object such as geometry, viewpoint, texture mapping, lighting, and shading information, and location and size of the 3D object to be overlaid. Information for generating an augmented reality image, such as scaling and torsion information. In this case, the information for generating the 3D object may be selected by a user's UI manipulation.

According to a more specific aspect, the object generator 550 may include an object information input unit 552 and a graphic processor 553. The user may directly input the 3D object generation information by selectively manipulating the UI of the 3D augmented reality image generating apparatus 500. The object information input unit 552 receives information necessary for generating a 3D object input by a user through UI manipulation of the stereoscopic augmented reality image generating apparatus 500.

That is, the object information input unit 552 may directly receive 3D object generation information that matches the parallax of the stereoscopic image, which is a live image, in order to have the same stereoscopic effect as the stereoscopic image generated by the stereoscopic image generator 540. .

The graphic processor 553 generates 3D object images of the left and right eyes by rendering the 3D object generation information to match the parallax of the received left and right eyes using the GPU (Graphic Processing Unit).

According to an additional aspect of the present embodiment, a predetermined pattern may be included in a single view image or a predetermined position of the left eye and right eye images input to the image input unit 520. At this time, the pattern included in the image is, for example, included in a live image through another monocular camera or binocular camera and photographed together, may be downloaded from the camera and stored in the image storage unit 510. The predetermined pattern includes various kinds of images such as a chessboard pattern, and the type of the 3D graphic object synthesized with the real world image may be determined according to the type of the pattern.

The object generation unit 550 may further include a pattern recognition unit 551. The pattern recognition unit 551 recognizes at least one of the patterns included in the left and right eye images received from the image converter 530 and extracts pattern information for generating 3D object images of the left and right eyes. That is, the pattern recognition unit 551 detects any one of the left eye and right eye images received from the image converter 530 to recognize the boundary of the pattern. When the boundary of the pattern is identified in the live image, the three-dimensional position information of the pattern is calculated, and the pattern information is extracted by estimating the size of the pattern and the degree of twist. When the pattern information is extracted, the pattern recognition unit 551 transmits the pattern information and the 3D graphic modeling information to the graphic processing unit 553. In this case, the 3D graphic modeling information corresponding to the various patterns may be stored in a separate memory unit (not shown), and the pattern recognition unit 551 may use the pattern information to extract the pattern information when the pattern information extraction is completed. The 3D modeling information may be read from the memory unit (not shown) and transmitted to the graphic processor 553.

In this case, the object information input unit 552 may further receive parallax information for representing a 3D object of the 3D object in the augmented reality image by the user's UI manipulation, and transmit the received parallax information to the graphic processor 553. The disparity information input from the user may be disparity information corresponding to the disparity of the stereoscopic image which is the real image in order to have the same stereoscopic sense as the stereoscopic image generated by the stereoscopic image generating unit 540.

The graphic processor 553 generates a 3D object by rendering using a graphic accelerator (GPU) based on pattern information, 3D object generation modeling information, parallax information, and the like.

The stereoscopic augmented reality image generator 560 overlays the left and right eye 3D objects generated by the object generator 550 at predetermined positions of the left and right eye images of the stereoscopic 3D image generated by the stereoscopic image generator 540. (Overlay) to generate a stereoscopic augmented reality image.

According to an additional aspect, the stereoscopic augmented reality image generating apparatus 500 may further include an image output unit 570. The image output unit 570 may include a stereoscopic augmented reality generator to match an output format of a display (line base, pixel_base, sub-pixel base, side-by-side, top-bottom, red-blue, red-green, etc.). The stereoscopic augmented reality image generated at 560 is formatted and output to the display.

5 is a flowchart illustrating a method of generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to an exemplary embodiment. A method of generating a stereoscopic augmented reality image using left and right eye images will be described with reference to FIG. 5.

FIG. 5 illustrates a method of generating a stereoscopic augmented reality image using the apparatus for generating a stereoscopic augmented reality image 100 according to the embodiment of FIG. 2. Since the apparatus 100 for generating a stereoscopic augmented reality image according to the exemplary embodiment of FIG. 2 has been described in detail above, a detailed description thereof will be omitted.

In the stereoscopic augmented reality image generation method using a stereoscopic 3D image and a PPP according to an embodiment, first, the image input unit 120 receives a left eye and a right eye image from a binocular camera module or an image storage unit 110, a stereoscopic image generation unit 140 and the object generating unit 150 (step S110).

Next, the stereoscopic image generating unit 140 receives the left and right eye images from the image input unit 120 and generates a stereoscopic 3D image from the received left and right eye images (step S120).

Next, the object generator 150 receives the left and right eye images from the image input unit 120 and generates 3D objects of the left and right eyes to be overlaid on the received left and right eye images (step S130).

More specifically, generating the 3D graphic object (step S130) includes extracting information for generating the 3D object (step S131), and generating the 3D object based on the extracted information (step S132). It may include.

In the extracting information for generating the 3D object (step S131), the object information input unit 152 may directly receive necessary information from the user. The pattern information can be extracted by recognizing. Hereinafter, a procedure of receiving information for generating a 3D object from a user or extracting pattern information and transmitting the extracted pattern information to the graphic processor 153 has been described above in detail with reference to the embodiment of FIG. 2, and thus a detailed description thereof will be omitted.

Generating the 3D object (step S132) is performed by the graphic processor 153 based on the information received from the pattern recognition unit 151 and the additional information received from the 3D object generation information input unit 152 using the left eye and the GPU. Create a 3D object in the right eye.

Next, the stereoscopic augmented reality image generating unit 160 generates a stereoscopic augmented reality image by overlaying the 3D objects of the left and right eyes on a predetermined position of the left and right eye images of the generated stereoscopic 3D image.

Finally, the stereoscopic augmented reality image output unit 170 formats and outputs the generated stereoscopic augmented reality image to fit the panel of the display device (not shown).

6 is a flowchart illustrating a method of generating a stereoscopic augmented reality image using a stereoscopic 3D image and a WiFi according to another embodiment. A method of generating a stereoscopic augmented reality image using a single viewpoint image will be described with reference to FIG. 6. FIG. 6 illustrates a method of generating a stereoscopic augmented reality image using the apparatus for generating a stereoscopic augmented reality image 300 according to the embodiment of FIG. 3. Since the apparatus 300 for generating a stereoscopic augmented reality image according to the embodiment of FIG. 3 has been described in detail above, a detailed description thereof will be omitted.

In the stereoscopic augmented reality image generating method using a stereoscopic 3D image and a WiFi according to another embodiment, first, the image input unit 320 receives a single view image from the monocular camera module or the image storage unit 310 (step S310).

Next, the image converter 330 converts the input single-view image into left and right eye images (step S320).

Next, the stereoscopic image generator 340 generates a stereoscopic 3D image from the converted left eye and right eye images (step S330).

Next, a 3D graphic object is generated from the converted left eye and right eye images (step S340). More specifically, generating the 3D graphic object (step S340) includes generating the 3D object generation information (step S341) and generating the 3D object based on the generated 3D object generation information (step S342). It may include.

In the extracting information for generating the 3D object (step S331), the object information input unit 352 may directly receive necessary information from the user. On the other hand, if a predetermined pattern is included in the input image, the pattern information may be extracted. The pattern information can be extracted by recognizing. Hereinafter, a procedure of receiving information for generating a 3D object from a user or extracting pattern information and transmitting the extracted pattern information to the graphic processor 353 has been described above in detail with reference to the embodiment of FIG. 2, and thus a detailed description thereof will be omitted.

Generating the 3D object (step S332), the graphic processor 353 uses the GPU based on the information received from the pattern recognition unit 351 and the additional information received from the 3D object generation information input unit 352 to determine the left eye and the eye. Create a 3D object in the right eye.

Next, the 3D graphic object is overlaid on predetermined positions of the left and right eye images of the generated stereoscopic 3D image to generate a stereoscopic augmented reality image (step S350).

Finally, the generated stereoscopic augmented reality image is formatted and output to fit the panel of the display device (not shown).

Hereinafter, a method of generating a stereoscopic augmented reality image using the stereoscopic augmented reality image generating apparatus 500 according to the embodiment of FIG. 4 will be described. Since the apparatus 500 for generating a stereoscopic augmented reality image according to the exemplary embodiment of FIG. 4 has been described in detail above, a detailed description thereof will be omitted.

In the stereoscopic augmented reality image generation method, first, the image input unit 520 receives a single view image or a stereoscopic image of left and right eyes from the image storage unit 510 and transmits it to the image conversion unit 530.

Next, the image converter 530 receives an image from the image input unit 520, and if the received image is a stereoscopic image of left and right eyes, the image converter 530 immediately transmits the image to the stereoscopic image generator 540 and the object generator 550. If the received image is a single view image, the single view image is converted into a left eye and a right eye image and then transmitted to the stereoscopic image generator 540 and the object generator 550.

Next, the stereoscopic image generator 540 receives the left and right eye images to generate a stereoscopic 3D image.

Next, the object generator 550 receives left and right eye images from the image converter 530, and generates a 3D graphic object from the received left and right eye images. More specifically, generating the 3D graphic object may include generating 3D object generation information and generating a 3D object based on the generated 3D object generation information.

In the extracting information for generating the 3D object, the object information input unit may directly receive necessary information from the user. On the other hand, when a predetermined pattern is included in the input image, the object information input unit may recognize the pattern and extract pattern information. Can be. Hereinafter, a procedure of receiving information for generating a 3D object from a user or extracting pattern information and transmitting the extracted pattern information to the graphic processor 553 has been described above in the embodiment of FIG. 2, and thus a detailed description thereof will be omitted.

Generating the 3D object may be performed by the graphic processing unit 553 based on the information received from the pattern recognition unit 551 and the additional information received from the 3D object generation information input unit 552. Create

Next, a 3D augmented reality image is generated by overlaying the 3D graphic object generated at predetermined positions of the left and right eye images of the generated stereoscopic 3D image.

Finally, the generated stereoscopic augmented reality image is formatted and output to fit the panel of the display device.

It will be understood by those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of the present invention is defined by the appended claims rather than the foregoing detailed description, and all changes or modifications derived from the meaning and scope of the claims and the equivalents thereof are included in the scope of the present invention Should be interpreted.

100 300 500: Stereoscopic augmented reality image generating apparatus 110 310 510: Image storage unit
120 320 520: Image input unit 330 530: Image converter
140 340 540: stereoscopic image generating unit 150 350 550: object generating unit
151 351 551: Pattern recognition unit 152 352 552: Object information input unit
153 353 553: graphics processing unit 160 360 560: stereoscopic augmented reality image generation unit
170 370 570: stereoscopic augmented reality image output unit

Claims (12)

A stereoscopic image generator which receives left and right eye images and generates stereoscopic 3D images from the received left and right eye images;
Receives left and right eye images, and is rendered using a graphic processing unit (GPU) based on 3D object generation information including pattern information to be overlayed on predetermined positions of the received left and right eye images An object generator including a graphics processor configured to generate a 3D object of the object; And
A stereoscopic augmented reality image generating unit for generating a stereoscopic augmented reality image by overlaying the generated left and right eye 3D objects on a predetermined position of the left and right eye images of the stereoscopic 3D image; Stereoscopic augmented reality image generating device. The method of claim 1,
A stereoscopic 3D image and a stereoscopic augmented reality image generating apparatus using a WiFi further comprising a; image input unit for receiving the left and right eye images from the binocular camera module and transmits to the stereoscopic image generator and the object generator. The method of claim 1,
An image input unit which receives a single view image from the monocular camera module and transmits the single view image to the image converter; And
A stereoscopic 3D image and a stereoscopic augmented reality image generating apparatus using a ppipp; further comprising a; image conversion unit for receiving a single view image from the image input unit and converts the left and right eye images to a stereoscopic image generator and an object generator. The method of claim 1,
An image input unit which receives an image from the image storage unit and transmits the image to the image conversion unit; And
If the image received from the image input unit is a left eye and a right eye image, it is directly transmitted to a stereoscopic image generating unit and an object generating unit. 3D augmented reality image generating apparatus using a three-dimensional 3D image and a WiFi further comprising an image conversion unit for transmitting. The method according to any one of claims 1 to 4, wherein the object generating unit,
And a pattern recognition unit recognizing at least one of the patterns included in the received left eye and right eye images to extract the pattern information. 3D augmented reality image generating apparatus using a 3D image and a GPU. The method according to any one of claims 1 to 4, wherein the object generating unit,
And an object information input unit configured to receive the 3D object generation information from a user. 3D augmented reality image generating apparatus using a stereoscopic 3D image and a WiFi. Receiving, by the stereoscopic image generator, left and right eye images and generating stereoscopic 3D images from the received left and right eye images;
The object generating unit receives the left and right eye images, and renders using the GPU (Graphic Processing Unit) based on the 3D object generation information including the pattern information to be overlayed on predetermined positions of the received left and right eye images. Generating 3D objects of left and right eyes; And
Generating a stereoscopic augmented reality image by overlaying the generated left and right eye 3D objects to a predetermined position of the left and right eye images of the stereoscopic 3D image by a stereoscopic augmented reality image generating unit; 3D augmented reality image generation method using. The method of claim 7, wherein generating the stereoscopic 3D image comprises:
And receiving the left and right eye images from the binocular camera module and transmitting the left and right eye images to the stereoscopic image generating unit and the object generating unit. 3D augmented reality image generation method using a stereoscopic 3D image and a WiFi. The method of claim 7, wherein generating the stereoscopic 3D image comprises:
The image input unit receives a single view image from the monocular camera module and transmits it to the image conversion unit; And
Receiving a single view image, the image conversion unit converts the left eye and right eye image and transmits to the stereoscopic image generating unit and the object generating unit; stereoscopic 3D image and a stereoscopic augmented reality image generation method using a WiFi. The method of claim 7, wherein generating the stereoscopic 3D image comprises:
The image input unit receives an image from the image storage unit and transmits the image to the image conversion unit; And
If the image received by the image converter is a left eye and a right eye image, it is directly transmitted to the stereoscopic image generating unit and the object generating unit. 3D augmented reality image generation method using a three-dimensional 3D image and JP including a transmitting. The method of claim 7, wherein the generating of the 3D object comprises:
And extracting the pattern information by recognizing at least one of the patterns included in the left eye and right eye images. 3D augmented reality image generation method using a 3D image and a GPU. The method of claim 7, wherein the generating of the 3D object comprises:
And receiving the 3D object generation information from a user. 3D augmented reality image generation method using a stereoscopic 3D image and a WiFi.

Images