KR102124564B1 - Apparatus and Method For Image Processing Based on Position of Moving Light Source in Augmented Reality - Google Patents

Abstract

The present invention discloses a method and apparatus for processing an image by reflecting the position of a moving light source in augmented reality.
According to an embodiment of the present invention, a) a plurality of optical transceivers located at different locations from a light source unit moved by a user transmits and receives beams to each other; b) 3D position and direction of the light source unit based on the 3D information of the light source unit measured by the inertial sensor located inside the light source unit and the beam transmitted or received by one or both of the light source unit and the plurality of optical transceivers , And checking an angle; c) processing an object illuminated by the light source and a virtual object image to be displayed on the background based on the check result in step b); And d) displaying the virtual object image on one or both of the object and the background, wherein the virtual object image is either the shadow of the object generated by the object and the light source or the background. Or it is characterized by a virtual image that is combined with both to realize augmented reality.

Description

Apparatus and Method For Image Processing Based on Position of Moving Light Source in Augmented Reality}

The present invention relates to an image processing method and apparatus, and more particularly, to a method and apparatus for processing an image by reflecting the position of a moving light source in augmented reality.

Recently, the development of an image platform that utilizes new technologies such as augmented reality and immersive images is active. The application area is also widening from the existing game field to various fields such as fine art, advertisement, entertainment, education, and museum exhibition. Therefore, the development of related technologies and the development of various video platforms using them are in need.

Augmented reality refers to a virtual reality image superimposed on the reality. Among the various image display methods for this purpose, the present invention is to develop a method of displaying a virtual object image on the surface of an actual object. For example, an image may be projected onto the surface of an object using a projector, or the monitor may be attached to the surface of an object to display the image as if the monitor is part of the object. Through this, an augmented reality object in which virtual reality is overlaid on an object of reality can be created, and an illusion as if a virtual object exists is caused to the user. Although this method is recognized for its special advantages over other augmented reality video display methods, there have not been many cases in which interactive content is combined with augmented reality video. Therefore, it is necessary to develop a new technology capable of providing a higher sense of reality and immersion through the combination of augmented reality technology and interactive content at this stage.

On the other hand, the tangible image may be defined as delivering the contents of the image through an additional sensation other than visual through the tangible interface. A typical example of such a tangible interface is a car steering wheel controller that a user can directly feel by using tactile sense in a car game. However, these tangible interfaces have been used only in a few typical forms for a long time, and it is necessary to develop cases applicable to new environments. Specifically, in recent years, the precision of the position tracking sensor essential for the sensory interface device has been significantly improved, and various possibilities have been sought accordingly. In particular, if the tangible interface is combined with augmented reality video, it is expected that a new format of interactive video platform can be realized.

The present invention was devised on the basis of the background technology as described above, and further enhances the reality of augmented reality by developing the existing technology, and enables interaction by combining augmented reality and a tangible interface, thereby ultimately enhancing immersion. It is an object to provide an image processing method and apparatus. Through the image processing method and apparatus of the present invention according to various embodiments, it is possible to implement a new type of user experience and to implement a platform that can be used for various contents, as well as to improve artistic and playfulness such as fun, beauty, and uniqueness. Can suggest potential.

A method for processing an image by reflecting the position of a moving light source in augmented reality according to an aspect of the present invention for achieving the above object is a) a plurality of optical transceivers located at different locations from the light source unit moving by the user Transmitting and receiving a beam; b) 3D position and direction of the light source unit based on the 3D information of the light source unit measured by the inertial sensor located inside the light source unit and the beam transmitted or received by one or both of the light source unit and the plurality of optical transceivers , And checking an angle; c) processing an object illuminated by the light source and a virtual object image to be displayed on the background based on the check result in step b); And d) displaying the virtual object image on one or both of the object and the background, wherein the virtual object image is either the shadow of the object generated by the object and the light source or the background. Or it is characterized by a virtual image that is combined with both to realize augmented reality.

In addition, in the augmented reality according to the second aspect of the present invention, an apparatus for processing an image by reflecting a position of a moving light source includes: a light source unit and a plurality of optical transceivers moved by users transmitting and receiving beams to each other; One or both of the plurality of optical transceivers and the light source unit receive three-dimensional information of the light source unit measured by the beam transmitted and received and an inertial sensor located inside the light source unit, and the light source unit and the plurality of optical units Based on the 3D information of the light source unit received from any one or both of the transceivers, the 3D position, direction, and angle of the light source unit are checked and displayed on the object and background illuminated by the light source unit based on the verification result. A control unit for processing the virtual object image; And an image display unit for displaying the virtual object image on either or both of the object and the background, wherein the virtual object image is synthesized with either or both of the shadow or the background generated by the object and the light source unit. It is characterized by being a virtual image to realize augmented reality.

According to the present invention described above, a virtual object image generated based on the position of a light source unit on a real object of reality is synthesized, and the reality of augmented reality can be further enhanced. In particular, by tracking the three-dimensional position of the light source unit and applying the light characteristics as it is to the computer graphic, it is possible to express more precisely and abundantly than the conventionally known augmented reality related light source recognition and application methods.

In addition, according to the present invention, since the user interacts with the augmented reality object through the tangible interface, the user reflects the lighting state according to the position of the light source unit used by the user, and the virtual object has a three-dimensional effect on the real object It is possible to project the image so that the user can directly control the lighting change of the light source unit as well as the improved realism, thereby obtaining an improved immersion and a unique user experience.

1A is a flowchart of a method of processing an image by reflecting the position of a light source according to an embodiment of the present invention.
1B is a block diagram showing the internal structure of an image processing apparatus according to an embodiment of the present invention.
2A and 2B are diagrams comparing before and after projecting a virtual object image according to a first embodiment of the present invention.
3A, 3B, 3C, and 3D are diagrams showing results of reflecting positions of different light sources in the first embodiment of the present invention shown in FIG. 2B, respectively.
4A, 4B, and 4C are diagrams respectively showing image processing results according to the second embodiment of the present invention.
5 is a view sequentially showing a virtual object image with the theme of growth of leaves according to a third embodiment of the present invention.
6 is a view sequentially showing a virtual object image with the theme of a flower and a butterfly according to a fourth embodiment of the present invention.
7 is a view sequentially showing a water circulation process according to a fifth embodiment of the present invention.
8 is a diagram illustrating a user experiencing augmented reality while exploring in the dark according to a sixth embodiment of the present invention.

In describing the embodiments in the present specification, descriptions of technical contents well known in the technical field to which the present invention pertains and which are not directly related to the present invention will be omitted. This is to more clearly communicate the subject matter of the present invention by omitting unnecessary description.

For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically illustrated. Also, the size of each component does not entirely reflect the actual size. The same reference numbers are assigned to the same or corresponding elements in each drawing.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the person having the scope of the invention, and the present invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

Also, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative implementations, it is also possible that the functions mentioned in the blocks occur out of order. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or it is also possible that the blocks are sometimes executed in reverse order according to a corresponding function.

At this time, the term'~ unit' used in this embodiment means a software or hardware component, and'~ unit' performs certain roles. However,'~ wealth' is not limited to software or hardware. The'~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. The functions provided within components and'~units' may be combined into a smaller number of components and'~units' or further separated into additional components and'~units'.

Hereinafter, the present invention will be described in detail with reference to examples and drawings of the present invention.

The present invention relates to a method capable of expressing a higher sense of reality by developing augmented reality in a manner of displaying a virtual object image on the surface of a real object. The present invention, as one example, as a video viewing interface device, may provide a lighting device to the user. And a separate video display device that provides the same characteristics (for example, brightness, color, light emission direction, etc.) to the lighting device through computer graphics (CG) (hereinafter referred to as "CG light source" if necessary) It can be used interchangeably with) to illuminate a virtual object to enhance realism. When the user moves the lighting device, not only the shadow (shadow) of the real object changes, but also reflects the change in the position or position coordinate of the CG light source, and thus the characteristics of the lighting device described above (eg, brightness, color, light emission) Direction, etc.), the image of the virtual object provided by the light of the CG light source having the same characteristics is illuminated. To this end, a special light source device with a position tracking sensor is used as a lighting device, and it is possible to wirelessly transmit the moving position and direction of the light source device to the image processing device in real time. In addition, the virtual object image provided by the light of the CG light source may be rendered in real time on an actual object according to the position and direction of the light source device tracked using the image processing device.

More specifically, the virtual object image rendered in real time may be mapped to a real object surface through an image display device, and the virtual object image overlaps with a shadow generated by illumination of a real light source device, and unique augmented reality Can be implemented. In an embodiment of the present invention, the light of the CG light source provided through the image display device based on characteristics (eg, brightness, color, light emission direction, etc.) according to the position and direction of the light source device according to the movement of the user Since the provided virtual object image is rendered on a real object with the same characteristics as the light source device of the user, a virtual object (hereinafter referred to as "augmented reality object") implemented on the real object by the virtual object image itself It is possible to express three-dimensionally, including shading. As a result, the present invention is capable of expressing an enhanced sense of reality as if the augmented reality object is real. Simultaneously, even when multiple users move using their respective light source devices, in the same manner as described above, virtual object images provided in real time, which are synthesized with shadows of real objects generated based on the position and direction of each light source, respectively. Since it can be rendered on this real object, realistic augmented reality can be provided to all users.

In the present invention, the light source device not only changes the illumination of the virtual object image provided in real time by the light of the CG light source, but also can be utilized as an interface device that interacts with the contents of the corresponding virtual object image, so that it is a realistic interface. It is also possible to further enhance the effect as.

1A is a flowchart of a method of processing an image by reflecting a position of a light source according to an embodiment of the present invention, and FIG. 1B is a block diagram showing an internal structure of an image processing apparatus according to an embodiment of the present invention.

Hereinafter, a method and apparatus for processing an image by reflecting the position and direction of a light source according to an embodiment of the present invention will be described with reference to FIGS. 1A and 1B.

First, as illustrated in FIG. 1B, an image processing apparatus according to an embodiment of the present invention may include a light source unit 110, an optical transmission/reception unit 120, a control unit 130, and an image display unit 140. . Here, the optical transceiver 120 may be implemented with a plurality of optical transceivers. In addition, the light source unit 110 may include a lighting device and a lighting device controller, and as described later, the light amount of the lighting device may be appropriately adjusted based on the visibility of the image display unit 140 and battery usage. in this case. The light source unit 110 may move together with a user, and accordingly, three-dimensional information (position, direction, and angle) of the light source unit 110 may be continuously changed.

Assuming a space in which an embodiment of the present invention is implemented, a plurality of optical transceivers 120 positioned at different predetermined positions are each moved by or with a user, or a light source device 110 or a light source unit 110 (hereinafter referred to as a “light source unit”) And can transmit and receive beams. After the plurality of optical transceivers 120 are transmitted to the light source unit 110, information on each beam received from the light source unit 110 and inertial sensors located inside the light source unit 110 (eg, (accelerometer, each The plurality of optical transceivers 120 may be used to check three-dimensional information (position, direction, and angle) of the light source unit 110 based on data sensed by a speedometer or the like.

Thereafter, the control unit 130 may process and display an actual object illuminated by the light source unit 110 and a virtual object image to be displayed on the background, based on the identified 3D information of the light source unit 110. .

First, in step S110 of FIG. 1A, beams may be received from a plurality of optical transceivers 120 that are installed at different positions by a user or a light source unit 110 moving together with the user.

Thereafter, in step S120, the light source unit 110 and the plurality of transmitting and receiving units 120 transmit and receive each other, and the three-dimensional information of the light source unit 110 measured by the inertial sensor located inside the light source unit 110) Data including is transmitted to the control unit 130, the control unit 130 may check the three-dimensional position, direction, and angle of the light source unit 110 from the data including the three-dimensional information of the received light source unit 110.

More specifically, the plurality of optical transceivers 120 may transmit/receive beams necessary to confirm 3D information of the light source unit 110 from the light source unit 110. That is, the plurality of optical transceivers 120 transmit beams to the light source unit 110 and receive beams from the light source unit 110, and measure data for confirming three-dimensional information of the light source unit 110 based on the received beam Alternatively, the light source unit 110 may receive a beam including predetermined location information of the plurality of optical transceivers 120 from the plurality of optical transceivers 120 to measure data for confirming 3D information. In this case, the light source unit 110 and the optical transmitting and receiving unit 120 may include a rechargeable battery and a power supply, a charging controller, and a wired/wireless data transmission unit, respectively. The rechargeable battery and the wireless data transmission unit enable wireless movement of the light source unit 110 by the user, thereby increasing convenience of use as an image viewing interface. The wire/wireless data transmission unit included in each of the light source unit 110 and the optical transmission/reception unit 120 may transmit information or data measured or confirmed by the light source unit 110 and the optical transmission/reception unit 120 to the control unit 130, respectively. . Here, the above-described rechargeable battery may be composed of a power supply unit and a battery mounting unit, and may be designed to be easily charged even if the user simply mounts the light source unit 110 on the battery mounting unit. Ease of use can be increased.

The control unit 130 uses the 3D information or data of the light source unit 110 received from the light source unit 110 and/or the plurality of optical transceivers 120, respectively, to provide a three-dimensional position, direction, and angle of the light source unit 110. You can check

As an embodiment of confirming the three-dimensional position, direction, and angle of the light source unit 110 as described above, the light source unit 120 is disposed in a plurality of different locations, each of which is a plurality of optical transceivers 120 emitting a laser beam ( It is possible to display a reference point for measuring the 3D information of 110). In order to increase the accuracy of the measurement, a plurality of optical transceivers 120 may be installed at predetermined and different locations. The light source unit 110 may include a plurality of laser beam detectors provided therein to measure its own 3D information (position, direction, and angle) from a reference point.

In addition, as another embodiment of confirming the three-dimensional position, direction, and angle of the light source unit 110, the light source unit 110 may be provided with a marker that can be measured from the outside. At this time, the marker may be configured as a self-luminous marker using an infrared LED or the like in order to increase the accuracy of the measurement. The laser beam detector (for example, a camera or an optical sensor) included in the plurality of optical transmission/reception units 120 may display three-dimensional information (position, direction, and angle, etc.) of the marker provided to the light source unit 110. Can be measured. When a plurality of optical transmission/reception units 120 measure markers provided to the optical unit 110, the plurality of optical transmission/reception units 120 may use the wired/wireless communication to the control unit 130 to control the light source unit 110. Data including 3D information of the measured marker can be transmitted.

Thereafter, in step S130, the control unit 130 may process the actual object illuminated by the light source unit 110 and a virtual object image to be displayed on the background based on the result of checking the 3D information of the light source unit 110. have.

Specifically, the control unit 130 based on the 3D information data of the light source unit 110 and/or the light source unit 110 received from the plurality of optical transceivers 120, the 3D position, direction, and Check the angle, etc. Thereafter, the controller 130 may process a real object illuminated by the light source 110 and a virtual object image to be displayed on the background based on the check result. In this case, the control unit 130 may check the 3D information of the light source unit 110 at predetermined time intervals. As the time interval is shorter, 3D information according to the movement of the light source unit 110 is reflected in real time. , It is possible to create and provide a more realistic virtual object image.

In one embodiment of processing the above-described virtual object image, when the virtual object interacts with the image content according to the position of the light source unit 110, the control unit 130 determines the 3D of the identified light source unit 110 Analyze data for position, direction, and angle. Thereafter, when the analysis result satisfies a predetermined condition, the control unit 130 may generate a predetermined virtual object and apply it to the video content. For example, as a virtual object, a tree is provided to the user, and when the user shines the lighting device of the light source 110 toward the tree, video content in which leaves grow on the tree may be provided to the user. In this case, the control unit 130 may analyze the position, direction, and angle of the light source unit 110 to determine whether the light source unit 110 illuminates the virtual object (tree), and the light source unit ( Based on the time when 110) illuminates the virtual object (tree), an additional virtual object may appear (leaves) or disappear (deciduous) to generate a virtual object image.

As another embodiment of processing the virtual object image, the controller 130 may generate a virtual object according to predetermined 3D model data. Thereafter, the controller 130 may position the generated virtual object at virtual coordinates that match the actual coordinates of the actual object to be synthesized. The reference points of the real coordinates and the virtual coordinates may be set based on respective positions, angles, and directions of the plurality of optical transceivers 120, for example. However, if necessary, the reference points of the actual and virtual coordinates may be arbitrarily set, and a coordinate mapping process between these coordinates may be additionally performed.

Meanwhile, the control unit 130 may generate a virtual light source (which may correspond to the above-described CG light source) that provides the same characteristics as the light characteristics (brightness, color, light emission direction, etc.) of the light source unit 110. . Thereafter, the control unit 130 may place the virtual light source at the same coordinates as the actual coordinates of the light source unit 110 and illuminate the generated virtual object on a real object to perform a rendering operation in real time.

In this case, the controller 130 may modify the virtual object image generated by itself based on the illumination time or the amount of light of the light source 110. That is, the virtual object image projected by the user according to the time when the light source 110 illuminates a real object may be variably changed. In addition, the control unit 130 may perform a graphic operation for allowing the virtual object image to interact with the shadow of the actual object generated by the light source unit 110. Due to the graphic work, the shadow of the augmented reality object in which a virtual object image is synthesized on a real object may provide a different shade from the original shadow of the augmented reality object as described in detail in the embodiment of the present invention described later. In this regard, augmented reality can increase a user's realism.

Meanwhile, although not illustrated in FIG. 1B, the control unit 130 may further include a location tracking device control unit of the light source unit 110, a location confirmation unit of the light source unit 110, and the like. The position confirmation unit applies the separation distance between the three-dimensional information data (position, angle, and direction) of the light source unit 110 checked by the control unit 130 and the light source unit 110 and the plurality of optical transmission/reception units 120 to the light source unit The three-dimensional position, direction, and angle of 110 may be checked, and if necessary, the moving speed, moving direction, and movement pattern of the light source unit 110 may be checked.

Thereafter, in step S140, the virtual object image processed by the control unit 130 may be displayed on the real object and the background illuminated by the light source unit 110 by the image display unit 140. The image display unit 140 may display a virtual object image processed by the controller 130 on a real object and a background. Accordingly, the virtual object image may be synthesized with a shadow of a real object and a real object generated by the light source unit 110 to realize augmented reality.

Meanwhile, the virtual object image may be divided into a virtual object projection image projected on the real object and a virtual object background image displayed on the background. The distinction between the virtual object projection image and the virtual object background image may be performed in the image processing step s130 of the control unit 130 or in the image display unit 140.

The image display unit 140 may project a virtual object image on the real object and the background using a plurality of image display devices. The plurality of image display devices project only a part of the virtual object projected image on the real object, or project only the virtual object projected image excluding the virtual object background image on the real object, or the virtual object projected only on the background of the real object. Only the object background image may be projected, or the virtual object projection image and the virtual object background image may be projected on the real object and the background, respectively, and the at least one projected virtual object image (ie, the virtual object projection image and the virtual object) may be projected accordingly. Any one or both of the background images or some or all of them) may be combined with a real object, and either or both of a shadow or a background of the real object to form an interaction.

Meanwhile, when the image display device is a monitor, the image display unit 140 may display a virtual object image on the monitor. In this case, the monitor can be used to display a virtual reality object in the form of a plane. When the image display apparatus is a projector, since the virtual object image can be projected on a curved surface of a real object, a complex shape virtual object image can be projected on a complex real object (for example, a sculpture). Can be. As a specific example, an augmented reality object in a manner of projecting a virtual object image of a coke can pattern onto a cylindrical shaped object can be implemented to represent a coke can. The image display unit 140 may match the characteristics of the projector (such as an angle of view) with the actual location, angle, and direction, and characteristics of the camera rendering the virtual object image (such as an angle of view) with the actual location, angle, and direction. have. In addition, if necessary, the virtual object image may be warped or trimmed.

When the image display device is a plurality of projectors, the image display unit 140 may perform processing such as warping, trimming, edge blending, and the like so that a plurality of image display areas are seamlessly connected. The virtual object image displayed by the image display unit 140 may be synthesized in a shadow and/or a background of a real object, thereby realizing augmented reality.

In addition, according to the above-described embodiment of the present invention, the virtual object (projection) image displayed on the real object reflects the virtual texture, characteristics, state changes, etc., and has the effect of displaying a 3D augmented reality object. An image that can be provided, and the virtual object (background) image displayed on the background of the real object is a shadow of the augmented reality object itself synthesized with the real object or displayed on the background of the augmented reality object synthesized with the real object. It may be an image that is combined with the shadow to display a shadow corresponding to a predetermined content different from that of the augmented reality object.

2A and 2B are diagrams comparing before and after projecting a virtual object image according to a first embodiment of the present invention.

In the first embodiment of the present invention, it is assumed that the position of the light source is confirmed in real time by the control unit.

First, FIG. 2A is a diagram showing a state before a virtual object image is projected according to a first embodiment of the present invention. In FIG. 2A, the background is illustratively set as the monitor 210. Referring to FIG. 2A together with FIGS. 1A and 1B, the actual object 220, which is the object illuminated by the light source 110, is located at a predetermined location, and accordingly, the user can know the location of the actual object 220 in advance. . The real object 220 may be an object existing in reality (which may correspond to the augmented reality object).

In the first embodiment of the present invention shown in FIG. 2A described above, for convenience of explanation, only one side is a background, and only one light source unit is used as an example, but those skilled in the art have four sides set as the background. , It will be fully understood that a plurality of light sources can be used as illumination.

2B is a view showing a state after a virtual object image is projected according to a first embodiment of the present invention. In the first embodiment, a virtual object image is projected (projected) on the surface of the real object 220.

Before the virtual object image is projected, it can be seen that the real object 220, which appeared as a single color, appears as the augmented reality object 230 in the shape of a brick house with doors and windows when the virtual object image is projected. That is, as shown in FIG. 2B, according to the first embodiment of the present invention, by projecting the virtual object image on the real object 220, the real object 220 in the form of a simple cube before projection of the virtual object image It is possible to implement the image as an image of the augmented reality object 230 having a door frame and a window frame, so that a stereoscopic and realistic feeling can be expressed to the user. That is, in the embodiment illustrated in FIG. 2B, shadows may be displayed on the door frame and the window frame of the augmented reality object 230 that are not present on the real object 220 before projecting the virtual object image. The shadow displayed on the augmented reality object 230 may be displayed using a plurality of image display devices.

In addition, it can be seen that the shadows 240 and 250 of the augmented reality object 230 are generated on the monitor 210 that is the background. The shade 240 is displayed on the monitor 210 as a shadow of the entire virtual object image. That is, the controller 130 virtually generates the shadow of the augmented reality object 230 on the monitor 210 in the background in accordance with the light projection angle of the light source 110. The image of the shadow 240 constitutes a virtual space as if there is a virtual wall at the position of the monitor 210, indicating that the upper part of the shadow 240 of the augmented reality object 230 is projected on the virtual wall Doing.

Another shade 250 is a shadow of the augmented reality object 230 formed on the actual bottom surface by the actual light emitted from the light source unit 110.

In the first embodiment, connection portions between the shades 240 and 250 may be matched. Assuming that the monitor 210 is attached to the floor, it is difficult for a user to visually distinguish a virtual object image from reality, so that a high reality can be experienced.

3A, 3B, 3C, and 3D are diagrams showing results of reflecting positions of different light sources in the first embodiment of the present invention shown in FIG. 2B, respectively.

Specifically, FIGS. 3A, 3B, 3C, and 3D show shadows changing in real time when a user changes the position and/or angle of the light source unit 110 in the first embodiment.

More specifically, FIG. 3A is a diagram illustrating a case in which the light source unit 110 is located at the upper right of the augmented reality object 220 with reference to FIG. 2B. The user moving the light source unit 110 has the brightest right side of the augmented reality object 220 and a shadow of the augmented reality object 220 itself on the left side of the augmented reality object 220, and augmented reality object 220 on the background. ) It can be recognized that there is a virtual wall because the entire shadow 320 is formed. It can be seen that the top 310 of the augmented reality object 220 is bright.

FIG. 3B is a view illustrating a case in which the user moves the light source unit 110 to the lower right in the embodiment of FIG. 2B with reference to FIG. 2B. Compared to the case of the embodiment shown in FIG. 3A, a shadow 330 is formed on the upper portion of the augmented reality object 220, and the background changes the entire augmented reality object 220 by reflecting the changed position of the light source unit 110. It can be seen that the shade 340 is generated.

FIG. 3C is a view showing a case in which the user moves the light source unit 110 to the lower left in the embodiment of FIG. 2B with reference to FIG. 2B. In the embodiment of FIG. 3C, similar to the embodiment shown in FIG. 3A, each shade reflecting the movement of the light source unit 110 on the upper 350, right 360, and background of the augmented reality object 220. (350,360,370) can be identified. The shadows 350, 360, and 370 may be made through projection of a virtual object image to display the augmented reality object 220 as shown in FIG. 3C.

FIG. 3D is a diagram illustrating a case where the user moves the light source unit 110 to the upper left in the embodiment of FIG. 2B with reference to FIG. 2B. Compared to FIG. 3C, it can be seen that the movement of the light source unit 110 is reflected from the shadows 330 and 390 of the upper 380 and the right 390 of the augmented reality object 220. If the light source unit 110 moves to a sufficiently high position, a shadow of the augmented reality object 220 may not be generated on a preset virtual wall.

4A, 4B, and 4C are diagrams respectively showing image processing results according to the second embodiment of the present invention.

Specifically, FIGS. 4A, 4B, and 4C respectively show a case in which the shadow itself projected on the background changes in the case of the first embodiment shown in FIGS. 2A and 2B. The change of the shading itself may be implemented by utilizing a shadow distortion phenomenon in which a shading generated by light is distorted according to an angle to show a shading different from the shading to be originally generated. As an example, using CG (computer graphics) to create a virtual shade of an augmented reality object, Other shapes can be created. The other form of shading may be set in a form imaginable from an augmented reality object. For example, in the embodiment of FIG. 4A described below, the augmented reality object 410 is a cuboid block, but the virtual shade 420 displayed in the background shows the house. The second embodiment utilizes a well-known shadow distortion phenomenon known to the user, and allows the user to feel an intellectual play, stimulates the imagination, and provides a unique augmented reality experience by engaging with the mystery of augmented reality. .

Specifically, FIG. 4A shows a case in which the light source is located at the bottom left of the augmented reality object. Compared to the first embodiment shown in FIGS. 2A and 2B, in the second embodiment of the present invention shown in FIG. 4A, a brick texture is not expressed on the augmented reality object 410, but the light source unit 110 ), it can be seen that the actual shadow appears on the top of the augmented reality object 410. In addition, a virtual shadow 420 of the augmented reality object 410 may be generated on the monitor in a direction corresponding to the actual shadow. In addition, the virtual shadow 420 may be displayed as a virtual shadow 420 showing a house with a chimney and a window rather than a cube of a cube corresponding to the simple augmented reality object 410.

The light source unit 110 used in FIGS. 4A, 4B, and 4C may be in the form of a spotlight, and shadows may appear only on bright portions illuminated by the light source unit 110. In addition, although the chimney smoke and the shadow of the person are not expressed in the virtual shade 420 of FIG. 4A, the chimney smoke and the shadow of the person may be expressed according to the three-dimensional position, direction, and angle of the light source 110. Can be.

FIG. 4B illustrates a case in which the user moves the light source unit 110 or the shadow 430 projected on the background varies based on the illumination time or the amount of light of the light source unit 110. The user may change the angle and distance from the augmented reality object by moving the light source unit 110, and accordingly, the virtual shadow 430 projected on the background may be changed. When the embodiment shown in FIG. 4B is compared with the embodiment shown in FIG. 4A, the user can see that the virtual shadow 430 has been changed by changing the angle and distance between the light source unit 110 and the augmented reality object. have.

Specifically, when the user changes the distance from the augmented reality object by moving the light source unit 110, chimney smoke that was not expressed in the shadow 420 of FIG. 4A as the lighting area of the light source unit widens. Can be expressed. In addition, when the user changes the angle with the augmented reality object by moving the light source unit 110, only one side of the augmented reality object may be illuminated. According to the change of the angle, the virtual shade 430 is changed so that only one side and a window of the virtual object (house) can be expressed.

4C shows another case in which the user moves the light source unit 110 or the shadow 430 projected on the background is variable based on the illumination time or the amount of light of the light source unit 110. Similar to the embodiment shown in FIG. 4B, when the embodiment shown in FIG. 4C is compared with the embodiment shown in FIG. 4A, the user changes the angle and distance between the light source unit 110 and the augmented reality object It can be seen that the virtual shadow 440 has been changed.

Specifically, the user may illuminate the other side of the light source unit 110 and the augmented reality object by changing the angle of FIG. 4B. The virtual shadow 440 generated by the lighting of the user may be generated in the background (monitor) in correspondence to a direction matching the actual shadow. In addition, a virtual object (person) not represented in FIG. 4A may be expressed according to the angle and distance between the light source unit 110 and the augmented reality object.

4A, 4B, and 4C illustrate embodiments of virtual shadows 420, 430, and 440 that change based on at least one of an angle and a distance between a light source unit 110 and augmented reality objects that the user changes. And the user can experience the augmented reality realized by the interaction of the actual shade and the virtual shade by manipulating the light source 110.

In addition, since the augmented reality uses light and shade as a medium, it can be implemented by utilizing light characteristics according to a predetermined theme. For example, augmented reality may be implemented by combining a virtual object image and a shadow of an augmented reality object, and augmented reality may be implemented in a manner in which the shadow of a virtual object expressed using a shadow distortion phenomenon is different. The subject can be varied as described below. And the user can have fun or have a special experience by experiencing the augmented reality.

5 is a view sequentially showing a virtual object image with the theme of growth of leaves according to a third embodiment of the present invention.

A tree may be provided to the user as an augmented reality object at a predetermined location (510). The tree may be provided as a virtual object image in a form without leaves, or may be implemented as a real object. When the user illuminates the light source unit 110 on the tree (augmented reality object) for a certain time, the virtual object image displayed by the image display unit 140 and the augmented reality object may appear to grow leaves on the tree branch. (520).

In addition, as the time for the user to illuminate the augmented reality object (tree) increases, the virtual object image may change. In the third embodiment, the color of the leaves generated by the virtual object image is changed to fall leaves (530), and the leaves fall (540) to return to the original leafless form (510). The virtual object image may be repeated based on the time when the user illuminates the light source unit 110, and the user may repeatedly experience the cycle of life.

6 is a view sequentially showing a virtual object image with the theme of a flower and a butterfly according to a fourth embodiment of the present invention.

As in FIG. 5, a plurality of flower gardens in which flowers are not expressed as augmented reality objects at predetermined positions may be provided to the user (610). In addition, the flower may be expressed by the virtual object image only on the augmented reality object (flower garden) illuminated by the user with the light source unit 620. If the flower is expressed for a certain period of time or more, a bee following the flower may be expressed as a virtual object image (630).

Then, when the augmented reality object (flower field) illuminated by the user is changed, flowers and bees may be expressed only in the illuminated flower field (640). In the fourth embodiment, a user may experience using a light source unit 110 like the sun to grow natural objects.

7 is a view sequentially showing a water circulation process according to a fifth embodiment of the present invention.

5 and 6, water and river (water level of the river may be 0) as an augmented reality object at a predetermined location may be provided to the user (710). When the user illuminates the water using the light source 110, water vapor may be generated and expressed as a virtual object image (720). When the time to illuminate the water is increased, the generation of water vapor becomes active (730), and a cloud formation may be expressed as a virtual object image (740).

In addition, when a cloud is generated, it may be expressed as a virtual object image (750) when it rains (740), or when the water level of the river realized by the augmented reality object increases and the fish living in the river increase (750). The user can experience the circulation process of the water by illuminating the light source unit 110.

Based on the embodiments of FIGS. 5, 6, and 7, interactive storytelling describing a causal relationship of an ecosystem may be provided. In addition, the above embodiments may be used as interactive video material for children's education, and according to the present invention, an effect of developing intellectual and emotional abilities of children and increasing imagination can be achieved.

8 is a diagram illustrating a user experiencing augmented reality while exploring in the dark according to a sixth embodiment of the present invention.

The user may be composed of at least one explorer (810, 811) illuminating the augmented reality object using the light source unit (110) and a party (812, 813) not using the light source unit (110). The space experienced by the user may be implemented such that a virtual object image is expressed in response to only the light source device provided to the user (which may correspond to the light source unit 110 described above) without lighting.

Various augmented reality including the first to fifth embodiments may be implemented in the space (820, 830), and the user can use the light source unit 110 having a limited emission angle of light to darken it with a flashlight. You can experience exploring the inside. Accordingly, when the user illuminates light using a light source device such as a flashlight to find the hidden augmented reality objects one by one, each augmented reality object and a shadow are checked, and accordingly, each augmented reality constituting the user to configure the exhibition space Causal relationships between objects and rules of space can be discovered. The sixth embodiment may use a flashlight exploration in the dark as a theme of a user experience, to provide a immersive experience to the user.

As described above, in the present invention, since the user can control the light source unit 110 to move itself, the augmented reality reality can be increased, thereby providing a unique experience and fun to the user. That is, a user's action or motion may cause a unique immersive experience that increases the sense of reality. In addition, even in the case of a user without the light source 110, it is possible to experience the same realism increase as other users by watching the actions or actions of other users with the light source 110.

The embodiments of the present invention disclosed in the specification and drawings are merely intended to easily describe the technical contents of the present invention and to provide specific examples to help understanding of the present invention, and are not intended to limit the scope of the present invention. It is apparent to those skilled in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

Claims (10)

In the method of processing an image by reflecting the position of a moving light source in augmented reality,
a) a plurality of optical transceivers located at different locations from the light source unit moved by the user, transmitting and receiving beams to each other;
b) 3D position and direction of the light source unit based on the 3D information of the light source unit measured by the inertial sensor located inside the light source unit and the beam transmitted or received by one or both of the light source unit and the plurality of optical transceivers , And checking an angle;
c) processing an object illuminated by the light source and a virtual object image to be displayed on the background based on the check result in step b); And
d) displaying the processed virtual object image on one or both of the object and the background,
The virtual object image is characterized in that the image that is combined with any one or both of the shadow or the background of the object generated by the object and the light source to implement augmented reality,
Step b),
Checking the three-dimensional position, direction, and angle of the light source unit at predetermined time intervals; And
And transmitting the result confirmed at the predetermined time interval to the image processing apparatus. delete The method of claim 1, wherein step c),
Generating the virtual object image to be displayed on the object and the background illuminated by the light source unit based on the confirmation result in step b);
Correcting the generated virtual object image based on an illumination time or light amount of the light source unit; And
And performing a graphic operation for interacting the shadow of the object generated by the light source unit with the modified virtual object image. The method of claim 1, wherein step d),
Distinguishing a virtual object projection image projecting the virtual object image onto the object and a virtual object background image displayed on the background; And
And displaying at least one or both of the distinguished virtual object projection image and the virtual object background image using at least one image display device,
At least one or both of the virtual object projected image and the virtual object background image displayed by the image display device is combined with either or both of the object and the shadow of the object or the background to realize augmented reality. How to do. According to claim 1,
The virtual object image displayed on the object is an image that reflects at least one of a virtual texture, characteristic, and state change,
The method of claim 1, wherein the virtual object image displayed on the background is an image that is synthesized with the shadow of the object and expresses predetermined content. In the apparatus for processing an image by reflecting the position of a moving light source in augmented reality,
A light source unit and a plurality of optical transceivers that are moved by users transmitting and receiving beams to each other;
One or both of the plurality of optical transceivers and the light source unit receive three-dimensional information of the light source unit measured by the beam transmitted and received and an inertial sensor located inside the light source unit, and the light source unit and the plurality of optical units Based on the 3D information of the light source unit received from any one or both of the transceivers, the 3D position, direction, and angle of the light source unit are checked and displayed on the object and background illuminated by the light source unit based on the verification result. A control unit for processing the virtual object image; And
And an image display unit for displaying the virtual object image on one or both of the object and the background.
The virtual object image is characterized in that the virtual image to realize augmented reality by being synthesized with either or both of the background or the shadow generated by the object and the light source unit,
The control unit,
And confirming the three-dimensional position, direction, and angle of the light source unit at predetermined time intervals, and transmitting the confirmed results at the predetermined time intervals to an image processing apparatus. delete The method of claim 6, wherein the control unit,
Based on the check result, the virtual object image to be displayed on the object and the background illuminated by the light source unit is generated, and the generated virtual object image is corrected based on the illumination time or the amount of light of the light source unit, and the correction is performed. And performing a graphic operation for interacting with the virtual object image and the shadow of the object generated by the light source unit. The method of claim 6, wherein the image display unit,
Distinguish between the virtual object projection image projecting the virtual object image on the object and the virtual object background image displayed on the background, and using at least one image display device to distinguish among the distinguished virtual object projection image and virtual object background image. Indicate at least one or both,
At least one or both of the virtual object projected image and the virtual object background image displayed by the image display device is combined with either or both of the object and the shadow of the object or the background to realize augmented reality. Device. The method of claim 6,
The virtual object image displayed on the object is an image that reflects at least one of a virtual texture, characteristic, and state change,
And the virtual object image displayed on the background is an image that is synthesized with the shadow of the object to express predetermined content.

Images