WO2009035199A1

WO2009035199A1 - Virtual studio posture correction machine

Info

Publication number: WO2009035199A1
Application number: PCT/KR2008/002304
Authority: WO
Inventors: Young-Dae Kim
Original assignee: Young-Dae Kim
Priority date: 2007-09-15
Filing date: 2008-04-23
Publication date: 2009-03-19

Abstract

A virtual studio posture correction machine using a 3-dimensional real-time character and graphic technology is provided. The machine mixes and displays images captured by a camera together with a character or background in 3-dimensional images using a chroma key, so a dancing person can dance or learn dancing while viewing his or her own posture together with the character's posture using the machine.

Description

Description VIRTUAL STUDIO POSTURE CORRECTION MACHINE

Technical Field

[1] The present invention relates to a posture correction machine, and more particularly, to a virtual studio posture correction machine which enables a user to correct his or her own posture by comparing his or her own motion and a character's motion in a three- dimensional (3D) graphic device. Background Art

[2] People who try to improve their posture may capture their posture on film, and view the captured images repeatedly to compare and analyze the images.

[3] Portable posture training devices are mainly used in golf swing posture correctors, baseball swing training devices, model walking training devices, acting expression training devices, dance training devices, facial expression correctors for education, or in other similar devices.

[4] For example, if a golf swing posture corrector captures the swing posture of someone practicing their golf stroke, he or she may play back, analyze and compare the video captured by the golf swing posture corrector to check his or her own posture, and thus may improve his or her golf stroke. In particular, it is very important to correct golfer's swing posture, because incorrect posture affects the direction a golf ball travels and may cause the golf ball not to reach the full distance of a golf club. Additionally, incorrect swing posture may cause damages to the waist, neck or legs, which are important parts of the human body.

[5] Accordingly, there is a need for an apparatus for instantaneously recording dynamic motions while training in a sport, such as golf, in which a posture is very important, or in entertainment fields in which dynamic motion, such as a model's gait, an actor's form, or a dancer's form, is very important, and for enabling users to repeatedly and easily practice their posture.

[6] Korean Patent Laid-Open No. 2003-0082203 entitled 'portable posture training device capable of recording video' describes the following technology.

[7] Referring to FIG. 1, a portable system includes a liquid crystal display (LCD) display panel 10 disposed on a front side thereof, a hinged support 20 disposed on a rear side thereof, and a built-in digital camera 30 disposed above the panel 10. The portable system has an instant recording function to record video content output from an external video player, such as a television, a video player, an external compact disc (CD) player, a digital versatile disc (DVD) player, a cable broadcasting apparatus, a computer, a cable television or a satellite broadcasting apparatus, for a required period of time and to utilize the recorded video as education teaching materials.

[8] In more detail, the portable system is able to select and record video content output from the external video player, and to capture a user's figure using the digital camera 30, so that a user can correct his or her own posture by comparing it with the recorded video.

[9] However, when a conventional dance training device is used as described above, it is difficult for a user to search for or store desired video. Even when the video required by the user is stored, it is also difficult for the user to compare the desired video to his or her own posture in real time and correct his or her posture.

[10] Additionally, since diverse and complicated postures are required to dance or learn to dance, there are limitations to learning dancing on-line. Furthermore, it is not easy to learn detailed dance steps even when a person registers for a dance academy off-line. It is hard to understand dance moves even when viewing the moves continuously. Disclosure of Invention Technical Problem

[11] Exemplary embodiments of the present invention overcome the above disadvantages and other disadvantages not described above. The present invention provides a virtual studio posture correcting device, which causes motion data regarding motion of various professionals to be displayed as a three-dimensional (3D) character, synthesizes a user's posture with the character in 3D space and shows a result of synthesizing in real time, so that the user is able to correct his or her own posture by comparing it to the character's posture. Technical Solution

[12] According to an aspect of the present invention, there is provided a virtual studio posture correction machine including a camera which photographs a subject; a chroma key unit which systematically removes certain image components from an image output from the camera and outputs only a subject image; a three-dimensional (3D) acceleration graphic unit which forms a 3D virtual studio image and synthesizes the 3D virtual studio image with the subject image acquired by the chroma key unit; and a display unit which displays an image synthesized by the 3D acceleration graphic unit. The 3D acceleration graphic unit may include a 3D image generator which generates a 3D virtual studio using information regarding a character 3D database (DB) and a background 3D DB, the 3D image generator having the character 3D DB, which is constructed using motion capture data of a professional dancer and which contains various dance moves matching the music, and the background 3D DB, which is constructed so as to enable a person who desires to dance or learn dancing to select a favorite background and dance move; and a rendering unit which performs video texturing on the subject image acquired by the chroma key unit, and synthesizes the textured image with the 3D virtual studio, which is generated by the 3D image generator and includes a character image and background image, and which outputs a synthesized image to the display unit.

[13] According to another aspect of the present invention, the 3D image generator may generate a 3D virtual studio using information regarding a character 3D DB and a background 3D DB. The 3D image generator may have the character 3D DB, which is constructed using motion capture data of a professional, such as a professional dancer, a sportsman, a model or an actor, and the background 3D DB, which is changeable according to user selections.

[14] The virtual studio posture correction machine may further include at least one virtual camera, a position of which is adjusted according to preset track information; a screen control through which the at least one virtual camera is selectable; and a joystick which changes position information of the camera and at least one virtual camera and which adjusts a screen of the 3D virtual studio. If a character image and background image selected using the screen control and joystick is combined with the subject image in 3D space, the rendering unit may perform 3D rendering on information regarding movement and arrangement of a combined image in response to a signal input by a user using the joystick, may synthesize the 3D virtual studio with the subject image, and may output a synthesized image in real time.

Advantageous Effects

[15] In the exemplary embodiment of the present invention, a virtual space synthesis technology for showing an actual video of a subject input to a graphic generator by a real camera together with three-dimensional (3D) characters dancing in 3D space is used, so a user who actually dances appears to be in virtual space. Additionally, the position and angles of a plurality of virtual cameras on virtual space may be adjusted, and accordingly the user can learn to dance or improve his or her dance moves while viewing his or her own posture together with the character's posture. Video effects or characters are used as background dancers, so the present invention is equally applicable to a machine for learning to dance or dancing or singing, or a machine for recording video or real-time Internet broadcasting.

[16] For example, if the present invention is applied to a karaoke system, dance moves of a singer of a song selected by a user may be performed on a screen by a 3D character, and a user's posture may also be displayed using a chroma key, so it is possible to learn the dance moves of a singer easily.

[17] Additionally, if the present invention is applied to a golf swing posture training device, a user may select golf swing data of a professional player whose stroke he or she desires to learn, and may view a simulation to compare his or her own posture to a professional player's posture on a screen, and it is thus possible to learn more effectively. Brief Description of the Drawings

[18] These and/or other aspects and advantages of the invention will be more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

[19] FIG. 1 illustrates a dance training device employing a general dance recording and analyzing apparatus;

[20] FIG. 2 is a block diagram showing main function blocks of a three-dimensional (3D) virtual posture correcting device according to an exemplary embodiment of the present invention; and

[21] FIG. 3 illustrates a situation in which a 3D virtual posture correcting device according to an exemplary embodiment of the present invention is applied to a dance training device.

Best Mode for Carrying Out the Invention

[22] Reference will now be made in detail to the present embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. The embodiments are described below in order to explain the present invention by referring to the figures.

[23] A three-dimensional (3D) virtual posture correcting device according to the exemplary embodiment of the present invention is configured based on a virtual studio apparatus described in Korean Patent Application No. 2005-0011164 entitled '3D virtual studio apparatus using video texturing' by DARIM VISION CO. LTD.

[24] FIG. 2 illustrates main function blocks of the 3D virtual posture correcting device according to the exemplary embodiment of the present invention. A virtual overlay technology is used in which video is processed as a single subject in a 3D rendering engine rather than background images overlapping foreground images in a conventional mixer, in order to create a virtual overlay.

[25] An acceleration graphic unit 200 includes a 3D image generator 210 and a rendering unit 220.

[26] The 3D image generator 210 may generate 3D data capable of being output in real time, using prepared moving images, for example a character 3D database (DB) 214 or a background 3D DB 216, which are processed by a 3D data processor 212, by launching a program for designating 3D subjects, namely an editing program such as 3D Studio Max or MAYA.

[27] In this exemplary embodiment of the present invention, a variety of information may be stored in the character 3D DB 214 according to the user's intention.

[28] If the 3D virtual posture correcting device of the present exemplary embodiment is employed in a dance training device, the character 3D DB 214 is constructed using motion capture data of a professional dancer. The character 3D DB 214 contains various dance moves matching the music. In particular, the character 3D DB 214 is constructed enabling a user to select various characters which may be viewed from different angles, so that the user can select a desired posture and learn dancing. Additionally, a plurality of characters may be displayed as background dancers which are dimensionally synthesized on a virtual screen.

[29] The background 3D DB 216 is constructed so that a user who desires to dance or learns dancing may select a desired background, character and dance move. Since the selected background contains pre-stored information on the movement of a virtual camera, preset special effects or the movement of the virtual camera may be performed automatically for each moment when the user dances. For example, if the user selects a background of a nightclub, the lighting and virtual camera may move automatically and the effect for dancing together with a plurality of background dancers may be provided automatically when the user dances. In this situation, the user may set the background differently from the basic settings by editing the background.

[30] Additionally, the background 3D DB 216 may store graphic data representing a large-scale dance studio or a virtual real world (for example, a natural setting such as Niagara Falls) or a fantastic setting such as outer space even when the 3D virtual posture correcting device of the present exemplary embodiment is mounted in a narrow space.

[31] The rendering unit 220 receives the 3D data generated by the 3D image generator

210, performs video texturing on a surface of a 3D subject in real time, and outputs a 3D image. Additionally, the rendering unit 220 performs 3D rendering on the 3D image together with 3D camera images captured by a plurality of real cameras 240 to 24N, and outputs images in real time using the virtual overlay technology. The virtual overlay technology enables the 3D camera images representing subjects captured by the plurality of real cameras 240 to 24N and the 3D image having the 3D data generated by the 3D image generator 210 to be input simultaneously to the rendering unit 220, and processed as an analog image 230, so that the analog image 230 may be displayed.

[32] The main technique of this exemplary embodiment of the present invention is that the selected background DB and character DB are displayed together with subjects in virtual space, so there is no need to limit information regarding the DBs.

[33] For example, the character 3D DB 214 and background 3D DB 216 may contain data for model gait training or data for correcting sports posture, such as a person's golf stroke, or may contain data regarding characters representing dancers or stages matching the music to be applied to a karaoke.

[34] Hereinafter, another exemplary embodiment of the present invention in which the 3D virtual posture correcting device using video texturing is employed in a dance training device will be described in detail with reference to FIG. 3.

[35] FIG. 3 illustrates a set graphic device in which all units to the left of a virtual device dividing line 300 are placed in a virtual space.

[36] The plurality of real cameras 240 to 24N capture a subject 270 dancing on a background 260. Subsequently, a chroma key unit 280 causes only the subject 270 in the captured image to appear to be in any location of a 3D graphic virtual space 320 having the same size as a screen of a virtual camera 310. Such a chroma key function is a technique for removing only a background from a subject displayed together with the background captured by a real camera and for making the background transparent. Since the chroma key function is known to those skilled in the art, detailed description thereof is omitted.

[37] The acceleration graphic unit 200 replaces the background which has been made transparent by the chroma key unit 280 with the 3D graphic virtual space 320 and background 340, which are formed by synthesizing the character 3D DB 214 with the background 3D DB 216, so that a virtual screen may be generated, using the virtual camera 310 included in the 3D image generator 210. The rendering unit 220 synthesizes video information regarding the subject 270 with the 3D graphic virtual space 320 and background 340, so that a synthesized image is input as texture of the 3D data, and then displays a final output image 250, so that the subject 270 appears to dance together with a character 330 in the virtual space.

[38] In order to automatically show various motions captured by the virtual camera 310 in a virtual space using a program, the virtual camera 310 may operate as a Pan/ Tilt/Zoom (PTZ) camera. For example, when a person who sings a song or dances is automatically tracked, the virtual camera 310 may combine the person with a certain background and characters and may cause a combined image to automatically appear on the entire screen. Additionally, in the case of a low-cost system, manual or stationary tracking may be performed in place of automatic tracking.

[39] The virtual camera 310 may be controlled by a screen control or a joystick or a predefined track in a 3D data device.

[40] In more detail, the screen control forms a screen through which a specific virtual camera is able to be selected from among a plurality of virtual cameras and preset background images (background screens or character images). The screen control may be implemented by a touch screen. In this situation, if a user adjusts a camera selected by touching the touch screen, adjustment data obtained by adjusting the camera is transmitted to a virtual camera 310, so that the angle of the virtual camera 310 or the like may be adjusted.

[41] In order to make such adjustment easier, a joystick is additionally required. When the joystick is used, the position of background image and subject image selected using the screen control may be precisely controlled using track information of the virtual camera selected according to the adjusted position of the joystick. Accordingly, the user is able to control the position of images to be combined at a desired camera angle in 3D space.

[42] Therefore, if an image in a certain position of 3D space which an observer is able to view through a virtual camera is displayed using the 3D video texturing technique, a subject may be caused to be positioned in virtual space using the 3D acceleration graphic unit easily, and different positions may be freely shown using the virtual camera. Accordingly, the image may be shown in various directions while being changed in real time, and the camera may seem to move without needing to use a costly camera moving device.

[43] Additionally, a plurality of virtual cameras may be disposed in virtual space, so it is possible to form diverse screens as if a plurality of real cameras operate.

[44] If information on the image captured by the virtual camera 310 is input to the 3D image generator 210, a background 340 is formed according to position information of the virtual camera 310 in virtual space and the background 340 is synthesized with video information regarding the dancing subject 270 acquired in real time by the chroma key unit 280. The 3D character 330 selected from among 3D characters is then added to the image of the subject 270 synthesized with the background 340, so that a composite screen may be displayed on a monitor 250.

[45] If the video texturing technique is used as described above, the user may freely adjust the position of the virtual camera 310, so that the effect may appear as if the plurality of real cameras 240 to 24N move, and superior virtual studio video may be generated.

[46] As a result, a person who desires to learn dancing can learn dancing at home while imitating dance moves of a 3D character based on data acquired by capturing the moves of a professional dancer. Accordingly, there may be the same effect as when the person registers for a dance academy and learns dancing.

[47] As described above, in the exemplary embodiment of the present invention, a virtual image including background 3D data forming a background in virtual space and 3D data of a continuously dancing character is synthesized with video of a subject captured by a plurality of real cameras. The video of the subject is obtained by removing a background using the chroma key function. The subject then appears to be in a location viewed by a virtual camera which moves in a certain direction according to a specific track in 3D virtual space, so that a composite screen may be displayed in real time. Therefore, a dancing person may view his or her own posture on a monitor as if he or she is dancing together with virtual characters in a virtual dance studio.

[48] Additionally, if the virtual camera 310 is automatically tracked according to the moves of a dancing person, the dancing person feels as if he or she is dancing together with professional dancers in a large-scale broadcasting studio or concert hall, and as if a plurality of cameras are moving and photographing him or her. Thus, it is possible to automatically generate a music video by which people can dance or learn dancing.

[49] While the dance training device has been described as an exemplary embodiment of the present invention, the present invention is also applicable to the entertainment field in which dynamic motions, such as sports posture, a model's gait, or an actor's form, are very important.

[50] For example, if the virtual posture correcting device according to an exemplary embodiment of the present invention is connected to a karaoke system, a dance composition of a singer or background dancers suitable for a selected song is stored in a database. If a user selects the song, the dance composition is displayed, so that the user may view characters which dance to the song sung by the user. Additionally, the moves of the user singing the song is displayed after the chroma key function is executed, so it is possible for the user to compare his or her own dance moves to the original moves of the singer using the characters displayed on a screen.

[51] If the virtual posture correcting device according to an exemplary embodiment of the present invention is connected to a golf swing posture training device, golf swing data of a professional player selected by a user may be displayed on a screen as a 3D character, so that the user may view a simulation to compare his or her own posture to the posture of the selected professional player through the screen, and it is possible to accurately correct the posture.

[52] Furthermore, if motion capture data regarding a professional such as a singer or a professional player is updated after a predetermined period of time and the updated data is stored in a database, it is possible to provide a user with a variety of information which continuously changes according to fashion.

[53] The foregoing exemplary embodiments and advantages are merely exemplary and are not to be construed as limiting the present invention. The present teaching can be readily applied to other types of apparatuses. Also, the description of the exemplary embodiments of the present invention is intended to be illustrative, and not to limit the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art. Industrial Applicability

[54] The present invention is applicable to an entertainment apparatus such as a posture correcting device or a dance training device, using virtual studio technology, and in particular, may desirably be applied to a golf swing posture training device and a karaoke system which are able to provide a 3D screen.

Claims

[1] A virtual studio posture correction machine comprising: a camera which photographs a subject; a chroma key unit which systematically removes certain image components from an image output from the camera and outputs only a subject image; a three-dimensional (3D) acceleration graphic unit which forms a 3D virtual studio image and synthesizes the 3D virtual studio image with the subject image acquired by the chroma key unit; and a display unit which displays an image synthesized by the 3D acceleration graphic unit, wherein the 3D acceleration graphic unit comprises: a 3D image generator which generates a 3D virtual studio using information regarding a character 3D database (DB) and a background 3D DB, the 3D image generator having the character 3D DB, which is constructed using motion capture data of a professional dancer and which contains various dance moves matching the music, and the background 3D DB, which is constructed so as to enable a person who desires to dance or learn dancing to select a favorite background and dance move; and a rendering unit which performs video texturing on the subject image acquired by the chroma key unit, and synthesizes the textured image with the 3D virtual studio, which is generated by the 3D image generator and comprises a character image and background image, and which outputs a synthesized image to the display unit.

[2] A virtual studio posture correction machine comprising: a camera which photographs a subject; a chroma key unit which systematically removes certain image components from an image output from the camera and outputs only a subject image; a three-dimensional (3D) acceleration graphic unit which forms a 3D virtual studio image and synthesizes the 3D virtual studio image with the subject image acquired by the chroma key unit; and a display unit which displays an image synthesized by the 3D acceleration graphic unit, wherein the 3D acceleration graphic unit comprises: a 3D image generator which generates a 3D virtual studio using information regarding a character 3D database (DB) and a background 3D DB, the 3D image generator having the character 3D DB, which is constructed using motion capture data of a professional, such as an actor, a sportsman or a model, and the background 3D DB, which is changeable according to user selections; and a rendering unit which performs video texturing on the subject image acquired by the chroma key unit, and synthesizes the textured image with the 3D virtual studio, which is generated by the 3D image generator and comprises a character image and background image, and which outputs a synthesized image to the display unit.

[3] The virtual studio posture correction machine as claimed in any one of claim 1 or

2, further comprising: at least one virtual camera, a position of which is adjusted according to preset track information; a screen control through which the at least one virtual camera is selectable; and a joystick which changes position information of the camera and at least one virtual camera and which adjusts a screen of the 3D virtual studio, wherein, if a character image and background image selected using the screen control and joystick is combined with the subject image in 3D space, the rendering unit performs 3D rendering on information regarding movement and arrangement of a combined image in response to a signal input by a user using the joystick, synthesizes the 3D virtual studio with the subject image, and outputs a synthesized image in real time.

[4] The virtual studio posture correction machine as claimed in claim 1, wherein the display unit is connected to a karaoke system and displays data corresponding to a song selected by the user stored in the 3D acceleration graphic unit.

[5] The virtual studio posture correction machine as claimed in claim 1, wherein the display unit is connected to a golf training system and displays swing data of a professional player selected by the user from among data stored in the 3D acceleration graphic unit together with user's stroke.