KR20020017576A - System and method for motion capture using camera image - Google Patents

Abstract

PURPOSE: A motion capture system and a method using images are provided to extract motion information of an object from information learned by analyzing the motion of the object using images to apply the motion to a virtual character. CONSTITUTION: A motion capture system using images includes an image capturing means(10) for capturing an image of an object, a digital signal converter(20) for converting the captured image into a digital signal, and a digital signal processor(30) for processing the digital signal to analyze the image so as to recognize a corresponding object from information learned in advance and calculating coordinates of each of parts of the recognized object. The system further has a main controller(50) for extracting motion information of the object from the input image, controlling a series of processes for applying the motion information to a corresponding character, and outputting coordinate information of the recognized object, a character control processor(60) for determining the motion of the character, and a memory(40) for storing the learned information with respect to the object and calculation information used for calculating motion tracking coordinates from the input image.

Description

System and method for motion capture using camera image}

The present invention relates to a motion capture device for tracking a motion of a moving object and applying the motion to a virtual character. The present invention relates to a motion capture system and a motion capture method using an image that can acquire and track a motion therefrom and apply the motion to a virtual character.

With the development of computer technology, it is possible to realize a virtual world different from reality, and also to realize a virtual image that is not real.

Such virtual images have been used a lot through movies, 3D animations, etc. In such images, the reality of moving the target object in reality is important.

Therefore, many devices have been developed and used to track the movement of a target object and apply such movement to the movement of a virtual character.

Such a device is called motion capture. In general, motion capture is a technology that extracts and analyzes human movements and combines them with three-dimensional cyber characters to make the three-dimensional characters equal to humans in virtual space. to be.

Various devices are used for such motion capture.

Most of the current motion capture systems that use markers or sensors on the human body measure the position and direction of each joint of the human body by analyzing marker images captured by the camera or data from the sensors.

Depending on how the markers or sensors used here are divided into four, each is called an ultrasonic, prosthetics, magnetic, optical method.

Among these, devices using magnetic and optical methods are the most widely used at present.

The ultrasonic method includes a sensor for generating a plurality of ultrasonic waves and three ultrasonic receivers.

Ultrasonic sensors attached to each of the actors' tracks for tracking and detecting movements sequentially generate ultrasonic waves, and calculate the distance from the sensor to the receiver using the time taken for the ultrasonic waves to be received by the receiver.

The three-dimensional position of each sensor can be obtained by triangulation principle using the values calculated at each of three receivers.

The problem with the ultrasonic method is that first, the sampling rate is not high and the number of ultrasonic generators that can be used at the same time is not large.

Second, the sensor is large, the actor's operation can be unnatural, and third, the sound device is affected by the reflection of the ultrasonic wave.

However, this method is advantageous in that it is possible to perform real-time processing and is inexpensive due to a small amount of calculation required for location measurement without being subjected to environmental interference.

In addition, the prosthetic motion capture system is composed of a complex consisting of a potentiometer and a slider for measuring the joint movement of the performer.

Since this system does not have a receiver, it is an absolute measuring device that is not affected by any interference or influence from other environments.

As a result, little initial setup is required, and no expensive studio environment is required.

In addition, the device is inexpensive compared to other devices, and has the advantage of acquiring motion data with a very high sampling frequency.

However, the system using the prosthetic method has a disadvantage in that the operation of the prosthetic device must be attached to the body of the performer, and the accuracy of the prosthetic device varies depending on how precisely the prosthetic device is positioned at each joint of the performer.

In general, the prosthetic system is used together with the existing animation production technology, keyframe method and motion control method, to generate keyframes or step motions.

In addition, the magnetic system attaches a sensor to measure the magnetic field at each joint of the performer, and calculates the change in the magnetic field measured by each sensor as a spatial change when the actor moves near the magnetic field generating device. That's the way it is.

Each sensor, magnetic field generator, and main unit are connected by cable. Recently, a wireless system is showing a line.

The advantages of magnetic systems are that they are relatively inexpensive, cost-effective, and enable real-time processing.

However, in the case of a wired sensor, the cables connected from the sensor limit the performance of the performer, which makes it difficult to express complex and fast movements naturally.

To compensate for this drawback, wireless systems that connect cables to radio transmitters have already been developed and marketed, but also large transmitters must be attached to the performer's body.

In addition, since it is possible only in the magnetic field region, the operating radius is limited.

Thus, the magnetic method is suitable for capturing motion moving in real time within a radius of 1 to 3 m.

The optical system attaches reflective markers that reflect infrared light to the main segment of the performer and generates two-dimensional coordinates of the reflected markers using a camera with three to six infrared filters.

Two-dimensional coordinates captured by multiple cameras are analyzed by a dedicated program and used for calculation and animation in three-dimensional space.

One of the advantages of the optical system is its high sampling frequency, which is useful when capturing very fast movements, such as the movement of a sports player.

In addition, since the size of the marker attached to the actor is small and light, the actor can freely express the movement, and can process more than 200 markers to simultaneously capture the movement of multiple actors.

Capable of capturing in a large space (5 * 5 * 2.5m) compared to other systems, the capture accuracy is higher than other products.

The disadvantage of the optical system, however, is that the marker's trajectory is lost for a while when the marker is covered by another object.

This problem requires post-processing of the motion data, which is a big obstacle to making real-time motion capture difficult.

In addition, this method is very expensive compared to other methods is a disadvantage.

According to the present invention, a corresponding object is recognized from previously learned information from an image input using a camera, which is an image acquisition means, and the movement of the recognized object is analyzed. It is to provide a motion capture that can apply the motion of) to the corresponding virtual character.

1 is a flowchart schematically illustrating an execution process of a motion capture method using an image of the present invention.

FIG. 2 is a flowchart illustrating a process of recognizing a target object and calculating movement coordinates of the recognized object in the execution procedure as shown in FIG. 1.

3A and 3B illustrate an example of motion of an input two-dimensional image.

4 is a diagram showing an installation example of a camera for using the multi-coordinate calculation method.

5 is a block diagram showing the configuration of a virtual reality game apparatus to which the present invention system is applied.

FIG. 6 is a diagram for one example of recognizing a face outline of a person for controlling a cyber character expression in the present invention. FIG.

7A and 7B illustrate an example of learning or detecting a pattern of a sign language in application to a sign language system in the present invention.

Motion capture system using the image of the present invention,

Image acquisition means for acquiring an image of a target object for extracting motion;

Digital signal conversion means for converting an image input from the image acquisition means into a digital signal capable of digital processing;

Digital signal processing means for processing the converted digital signal, analyzing the input image to recognize a corresponding object from previously learned information, and calculating coordinates for each part of the recognized object;

Main control means for extracting the motion information of the target object from the input image, controlling a series of processes for applying to the corresponding character, and outputs the corresponding coordinate information of the object recognized from the digital signal processing means;

Character control processing means for receiving the recognition information of the target object and the calculated coordinate information from the main control means to determine the movement of the corresponding character;

And memory means for storing arithmetic information for calculating the motion tracking coordinates from the inputted image and the learning information about the object.

The image capture device of the present invention having such a configuration can analyze the pure input image of the moving object obtained from the image acquisition means, obtain coordinates for each part of the object, and extract motion information of any character therefrom. Is characterized by the fact that

The image acquisition means is composed of one or more cameras, it is installed at each position of the target object for extracting the motion information.

In this way, the input image obtained from the image acquisition means is converted into a signal capable of digital signal processing through the digital signal conversion means, and the converted digital image signal is input to the digital signal processing means.

The digital signal processing means recognizes the corresponding object by comparing the learning information about the object already learned about the input image and stored in the memory means, and detects the motion coordinates of the recognized object.

The motion information of a corresponding character is determined according to the information on the recognized object and the detected coordinates, and the character control processing means controls and displays the character.

As described above, the motion capture system and the capture method using the image of the present invention which detects the movement of the target object from the input image will be described.

Learning through video images for each object, and storing the learned information;

Taking a target object for extracting motion information and dividing the input image into a predetermined size so as to have connectivity between input image frames through a preprocessing process;

Recognizing a target object based on the information learned from the image obtained through the process,

The recognized object is characterized in that the process of calculating the movement coordinates for each object by extracting the amount of movement between the frame and the object for each object already pre-learned after the center coordinate adjustment.

And, the process of recognizing the target object,

Extracting a motion vector component of the object from a previous input frame;

Retrieving the target object by performing motion estimation on the target object of the next frame from the extracted motion vector component;

Recognizing and labeling an object when a target object exists at a corresponding coordinate estimated by the search result;

If the target object does not exist in the coordinates whose motion is estimated as a result of the search, the target object is searched for in each direction from the corresponding position to find and label the target object.

In addition, in the initial input frame and the second input frame in which there is no previous input frame necessary to search for the target object through motion estimation, the target object on the coordinate set by learning is recognized, and the reference coordinate is also calculated in the calculation of the movement coordinate. It is characterized in that the calculation.

The motion capture method in the motion capture system using the image of the present invention having such a feature, in the process of calculating the motion of the object, first recognizes the type of the object and recognizes the motion component according to the time series between frames. To detect and digitalize this, it will be described with reference to the accompanying drawings, FIGS. 1 and 2 as follows.

FIG. 1 is a flow chart schematically showing an execution procedure of a motion capture method in a motion capture system using an image of the present invention, and FIG. 2 is an object recognized in the execution procedure as shown in FIG. A flowchart illustrating a process of calculating the moving coordinate of.

It learns and stores the object itself (target object) that causes actual movement and recognizes it from the stored information.

In the present invention, before the motion capture, for example, the head, the chest, and each specific part of the human body are learned from a corresponding general input image.

In this way, the learned information is stored, and from this, the target object in the video image which is sequentially input thereafter is found.

First, image frames are sequentially input from one or more image acquisition means, and the input image is preprocessed to perform a process having connectivity between the frames.

In this way, the main control unit recognizes the shape of the object based on the learning information previously learned and stored through the image frame in which the preprocessing process is completed, and recognizes it.

Then, the object recognized as described above undergoes the coordinate adjustment of the center, and based on the previously learned information on the recognized object, the movement amount is extracted between the frame and the frame to calculate the coordinates of the moving object.

At this time, in recognizing the corresponding object and calculating the coordinates, the target object to track the movement is found by searching whether there is an object corresponding to the previously predicted position through the motion estimation.

As shown in FIG. 2, when the next frame N + 1 is input, it is determined whether the frame N-1 before the current frame N exists.

Here, the current frame (N) means a frame that is currently being processed, the previous frame (N-1) is a moving position of the target object to be tracked through the motion estimation for the next frame (N + 1) from the current frame (N) In order to estimate the target object and to calculate the movement coordinates, the next frame (N + 1) means the input frame of the next target for motion tracking.

When the previous frame N-1 is present, the motion vector component between the current frame N is extracted from the previous frame N-1, motion estimation is performed, and the corresponding input of the next input frame N + 1 is performed. The position of the object to be predicted is predicted, and the object to be searched, that is, the object to be tracked, is searched based on the learning information.

If the object exists at the corresponding position, the object is recognized and the coordinates of the recognized object are set.

However, when the target object does not exist at the predicted position, the coordinates are set by searching for the target object to be found in each direction (four directions) from the predicted position.

In this case, in the case of the first frame to be initially input, the target object is found by searching based on the position set by the learning.

For example, in the case of a person, each object (target object) such as a head, an arm and a leg at each position is found from a standing shape.

Even in the second frame, since there is no previous frame of the first frame that is the current frame, the target object is searched from the position set by the learning from the first frame. From the third frame, when the next frame is input, the previous frame is Since it exists, as described above, the target object is searched and found to set the coordinates.

Tracking of 3D coordinates of 2D image should basically support the distance error of input image.

This means that the input video cannot be distinguished on a single screen. In other words, at least two cameras should be installed at different points in order to be at least three-dimensional, but in the present invention, rather than extracting the image using the stereo vision as described above, the target object is recognized and tracked and then three-dimensional in three dimensions. Dimensional components can be extracted.

3A and 3B are two-dimensional images. However, if the movement of each object is represented by lines and angles, three-dimensional images can be inferred.

First, the height and angle between point A and point B have been changed, and the length between point A and point B becomes short.

Therefore, the three-dimensional motion can be inferred only by the angle between each input component and the distance between the points.

That is, the points A and B decrease the distance of A and B when A moves to the three-dimensional spatial coordinate axis, and the angles of A and B also change at different angles to calculate the actual spatial coordinate axis of the object. will be.

First, the object to calculate the motion extracts the coordinate data in two dimensions, and when the pattern of the motion is overlapped or lost, the input image itself is compared with the previously learned data on the spatial coordinate axis in a special case to predict the motion. will be.

The above technique is used when the installed number of cameras, which are image acquisition means, is small, and when the installed number of cameras is increased, the 3D coordinates can be obtained by continuously tracking the corresponding coordinates and objects to the next camera.

Figure 4 shows an installation example of a camera for using such a multi-coordinate calculation method, the multi-coordinate calculation method, the three-dimensional change of the two-dimensional coordinates is extracted from the angle of the moving object at the input end of the front portion It refers to the processing of the point coordinates and the object's characteristics.

Of course, in order to obtain more accurate data, the number of input terminals (the number of cameras) must be large, but it can be effectively used at a very low cost to detect a specific movement (that is, a limited motion).

In addition, since the object itself is recognized instead of the point of the object, it is efficient to increase the number of input stages to obtain more accurate spatial coordinates.

The present invention as described above is applicable to each of the character control processing means, which can be applied as shown in the drawings, and will be described with reference to the accompanying drawings.

[Virtual Reality Game Device]

5 is a block diagram of a virtual reality game apparatus to which the motion capture system of the present invention is applied.

The configuration thereof will be described by a camera 10 for acquiring an image of a gamer and a video codec 20 for converting an image input from the camera 10 into a signal capable of digital signal processing. And a digital signal processor (DSP) 30 which analyzes an image frame input from the video codec 20 to recognize a tracking object and calculates coordinate information of the recognized object, and a dictionary for recognizing the target object. A main memory microprocessor 50 for controlling a series of processes for applying the learning information of the object learned to the object, the calculation information for calculating coordinates, and the applied character to the corresponding target character from the detected calculation coordinates. ), And a series of control programs for controlling the game program are built-in, and the game program is generated by using the operation coordinate information of the target object input from the main controller microprocessor 50. It is configured to include a game processor 60, which controls the movement of the target character in the program.

The virtual reality game apparatus to which the present invention is applied is characterized in that the characters in the game program having a predetermined content plot can be moved according to the motion of the gamer.

The camera 10 receives an image of the gamer's movement, converts the image frame into a digital signal through the video codec 20, and converts the image frame into a signal that can be processed by the digital signal processor 30.

The digital signal processor 30 analyzes the video frame of the gamer input as described above, recognizes each body part of the gamer from information previously learned and stored in the memory 40, and calculates coordinates for each part.

The coordinate information calculated as described above is input to the game processor 60 through the main controller microprocessor 50, and the game processor 60 controls the movement of a specified character using such coordinate information.

Such an embodiment of the present invention will be characterized in that by applying the motion capture system using the image of the present invention to the game machine device, the gamers' motion is reflected in the game as it is.

[Cyber Character Expression Control]

In addition, the present invention can apply a change in facial expression to a character expression in cyberspace in the case of a chat or an avatar which is his alter ego in cyberspace.

The same device as shown in FIG. 5 is applied as it is, but the game processor may be applied as a chat or a separate program processing device.

As in Figure 6, the recognition of the facial expression first focuses on finding the outer appearance of a person.

First, the image pattern between the eye and the eye is detected and face is recognized based on the image pattern.

The recognized face calculates movements of eyes, nose, eyebrows, lips, and the like according to neural network information previously learned for each part.

That is, each part of the face, which is the target object, is recognized, and as described above, the coordinate information is calculated, and a corresponding processor can be expressed by applying to the virtual character in the cyber space from such information.

[Application example to sign language system]

In addition, if the present invention is applied, the bar can be configured to output a sign language as text or voice when analyzing the movement of the hand as an image.

The present embodiment is characterized in that when the learned pattern is extracted from the matching pattern during the movement of the hand, it is displayed by voice or text.

As shown in FIGS. 7A and 7B, in sign language, a hand movement is learned in advance, and information indicating the movement is stored together in a memory.

In this embodiment, the target object is recognized with respect to the input image frame, that is, each part of the hand is recognized, and a pattern related to the direction of the finger or the like is extracted.

Thereafter, the directionality test is performed to compare the extracted pattern with the movement pattern of the hand stored as the learned information.

When the same pattern is detected, the meaning of the pattern is read and transmitted to an image output means for outputting as text or an audio output means for outputting as audio, and the meaning of sign language is displayed as text or voice.

Such a present invention can be variously applied in the related art to which a conventional motion capture system is applied.

As described above, by implementing the motion capture system by analyzing the motion of the target object using the image to extract the motion information of the target object from the learned information and apply the motion to the virtual character. In addition, the use of equipment for extracting the movement of the target object is simple and does not require expensive equipment.

Claims (5)

Image acquisition means for acquiring an image of a target object for extracting motion; Digital signal conversion means for converting an image input from the image acquisition means into a digital signal capable of digital processing; Digital signal processing means for processing the converted digital signal, analyzing the input image to recognize a corresponding object from previously learned information, and calculating coordinates for each part of the recognized object; Main control means for extracting the motion information of the target object from the input image, controlling a series of processes for applying to the corresponding character, and outputs the corresponding coordinate information of the object recognized from the digital signal processing means; Character control processing means for receiving the recognition information of the target object and the calculated coordinate information from the main control means to determine the movement of the corresponding character; And memory means for storing learning information about an object and calculation information for calculating motion tracking coordinates from an input image. Learning through video images for each object, and storing the learned information; Photographing a target object for extracting motion information and dividing an input image into a predetermined size so as to have connectivity between input image frames through a preprocessing process; Recognizing a target object based on the information learned from the image obtained through the process, Recognized object is a process of extracting the amount of movement between the frame and the frame for each object already pre-learned after adjusting the center coordinates, and calculating the movement coordinate for each object. Motion capture method. The process of claim 2, wherein the recognizing the target object and setting movement coordinates of the recognized target object include: Extracting a motion vector component of the object from a previous input frame; Retrieving the target object by performing motion estimation on the target object of the next frame from the extracted motion vector component; Recognizing and labeling an object when a target object exists at a corresponding coordinate estimated by the search result; If a target object does not exist in the coordinates whose motion is estimated as a result of the search, the target object is searched for in each direction from the corresponding position to find and label the target object. And if there is no previous input frame necessary to search for the target object through motion estimation, recognizing the target object on the coordinate set by learning, and calculating the reference coordinate in the calculation of the movement coordinate. Motion capture method using an image. A camera for acquiring an image of a gamer, a video codec for converting an image input from the camera into a signal capable of digital signal processing, and an image frame input from the video codec for analysis. Digital signal processor (DSP) for recognizing an object and calculating coordinate information of the recognized object, a memory for storing learning information of an object previously learned for recognizing a target object and calculation information for calculating coordinates And a main control microprocessor for controlling a series of processes to be applied to the corresponding target character from the detected operational coordinates, and a series of control programs for controlling the game program, and the operation coordinates of the target object input from the main control microprocessor. To control the movement of the target character in the game program using the information. Im processor is configured to include a motion capture system using an image, characterized in that applied to a virtual reality game device. The digital signal processing means according to claim 1 or 4, wherein the learned and stored information stored in the memory further includes a movement pattern information of a hand or a finger following a sign language, and content information of the pattern. Is compared to the movement pattern of the hand or finger stored in the memory from the input frame, and is configured to read and output semantic information according to the corresponding movement pattern, Character output means for outputting the meaning of sign language as text under the control of the main control means, and voice output means for outputting the meaning of sign language as voice under the control of the main control means; A motion capture system using an image, which is applied to a sign language system.