WO2021238804A1 - Mixed reality virtual preview photographing system - Google Patents

Abstract

The present invention relates to the technical field of motion capture. Disclosed is a mixed reality virtual preview photographing system, used for improving the efficiency of mixed reality virtual video photographing and reducing the production cost. The mixed reality virtual preview photographing system comprises a real photographing system, a virtual photographing system, an optical positioning and motion capture system, a central control system, and a display system; the real photographing system comprises a video camera and a video capture card; the virtual photographing system comprises a virtual camera, an inertia measurement apparatus, and a wireless transmission apparatus; the optical positioning and motion capture system comprises an optical capture processing server and a plurality of optical motion capture cameras, and the optical capture processing server is in communication connection with the plurality of optical motion capture cameras and the wireless transmission apparatus; the central control system comprises a central management server which is respectively in communication connection with the video capture card, the wireless transmission apparatus, and the optical capture processing server; and the display system comprises a display device which is in network connection with the central management server.

Description

Mixed reality virtual rehearsal shooting system Technical field

The invention relates to the technical field of motion capture, in particular to a mixed reality virtual rehearsal shooting system.

Background technique

Mixed reality technology is a further development of virtual reality technology. By presenting virtual scene information in the real scene, an information loop of interactive feedback is set up between the real world, the virtual world and the user to enhance the realism of the user experience.

In traditional film and television visual design, traditional virtual studios support relatively few cameras and lenses. Most of the lenses are fixed or non-movable. The docking rendering engine is single or old. The synthesis relationship between characters and visual design CG is mostly single. The front and back layers are superimposed, and there is no complicated mixed virtual and real interactive part. At the same time, in order not to wear the help, the position of the camera needs to be completely fixed at a fixed angle, which is difficult to temporarily adjust on-site. It is purely a visual illusion to improve the sense of integration and cannot respond to light.

In the prior art, starting from the time the director’s crew is obtained, the offline method is adopted, and the 3D 3D animator is used to produce the shots, and the production cycle is long. Therefore, the hybrid virtual and real shooting system lacks on-site shots and previews. As a result, the efficiency of computer synthesis of 3D video in the later stage is low.

Summary of the invention

The main purpose of the present invention is to solve the problem that the mixed virtual and real shooting cannot be used for on-site split-shot rehearsal, which leads to the low efficiency of post-computer production.

The first aspect of the present invention provides a mixed reality virtual rehearsal shooting system. The mixed reality virtual rehearsal shooting system includes a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system, and a display system that are communicatively connected; in,

The real shooting system includes a video camera and a video capture card, and the video camera and the video capture card are in communication connection;

The virtual shooting system includes a virtual camera, an inertial measurement device, and a wireless transmission device, the inertial measurement device is installed on the virtual camera, and the inertial measurement device is in communication connection with the wireless transmission device;

The optical positioning and motion capture system includes a light capture processing server and a plurality of optical motion capture cameras, and the light capture processing server is respectively communicatively connected with the plurality of optical motion capture cameras and the wireless transmission device;

The central control system includes a central management server, and the central management server is respectively communicatively connected with the video capture card, the wireless transmission device, and the optical capture processing server;

The display system includes a display device, and the display device is network connected to the central management server.

Optionally, the video camera is used to shoot the object to be captured in the actual shooting scene to obtain real-shot video data; the video capture card is used to send the real-shot video data to the central management server.

Optionally, the object to be captured wears a motion capture suit with a plurality of optical marking points affixed in advance, and the plurality of optical marking points are used to locate each joint position of the object to be captured.

Optionally, a preset number of optical marking points are installed on the video camera and the virtual camera respectively;

The inertial measurement device is used to collect the inertial navigation data corresponding to the virtual camera in moving shooting through a nine-axis inertial sensor;

The wireless transmission device is used to send the corresponding inertial navigation data to the optical capture processing server.

Optionally, the light capture processing server further includes a three-dimensional motion capture processing module and an optical inertial fusion module, the three-dimensional motion capture processing module is communicatively connected with the plurality of optical motion capture cameras, and the optical inertial fusion module is respectively connected with The central management server is in a communication connection with the wireless transmission device;

The multiple optical motion capture cameras are used for positioning and shooting each optical marking point to obtain respective corresponding two-dimensional image data;

The three-dimensional motion capture processing module is configured to obtain the corresponding three-dimensional coordinate data and the motion posture information of the object to be captured according to the corresponding two-dimensional image data;

The optical inertial fusion module is used to perform coordinate system calibration and posture fusion processing in sequence according to the three-dimensional coordinate data corresponding to each optical mark point on the virtual camera and the corresponding inertial navigation data to obtain camera posture data, and then The respective corresponding three-dimensional coordinate data, the camera posture data and the motion posture information are sent to the central management server.

Optionally, the optical capture processing server further includes an inertial navigation setting module, which is communicatively connected with the optical inertial fusion module and configured to use a preset link port number and a preset rigid body name to connect the nine-axis The inertial sensor and the preset number of optical marking points are installed on the virtual camera according to a preset positional relationship.

Optionally, the central management server includes a virtual rehearsal shooting synthesis module and a rendering synthesis module, the virtual rehearsal shooting synthesis module is respectively communicatively connected with the video capture card and the optical inertial fusion module, and the rendering synthesis module is respectively Communicatively connected with the virtual rehearsal shooting synthesis module and the wireless transmission device;

The virtual rehearsal shooting synthesis module is used to perform real-time keying of the real-shot video data, so as to generate an avatar in a virtual scene, and adjust the angle of the virtual scene in real time;

The rendering synthesis module is configured to convert the keying image information and the adjusted virtual scene into a three-dimensional virtual reality mixed video stream according to the camera posture data, the respective corresponding three-dimensional coordinate data, and the motion posture information, and The three-dimensional virtual reality mixed video stream is sent to the virtual camera and the display device respectively.

Optionally, the display device is configured to receive and synchronously display the 3D virtual reality mixed video stream, so that the target person previews the 3D virtual reality mixed video stream during the displacement of the video camera and the virtual camera , And adjust the shooting action and shooting angle of the object to be captured in real time.

Optionally, the optical positioning and motion capture system further includes a calibration device, and the calibration device is used to calibrate the positions of the plurality of optical motion capture cameras in the actual shooting scene through a calibration rod.

Optionally, the mixed reality virtual rehearsal shooting system further includes a camera setting system, which is communicatively connected with the central control system, and is used to obtain the site parameters in the actual shooting scene, and according to the site The parameters determine the number of cameras corresponding to the optical motion capture camera and the corresponding camera installation positions, so that each optical mark point is captured by any three optical motion capture cameras, and the site parameters include large space site length information and width information.

In the technical solution provided by the present invention, the mixed reality virtual rehearsal shooting system includes a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system, and a display system that are communicatively connected; wherein, the real shooting system includes A video camera and a video capture card, the video camera and the video capture card are in communication connection; the virtual shooting system includes a virtual camera, an inertial measurement device and a wireless transmission device, the inertial measurement device is installed on the virtual camera, The inertial measurement device is in communication connection with the wireless transmission device; the optical positioning and motion capture system includes a light capture processing server and a plurality of optical motion capture cameras, and the light capture processing server is respectively connected to the plurality of optical motion capture The camera is in communication connection with the wireless transmission device; the central control system includes a central management server, and the central management server is respectively in communication connection with the video capture card, the wireless transmission device, and the light capture processing server; The display system includes a display device, and the display device is network connected to the central management server. In the embodiment of the present invention, the real shooting system, the virtual shooting system, the optical positioning and motion capture system, the central control system, and the display system realize motion track recording, data export and fusion, redundant target shielding, and real-time imaging preview. Compared with the traditional green screen keying, the present invention obtains the three-dimensional motion trajectory information of the object to be captured in real time, realizes the superimposition of the scene and the character in the three-dimensional space, and has the effect of synthesizing the three-dimensional depth information; at the same time, through virtual reality interaction, real-time preview of the captured image The visual effect of the moving mirror of the film improves the efficiency of mixed reality virtual video shooting, and reduces the production cost and production cycle.

Description of the drawings

FIG. 1 is a schematic structural diagram of a mixed reality virtual rehearsal shooting system in an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an optical positioning and motion capture system in an embodiment of the present invention;

Figure 3 is a schematic diagram of a structure of a central management server in an embodiment of the present invention;

4 is another schematic diagram of the structure of the mixed reality virtual rehearsal shooting system in the embodiment of the present invention;

FIG. 5 is a schematic diagram of an application scenario of the mixed reality virtual rehearsal shooting system in an embodiment of the present invention.

Detailed ways

The embodiment of the present invention provides a mixed reality virtual rehearsal shooting system, through virtual reality interaction, real-time preview of the lens moving visual effect of a filmed film, improving the efficiency of mixed reality virtual video shooting, and reducing production costs and production cycles.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, without having to use To describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in a sequence other than the content illustrated or described herein.

For ease of understanding, the following describes the embodiments of the present invention, please refer to FIG. 1. In the embodiment of the present invention, the mixed reality virtual rehearsal shooting system includes a real shooting system 1, a virtual shooting system 2, an optical positioning and motion capture system 3 connected in communication. , Central control system 4 and display system 5; among them,

The real shooting system 1 includes a video camera 11 and a video capture card 12, and the video camera 11 and the video capture card 12 are in communication connection;

The virtual shooting system 2 includes a virtual camera 21, an inertial measurement device 22, and a wireless transmission device 23. The inertial measurement device 22 is installed on the virtual camera 21, and the inertial measurement device 22 is in communication connection with the wireless transmission device 23;

The optical positioning and motion capture system 3 includes a light capturing processing server 31 and a plurality of optical motion capturing cameras 32, and the light capturing processing server 31 is respectively communicatively connected with a plurality of optical motion capturing cameras 32 and a wireless transmission device 23;

The central control system 4 includes a central management server 41, and the central management server 41 is in communication connection with the video capture card 12, the wireless transmission device 23, and the light capture processing server 31, respectively;

The display system 5 includes a display device 51, and the display device 51 is connected to the central management server 41 in a network.

Specifically, the real shooting system 1 is used to receive preset shooting instructions, and based on the preset shooting instructions to control the shooting of the video camera 11, the video camera 11 is used to perform video shooting of the object to be captured in the actual shooting scene, Obtain real-time video data. Further, the video capture card 12 is used to upload the real shot video data to the central management server 41, so that the central management server 41 performs data storage and data processing on the real shot video data. Among them, the object to be captured is a moving object, including people and props, and the real shot video data is used to indicate the moving picture information of the object to be captured in the actual shooting scene. Optionally, the actual shooting scene includes a green screen and a truss. The truss is used to arrange multiple optical motion capture cameras 32. The background of the actual shooting scene may also be a blue screen, which is not specifically limited here. It should be noted that the video camera 11 is also used to support the lens translation, up and down movement, zoom in and out of the fixed PTZ camera, and to support the movement of the camera with a large rocker arm in a large range, for example, the video camera 11 is set to 10 Used in actual shooting scenes up to 2000 square meters.

Specifically, the virtual camera 21 in the virtual shooting system 2 is a handheld camera with six degrees of freedom, and the virtual camera 21 is used to move and shoot a preset virtual scene corresponding to the actual shooting scene. The six degrees of freedom means that the object has a three-dimensional space. Six degrees of freedom, that is, the degrees of freedom of movement along the directions of the three rectangular coordinate axes of x, y, and z, and the degrees of freedom of rotation around the three coordinate axes of x, y, and z. Therefore, the virtual camera 21 can be used to adjust the shooting angle of a preset virtual scene, can also be used to adjust the focal length and aperture value, and to control the start and end time of shooting for the preset virtual scene. For example, when the start button in the virtual camera 21 is activated, the virtual camera 21 is used to receive a start recording instruction and start shooting a preset virtual scene. When the pause button or stop button of the virtual camera 21 is activated, the virtual camera 21 21 is used to receive a pause recording instruction or a stop recording instruction, and stop shooting a preset virtual scene. And it is used to synchronously display the 3D virtual reality mixed video stream sent by the central management server 41. It should be noted that during the movement of the virtual camera 21, the object to be captured in the preset virtual scene is photographed, and there are changes in the shooting angle and moving speed. Therefore, the inertial measurement device 22 is used to capture the virtual camera 21. The movement posture information, the wireless transmission device 23, is used to upload the collected movement posture information to the light capture processing server 31 in the optical positioning and motion capture system 3. The wireless transmission device 23 can be wireless Bluetooth and both wireless The Bluetooth image transmission function can also be wireless WIFI.

Specifically, the multiple optical motion capture cameras 32 in the optical positioning and motion capture system 3 are used to identify the optical marking points bound to different parts of the object to be captured, and the light capturing processing server 31 is used to obtain the actual position of each optical marking point. The position and orientation in the shooting scene are further determined to determine the movement trajectory of the object to be captured in the actual shooting scene, and synchronously imported into the central management server 41 in real time. Optionally, the object to be captured wears a motion capture suit with multiple optical marking points affixed in advance. The multiple optical marking points are used to locate the joint positions of the object to be captured. The optical marking points include reflective marking points and active markings. Reflective marking points can use reflective balls. Active marking points are suitable for situations where ambient lighting conditions make it difficult to locate and track the reflective marking points, so that the optical positioning and motion capture system 3 can be applied both indoors and outdoors. Further, the video camera 11 and the virtual camera 21 are respectively preset with a preset number of optical marking points, and the preset number of optical marking points are combined with a plurality of optical motion capture cameras 32 to perform the operation on the video camera 11 and the virtual camera 21. Positioning space location information, where the preset number is a positive integer. For example, 3, 4, or 5, and the details are not limited here.

For example, different reflective marking points (reflective balls) are arranged on the head, hands, feet, and back waist of the object to be captured, and multiple optical motion capture cameras 32 are used to track the reflective marking points on the object to be captured. Realize the precise positioning of the spatial position and orientation of the reflective marking point, so as to obtain the extremity position of the object to be captured. The light-capturing processing server 31 is configured to determine the motion posture information of the object to be captured according to the position of the extremities of the object to be captured.

Specifically, the central management server 41 in the central control system 4 is first used to perform keying processing on the real shot video data captured by the film and television camera 11 to obtain keyed image information, where the keyed image information includes character model information, Secondly, it is used to synthesize the keying image information into a preset virtual scene (virtual scene corresponding to the actual shooting scene) through 3D synthesis software or a preset engine, or to obtain the virtual character corresponding to the character model information, and merge the corresponding virtual character Go to the corresponding preset virtual scene, and then drive the character model information or virtual character movement through forward kinematics and motion posture information. For example, when a motion capture actor (object to be captured) wearing a green motion capture suit performs an action performance in a green-screen shooting space, the light capture processing server 31 and multiple optical motion capture cameras 32 are used to obtain all the motion capture actor's Sports posture information. The central management server 41 is used to perform overall picture calculations on the real shot video data shot by the real shooting system 1, to obtain the keying image information containing the motion capture actors and the virtual characters corresponding to the keying image information, and use three-dimensional synthesis software or The preset engine synthesizes the corresponding virtual characters into the preset virtual scenes that are actually needed, and uses the motion posture information to drive the preset virtual characters. The virtual characters are virtual objects that dance with music to obtain 3D virtual reality mixed video flow. Further, the central management server 41 is configured to send the three-dimensional virtual reality mixed video stream to the virtual camera 21 and the display device 51. Optionally, the central management server 41 can be used to superimpose and synthesize a preset virtual scene and a virtual character, and can also be used to perform three-dimensional depth of the real character and the preset virtual scene, and the preset real scene and the virtual character. The synthesis process is not specifically limited here. It should be noted that the central control system 4 also includes input devices and output devices, such as a mouse, a keyboard, and a display.

Specifically, the display system 5 includes a display device 51 for receiving and synchronously displaying the three-dimensional virtual reality mixed video stream sent by the central management server 41, so that the target person can adjust the shooting action and shooting angle of the object to be captured in real time, and display it on the video camera 11 During the displacement process of the virtual camera 21, the visual effect of the lens movement of the filmed film is previewed, and the target personnel include the director and the photographer. Therefore, the display device 51 is used to provide a real-time scheduling preview of the lens frame composition, lens motion, and composite effect, so that the target person can adjust and modify the creativity and frame composition in time. In this way, the problems of high production cost and long production cycle of traditional film and television production in the early and late stages of production are avoided, and the visualized shooting effect is provided in real time through the virtual rehearsal shooting method.

In the embodiment of the present invention, a hybrid display virtual rehearsal shooting system is provided, which realizes motion track recording, data export and fusion, and redundant targets through a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system, and a display system. (Objects other than the object to be captured) Real-time preview of shielding and imaging. Compared with the traditional green screen keying, the present invention obtains the three-dimensional motion trajectory information of the object to be captured in real time, realizes the superimposition of the scene and the character in the three-dimensional space, and has the effect of synthesizing the three-dimensional depth information; at the same time, through virtual reality interaction, real-time preview of the captured image The visual effect of the moving mirror of the film improves the efficiency of mixed reality virtual video shooting, and reduces the production cost and production cycle.

1 and 2, the optical capture processing server 31 also includes a three-dimensional motion capture processing module 311 and an optical inertial fusion module 312. The three-dimensional motion capture processing module 311 is communicatively connected with a plurality of optical motion capture cameras 32, and the optical inertial fusion module 312 Respectively communicate with the central management server 41 and the wireless transmission device 23;

A plurality of optical motion capture cameras 32 are used for positioning and shooting each optical marking point to obtain respective corresponding two-dimensional image data;

The three-dimensional motion capture processing module 311 is configured to obtain the corresponding three-dimensional coordinate data and the motion posture information of the object to be captured according to the corresponding two-dimensional image data;

The optical inertial fusion module 312 is used to perform coordinate system calibration and posture fusion processing in sequence according to the motion posture information corresponding to each optical mark point on the virtual camera 21 and the corresponding inertial navigation data, to obtain camera posture data, and to compare the corresponding The three-dimensional coordinate data, camera posture data, and motion posture information are sent to the central management server 41.

The multiple optical motion capture cameras 32 and the light capture processing server 31 may be connected by wire or wirelessly, for example, by a POE switch. Specifically, multiple optical motion capture cameras 32 are used to collect the two-dimensional image information corresponding to each of the video camera 11, the virtual camera 21, and the multiple optical marker points, that is, the video camera 11, the virtual camera 21, and the object to be captured Each optical marking point is captured by at least two or more optical motion capture cameras 32 at the same time; the three-dimensional motion capture processing module 311 is used to convert the corresponding two-dimensional image data into each corresponding three-dimensional coordinate data, that is, spatial position information, And the movement posture information of the object to be captured, wherein the respective corresponding three-dimensional coordinate data is used to indicate the position and orientation of each optical mark point in the world coordinate system, so as to realize the positioning and tracking of the moving object; and the optical inertial fusion module 312 is used for Receive the inertial navigation data corresponding to the virtual camera 21, and perform coordinate system calibration and posture fusion calculation based on the three-digit coordinate data corresponding to each optical mark point on the virtual camera 21 and the corresponding inertial navigation data to obtain the camera posture data and correspond to each The three-dimensional coordinate data, camera posture data, and motion posture information of the camera are sent to the central management server 41. Optionally, the optical positioning and motion capture system 3 further includes a calibration device 33, which is used to calibrate the positions of a plurality of optical motion capture cameras 32 in the actual shooting scene through a calibration rod. For example, the shape of the calibration rod may be a T-shaped structure, and the dimensions of the calibration rod are 418 mm in the length of the horizontal rod and 578 mm in the length of the vertical rod.

Further, the inertial measurement device 22 is used to collect the inertial navigation data corresponding to each of the virtual camera 21 and the multiple optical marker points during the moving shooting through the nine-axis inertial sensor, and send the corresponding inertial navigation data to the wireless transmission device 23 , And the inertial measurement device 22 is an inertial measurement unit (IMU); the wireless transmission device 23 is used to send respective corresponding inertial navigation data to the optical inertial fusion module 312. Optionally, the optical capture processing server 31 further includes an inertial navigation setting module 313. The inertial navigation setting module 313 is communicatively connected with the optical inertial fusion module 312, and is used to connect the nine-axis inertial sensor with the preset A set number of optical marking points are installed on the virtual camera 21 according to a preset positional relationship. Among them, the nine-axis inertial sensor includes a three-axis angular velocity sensor (gyro), a three-axis acceleration sensor and a three-axis magnetic induction sensor. The nine-axis inertial sensor can ensure high-precision calculation of the rotation angle of the rigid body, and the error range is within 0.05 degrees, so that the synthesized 3D virtual reality mixed video stream can be more stable and can be interacted in real time with low latency. For example, using 200 The refresh rate of Hertz angle data is used to update the picture of the dimensional virtual reality hybrid video stream. It is understandable that at least 3 reflective balls can be used for the preset number of optical marking points. The reflective ball is a rigid body and also a motion tracker, and its motion trajectory represents a change in spatial coordinates. In order to obtain a more accurate pose, a nine-axis inertial sensor (inertial measurement device 22) is generally used together, and the spatial coordinates of the rigid body and the nine-axis inertial sensor are unified to finally obtain the precise pose data of the rigid body. The purpose of the calibration of the rigid body and the nine-axis inertial sensor is to unify the space coordinate system. The motion posture information of the rigid body is the position and posture in the custom world coordinate system. The posture of the nine-axis inertial sensor (corresponding to the inertial navigation data) is relative to the hardware The posture change when the body is started, the coordinate system alignment processing of the rigid body and the nine-axis inertial sensor, and the posture fusion processing can all improve the calculation accuracy of the rigid body’s pose.

It should be noted that the reflective ball (optical marking point) and the nine-axis inertial sensor are bound flexibly, and the preset rigid body name and the preset link port number have a one-to-one mapping relationship. Therefore, the inertial navigation setting module 313 is used to pass the Set the rigid body name and the preset link port number to update the configuration and clear the existing configuration, and the binding and unbinding of the reflector (optical mark) and the nine-axis inertial sensor can be realized. At the same time, the nine-axis inertial sensor reduces the circuit board and overall space, and the corresponding hardware is small. It is suitable for portable installation on moving objects that accurately capture rotation data, and improves the calculation accuracy of a single sensor in locating the spatial position and movement direction. It is understandable that the inertial navigation setting module 313 is not only used to determine the motion posture data of the virtual camera 21, but also can be used to determine the motion posture data of the object to be captured, determine according to the actual shooting scene, and set the nine-axis inertial sensor and optical markers. The relationship between points, so as to achieve high-precision motion detection.

1 and 3, optionally, the central management server 41 includes a virtual rehearsal shooting synthesis module 411 and a rendering synthesis module 412. The virtual rehearsal shooting synthesis module 411 is respectively communicatively connected with the video capture card 12 and the optical inertial fusion module 312, The rendering synthesis module 412 is in communication connection with the virtual rehearsal shooting synthesis module 411 and the wireless transmission device 23 respectively;

The virtual rehearsal shooting synthesis module 411 is used to perform real-time keying of the real shot video data, so as to generate a virtual image in the virtual scene, and adjust the angle of the virtual scene in real time;

The rendering and synthesis module 412 is used to convert the keying image information and the adjusted virtual scene into a 3D virtual reality mixed video stream according to the camera pose data, the respective corresponding three-dimensional coordinate data and motion posture information, and the 3D virtual reality mixed video stream They are sent to the virtual camera 21 and the display device 51 respectively.

The rendering synthesis module 412 receives the keying image information sent by the virtual rehearsal shooting synthesis module 411, where the keying image information is image information containing the object to be captured, and the rendering synthesis module 412 is used to use Unreal Engine UE4 for 3D animation character settings and sound Simulation and lighting rendering. Specifically, the virtual rehearsal shooting synthesis module 411 is used to set the virtual character corresponding to the keying image information, and to adjust the angle of the virtual scene in real time; the rendering synthesis module 412 is used to use the Unreal Engine UE4 according to the camera pose data and respective corresponding three-dimensional coordinates The data and the motion posture information are synchronized to set the motion posture of the virtual character; at the same time, the preset voice information is added to the preset virtual scene to obtain a three-dimensional virtual reality mixed video stream. Further, the wireless transmission device 23 is used to send the three-dimensional virtual reality mixed video stream to the virtual camera 21 and the display device 51 respectively, so that the target person can preview the shooting effect in real time. Among them, the display device 51 may use a large-screen projection method to display the split lens, so as to achieve the visualization of the split lens.

Optionally, the central management server 41 is also used to store the camera posture data, the respective corresponding three-dimensional coordinate data and motion posture information, and the three-dimensional virtual reality mixed video stream to facilitate post-production and synthesis.

Optionally, the central management server 41, before performing video synthesis, is also used to align the adjusted virtual scene with the actual shooting scene in the three-dimensional space coordinate system according to the camera pose data and the respective corresponding three-dimensional coordinate data, so that The output 3D virtual reality mixed video stream has high frame rate, stable, no-delay spatial positioning and virtual-real synchronization effects.

It is understandable that the virtual rehearsal shooting synthesis module 411 is used to instruct the color keying process, that is, to define the transparency according to the specific color value or brightness value of the image in the real shot video data. When a certain value is keyed out, all have Pixels with similar colors or brightness values will all become transparent. Further, the virtual preview shooting synthesis module 411 is used to obtain keyed image information from the real shot video data.

Please refer to Figure 4. Optionally, the mixed reality virtual rehearsal shooting system also includes a camera setting system 6. The camera setting system 6 is in communication connection with the central control system 4 for obtaining site parameters in the actual shooting scene and determining them according to the site parameters The number of cameras corresponding to the optical motion capture cameras 32 and the corresponding camera installation positions, so that each optical mark point is captured by any three optical motion capture cameras 32, and the field parameters include large space field length information and width information.

In the mixed reality virtual rehearsal shooting system, it is necessary to locate and track the motion trajectory of multiple optical marking points. In order to improve the accuracy of positioning, each optical marking point needs to be captured by any three optical motion capture cameras 32 to determine each optical point. The spatial location information of the marker points, and the excessive number of camera configurations will lead to the problem of excessive construction costs. Therefore, the camera setting system 6 is used to lay out the optical motion capture camera 32. Please refer to Figures 4 and 5. The actual shooting scene is an open field with a height of 3.5 meters to 7 meters, and is equipped with professional film and television lights, trusses, and green boxes. Or blue box, where the truss includes a single-layer truss, a two-layer truss, and a multi-layer truss. The site parameters of the actual shooting scene include the large space site length information of 20 meters and width information of 10 meters. Further, the camera setting system 6 is used to calculate the cost value of the camera configuration in the actual shooting scene according to the site parameters of the actual shooting scene, and determine the number of cameras corresponding to the optical motion capture camera 32 and the corresponding camera installation position through the camera configuration cost value . The multiple optical motion capture cameras 32 are arranged at different positions of the truss in the actual shooting scene, and the visual range of each optical motion capture camera 32 covers the entire actual shooting scene. The video camera 11 and the virtual camera 21 are used to shoot the object to be captured.

It is understandable that the hybrid display virtual rehearsal shooting system integrates multiple functions of real-time rehearsal, virtual studio and motion capture. On the basis of traditional keying technology, it uses spatial positioning motion capture technology to synchronize the film and television cameras in the space 11 and virtual cameras. 21 and the position of all virtual objects (for example, props), enabling real-time interaction and synthesis effect preview in Unreal Engine. In the smooth rendering of the picture and the realistic three-dimensional fusion picture, the effect of combining virtual and real is realized, which improves the storytelling and visual expressiveness of film and television shooting techniques. Not only can the target person (director) have a clear need for interaction on the spot, obtain the preset composition angle according to the demand, preview the shooting effect in real time, and determine the effects and elements of each lens in real time through the interactive visualization lens. The shooting process is in place in one step, and later The direction of the production has been determined during the shooting stage. At the same time, for real-time interactions between real actors, virtual actors, and virtual characters, the interaction between characters and CG is no longer simply superimposed back and forth, and the positions of objects all have 3D depth information. For example, real people interspersed into the virtual ancient city, real people walk around the CG car, real people and virtual characters hug, fight with complex and natural performance effects, the position of the camera is directly recorded on the set, and seamlessly imported into Maya 3D software for rendering and synthesis. Reduce the cost of repeated communication and modification from the early stage to the post-production stage. Moreover, under any shooting conditions, the virtual character scene can be accurately tracked in the real environment, the 11-position shift lens of the video camera is zoomed in and out, and the combined virtual and real scenes can be fully presented in real time. So as to realize the animation production of film, television, animation and games, as well as the virtual reality preview shooting of studios, studios and large exhibition halls.

In the embodiment of the present invention, a hybrid display virtual rehearsal shooting system is provided, which realizes motion track recording, data export and fusion, and redundant targets through a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system, and a display system. Real-time preview of shielding and imaging. Compared with the traditional green-screen keying, the real-time acquisition of the three-dimensional motion trajectory information of the object to be captured, the superimposition of the scene and the character in the three-dimensional space, and the synthesis and processing effect of the three-dimensional depth information; at the same time, through the virtual reality interaction, the real-time preview of the film has been taken Mirroring visual effects, improving the efficiency of mixed reality virtual video shooting, and reducing production costs and production cycles.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that: The technical solutions recorded in the embodiments are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A mixed reality virtual rehearsal shooting system, characterized in that the mixed reality virtual rehearsal shooting system includes a real shooting system, a virtual shooting system, an optical positioning and motion capture system, a central control system, and a display system that are connected in communication;

   The real shooting system includes a video camera and a video capture card, and the video camera and the video capture card are in communication connection;

   The virtual shooting system includes a virtual camera, an inertial measurement device, and a wireless transmission device, the inertial measurement device is installed on the virtual camera, and the inertial measurement device is in communication connection with the wireless transmission device;

   The optical positioning and motion capture system includes a light capture processing server and a plurality of optical motion capture cameras, and the light capture processing server is respectively communicatively connected with the plurality of optical motion capture cameras and the wireless transmission device;

   The central control system includes a central management server, and the central management server is respectively communicatively connected with the video capture card, the wireless transmission device, and the optical capture processing server;

   The display system includes a display device, and the display device is network connected to the central management server.
2. The mixed reality virtual rehearsal shooting system according to claim 1, wherein:

   The video camera is used to shoot the object to be captured in the actual shooting scene to obtain real-shot video data; the video capture card is used to send the real-shot video data to the central management server.
3. The mixed reality virtual rehearsal shooting system according to claim 2, wherein:

   The object to be captured wears a motion capture suit with a plurality of optical marking points affixed in advance, and the plurality of optical marking points are used to locate each joint position of the object to be captured.
4. The mixed reality virtual rehearsal shooting system according to claim 2, wherein:

   The video camera and the virtual camera are respectively installed with a preset number of optical marking points;

   The inertial measurement device is used to collect the inertial navigation data corresponding to the virtual camera in moving shooting through a nine-axis inertial sensor;

   The wireless transmission device is used to send the corresponding inertial navigation data to the optical capture processing server.
5. The mixed reality virtual rehearsal shooting system according to claim 4, wherein:

   The optical capture processing server further includes a three-dimensional motion capture processing module and an optical inertial fusion module, the three-dimensional motion capture processing module is communicatively connected with the plurality of optical motion capture cameras, and the optical inertial fusion module is respectively connected to the central management The server is in communication connection with the wireless transmission device;

   The multiple optical motion capture cameras are used for positioning and shooting each optical marking point to obtain respective corresponding two-dimensional image data;

   The three-dimensional motion capture processing module is configured to obtain the corresponding three-dimensional coordinate data and the motion posture information of the object to be captured according to the corresponding two-dimensional image data;

   The optical inertial fusion module is used to perform coordinate system calibration and posture fusion processing in sequence according to the three-dimensional coordinate data corresponding to each optical mark point on the virtual camera and the corresponding inertial navigation data to obtain camera posture data, and then The respective corresponding three-dimensional coordinate data, the camera posture data and the motion posture information are sent to the central management server.
6. The mixed reality virtual rehearsal shooting system according to claim 5, wherein:

   The optical capture processing server also includes an inertial navigation setting module, which is communicatively connected to the optical inertial fusion module, and is configured to use a preset link port number and a preset rigid body name to connect the nine-axis inertial sensor to the The preset number of optical marking points are installed on the virtual camera according to a preset position relationship.
7. The mixed reality virtual rehearsal shooting system according to claim 5, wherein:

   The central management server includes a virtual rehearsal shooting synthesis module and a rendering synthesis module, the virtual rehearsal shooting synthesis module is respectively connected to the video capture card and the optical inertial fusion module in communication, and the rendering synthesis module is respectively connected to the virtual The rehearsal shooting synthesis module is in communication connection with the wireless transmission device;

   The virtual rehearsal shooting synthesis module is used to perform real-time keying of the real-shot video data, so as to generate an avatar in a virtual scene, and adjust the angle of the virtual scene in real time;

   The rendering synthesis module is configured to convert the keying image information and the adjusted virtual scene into a three-dimensional virtual reality mixed video stream according to the camera posture data, the respective corresponding three-dimensional coordinate data, and the motion posture information, and The three-dimensional virtual reality mixed video stream is sent to the virtual camera and the display device respectively.
8. The mixed reality virtual rehearsal shooting system according to claim 7, wherein:

   The display device is configured to receive and synchronously display the 3D virtual reality mixed video stream, so that the target person previews the 3D virtual reality mixed video stream during the displacement process of the film and television camera and the virtual camera, and responds to all The shooting action and shooting angle of the object to be captured are adjusted in real time.
9. The mixed reality virtual rehearsal shooting system according to claim 2, wherein:

   The optical positioning and motion capture system further includes a calibration device, which is used to calibrate the positions of the plurality of optical motion capture cameras in the actual shooting scene through a calibration rod.
10. The mixed reality virtual rehearsal shooting system according to any one of claims 1-9, wherein:

    The mixed reality virtual rehearsal shooting system further includes a camera setting system, which is communicatively connected with the central control system, and is used to obtain the site parameters in the actual shooting scene and determine the site parameters according to the site parameters. The number of cameras corresponding to the optical motion capture camera and the corresponding camera installation positions, so that each optical mark point is captured by any three optical motion capture cameras, and the site parameters include large space site length information and width information.

Images