CN115914606A - Video processing method, system, readable storage medium and equipment terminal - Google Patents

Abstract

The invention is suitable for the technical field of video processing, and provides a video processing method, a system, a readable storage medium and a device terminal, wherein the method comprises the steps of separating an audio signal and a video signal in an acquired video to be converted to obtain audio data and video data; splitting the video data frame by frame to obtain an image to be converted of each frame; respectively carrying out depth processing on the image to be converted of each frame, and carrying out image processing on the image to be converted according to the depth processing result to generate a corresponding virtual right image; respectively carrying out image processing on the image to be converted of each frame and the corresponding virtual right image together to generate a three-dimensional image; combining the three-dimensional images of all the frames to obtain three-dimensional video data; and merging the separated audio data and the generated three-dimensional video data to obtain the three-dimensional video. The video processing method provided by the invention solves the problems of high cost and low efficiency of the existing 3d video production.

Description

A video processing method, system, readable storage medium and device terminal

technical field

The invention belongs to the technical field of video processing, and in particular relates to a video processing method, system, readable storage medium and equipment terminal.

Background technique

At present, with the development of science and technology and the improvement of people's living standards, more and more users are willing to experience new technological products, and naked-eye 3D technology and corresponding products are one of the technologies that users are willing to experience. .

Among them, naked-eye 3D technology has been developed for a long time since 2013, but it is less popular, and its development is more reflected in the development of hardware, while the development of ordinary 2D video to 3D video is extremely slow. There is no software on the Internet that can automatically generate the virtual perspective of the 3D right picture, and it can only be processed manually. As a result, the naked-eye 3D film sources that are currently popular on the market are mainly manually processed by manufacturers frame by frame. For example, a 2-minute video usually has About 4,600 pictures need to be manually cut out, which makes the labor cost extremely high. At the same time, the efficiency of naked-eye 3D video production is also extremely low, which also leads to the lack of naked-eye 3D videos available for playback on the market.

Contents of the invention

The purpose of the embodiments of the present invention is to provide a video processing method, aiming at solving the problems of high cost and low efficiency of existing 3D video production.

The embodiment of the present invention is achieved in this way, a video processing method, the method comprising:

Separating the acquired audio signal and video signal in the video to be converted to obtain audio data and video data;

Split the video data frame by frame to obtain the image to be converted for each frame;

Perform depth processing on the image to be converted for each frame, and perform image processing on the image to be converted according to the result of the depth processing to generate a corresponding virtual right image, the virtual right image is a virtual viewpoint image after the image to be converted has been processed by depth conversion ;

Carrying out image processing on the image to be converted and the corresponding virtual right image of each frame respectively to generate a three-dimensional image, the three-dimensional image is an image in which the image to be converted and the virtual right image are respectively located on the left and right sides;

Merging the three-dimensional images of all frames to obtain three-dimensional video data;

Combining the separated audio data and the generated 3D video data to obtain 3D video.

Furthermore, the steps of performing depth processing on the image to be converted in each frame respectively, and performing image processing on the image to be converted according to the result of the depth processing to generate a corresponding virtual right image include:

Depth processing is performed on the image to be converted for each frame to obtain a corresponding depth map;

Dynamically convert the depth map of each frame into the corresponding mask;

Separate the foreground and background from the corresponding image to be converted according to the mask of each frame;

The extracted foreground of each frame is rotated, stretched and left-shifted respectively, and the transformed result is filled into the corresponding background;

The images filled in the background of each frame are respectively image restored to obtain the corresponding virtual right image.

Further, the step of performing depth processing on each frame of the image to be converted to obtain a corresponding depth map includes:

The image to be converted of each frame is converted into a matrix containing information of three primary colors respectively, and the information of three primary colors includes three color channel information of red, green and blue;

Input the matrix of the three primary color information included in each frame into the depth processing algorithm for algorithm processing to obtain the corresponding depth map.

Further, the step of dynamically converting the depth map of each frame into a corresponding mask respectively includes:

Improve and adjust the contrast ratio of the depth map of each frame;

Convert the depth map of each frame into a grayscale image respectively;

Binarize the depth image converted to grayscale image for each frame to obtain the corresponding mask.

Furthermore, the step of performing image restoration on the images filled into the background of each frame to obtain the corresponding virtual right image includes:

Draw a mask based on the pixel positions of the image that is filled into the background each frame;

Fill the holes outside the mask to get the corresponding virtual right image.

Furthermore, the step of performing image processing on the image to be converted of each frame together with the corresponding virtual right image to generate a three-dimensional image includes:

Convert the image to be converted and the corresponding virtual right image of each frame into corresponding two sets of arrays;

Perform matrix broadcasting on the two sets of arrays in each frame, and merge the arrays;

Transform the merged arrays in each frame into corresponding 3D images respectively.

Furthermore, after the step of splitting the video data frame by frame to obtain the image to be converted for each frame, it also includes:

Add text subtitles to each frame of the image to be converted.

The purpose of another embodiment of the present invention is to provide a video processing system, the system includes:

A separation module, configured to separate the acquired audio signal and video signal in the video to be converted to obtain audio data and video data;

A splitting module is used to split the video data frame by frame to obtain an image to be converted for each frame;

The first image processing module is used to perform in-depth processing on the image to be converted in each frame, and perform image processing on the image to be converted according to the result of the in-depth processing to generate a corresponding virtual right image, the virtual right image is the image to be converted through image processing The virtual viewpoint image after depth conversion processing;

The second image processing module is used to respectively perform image processing on the image to be converted and the corresponding virtual right image of each frame to generate a three-dimensional image, and the three-dimensional image is the image to be converted and the virtual right image respectively located on the left and right sides image;

The first combining module is used to combine the three-dimensional images of all frames to obtain three-dimensional video data;

The second merging module is used for merging the separated audio data and the generated 3D video data to obtain 3D video.

The purpose of another embodiment of the present invention is to provide a readable storage medium, which stores a program, and when the program is executed by a processor, the above-mentioned video processing method is realized.

The purpose of another embodiment of the present invention is to provide a device terminal, including a memory, a processor, and a program stored in the memory and operable on the processor. When the processor executes the program, the above-mentioned video processing method.

In the video processing method provided by the embodiment of the present invention, the original video to be converted that needs to be made into a naked-eye 3D video is first separated from the audio and video, so that the audio data and video data are first obtained, and then the video data is processed frame by frame. In order to obtain the image to be converted for each frame, and then use the visual processing technology to convert the image to be converted into a virtual right image of a specific 3D virtual perspective of the right image, and splice the image to be converted with the virtual right image to generate a three-dimensional image, and then pass Merge the 3D images of all frames to obtain 3D video data, and incorporate the separated audio data into the 3D video data, so that the final 3D video can be generated, so that the process of converting the original image into a naked-eye 3D image can be automated, and the automation will The original video is converted into the corresponding naked-eye 3D video, which greatly reduces the high labor cost of naked-eye 3D video production, improves the production of naked-eye 3D video, is conducive to the popularization of naked-eye 3D, and solves the high cost and efficiency of existing 3D video production. low problem.

Description of drawings

Fig. 1 is a flowchart of a video processing method provided by an embodiment of the present invention;

Fig. 2 is another flow chart of the video processing method provided by the embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a video processing system provided by an embodiment of the present invention;

Fig. 4 is a schematic structural diagram of a device terminal provided by an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

In the present invention, unless otherwise clearly specified and limited, terms such as "installation", "connection", "connection" and "fixation" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection , or integrally connected; it may be mechanically connected or electrically connected; it may be directly connected or indirectly connected through an intermediary, and it may be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Embodiment one

Please refer to FIG. 1, which is a schematic flowchart of the video processing method provided by the first embodiment of the present invention. For the convenience of description, only the parts related to the embodiment of the present invention are shown. The method includes:

Step S10, separating the acquired audio signal and video signal in the video to be converted to obtain audio data and video data;

Wherein, in one embodiment of the present invention, the video processing method is used to process and convert existing common 2D video into 3D video that can be displayed in 3D. Specifically, the video processing method can be applied to a device terminal, where the device terminal can be a terminal for playing video sources such as a smart terminal, a smart tablet, a TV, a projector, etc. Of course, the device terminal can also be a VR device, a VR box, etc. In addition, the device terminal can also be a terminal for data processing such as a server.

Wherein, the video to be converted is stored in the device terminal, and the video to be converted can be an existing video or a video captured by the device terminal. The method first converts the acquired video to be converted. Signal and video signal are separated to obtain audio data and video data. Specifically, it decodes and decapsulates the video format to be converted. After the audio track is identified, it extracts the audio channel in the video, saves the extracted video data and audio data separately, and combines the audio data with the video The data is separated, and at this time, the video data is a video without audio information. It should be pointed out that, when the video to be converted does not contain audio, the corresponding audio data will not be separated, and the process directly enters step S20.

In one embodiment of the present invention, during specific operations, it utilizes the moviepy.editor open source module in Python to extract the audio channel in the video to be converted, and uses ffmpeg in its moviepy to read and export video and audio files, And use VideoFileClip to read the video, get the VideoFileClip object, record it as clip, and then perform arbitrary operations on the clip, such as cutting, merging, adjusting brightness, speed, extracting audio and splicing together with other clips, etc. and so on.

Step S20, splitting the video data frame by frame to obtain an image to be converted for each frame;

Wherein, in one embodiment of the present invention, after the video to be converted is separated into audio data and video data, it divides the separated video data into frames, so that the video data is split to obtain the Images to be converted, and all the separated images to be converted are saved separately in a folder for subsequent processing.

In one embodiment of the present invention, during specific operation, after it utilizes python to decode above-mentioned video to be converted, utilizes the Opencv (Open Source Computer Vision Library) open source library of python to call cv.VideoCapture API to open video data, and frame by frame Save the image so that the separated video data is extracted frame by frame to obtain the image to be converted for each frame, and then saved separately in the folder for subsequent processing.

Step S30, respectively performing in-depth processing on the image to be converted in each frame, and performing image processing on the image to be converted according to the result of the in-depth processing to generate a corresponding virtual right image;

Wherein, in one embodiment of the present invention, after the video data is divided into frames to obtain the image to be converted in each frame, depth processing is performed on the images to be converted in all frames to obtain the depth corresponding to the image to be converted in each frame At this time, through the algorithm, the image to be converted in each frame is processed based on the result of the corresponding depth map to generate the corresponding virtual right image, where the virtual right image is the virtual viewpoint image after the image to be converted has been processed by depth conversion , which is usually the image of the user under the virtual perspective of the right eye.

In the embodiment of the present invention, specifically, the image to be converted in each frame is input into the depth processing algorithm, and the depth processing algorithm adopts the depth estimation NeW CRFs and Midas open source algorithm of CVPR2022 in the vision top meeting, in the algorithm After processing, create a separate folder for the generated depth image of each frame to save, which is used to check whether the depth image generation is reasonable (specifically check whether the image depth extraction is correct, to prevent subsequent extraction errors) and facilitate subsequent steps .

The principle of depth estimation is: to obtain the distance between the object and the shooting point, and finally obtain a depth map, also known as the optical flow map, which records the parallax of the same object between different images, and then through the camera parameters, two The positional information between the shooting points can be converted into the distance between the object and the shooting point. For example, at a distance of 100m, the height of the car in the image is roughly 3cm. At this time, multiple objects are used to correct the depth of each other, so as to obtain relatively accurate prediction results. Among them, depth estimation is a subtask in the field of computer vision. Its purpose is to obtain the distance between the object and the shooting point. A series of tasks provide in-depth information.

Further, after the depth estimation algorithm is processed, based on the result of the depth estimation, image processing is performed on each initial frame to be converted to generate a virtual right image corresponding to each frame of the image to be converted.

Step S40, performing image processing on each frame of the image to be converted and the corresponding virtual right image together to generate a three-dimensional image;

Wherein, in one embodiment of the present invention, after converting the image to be converted in each frame into the corresponding virtual right image, the image to be converted in each frame is processed together with the corresponding virtual right image to generate A three-dimensional image, wherein the three-dimensional image is an image in which the image to be converted and the virtual right image are respectively located on the left and right sides, that is, the image to be converted and the virtual right image are respectively loaded on the left and right sides to splicing into a three-dimensional image. At this time, the image to be converted and the newly generated virtual right image are spliced left and right to generate the three-dimensional image required in the left and right form of the 3d video source in the naked eye 3d technology.

Step S50, merging the three-dimensional images of all frames to obtain three-dimensional video data;

Wherein, in one embodiment of the present invention, after image processing is performed on the image to be converted and the virtual right image together to generate a three-dimensional image, it is judged whether the image of each frame has been converted, and if not, continue to convert the remaining image to be converted Perform image processing on the converted image and the virtual right image to generate a three-dimensional image; if so, merge the converted three-dimensional images of all frames to obtain a video without sound. Specifically, in the embodiment of the present invention, with reference to the foregoing, the open source module moviepy.editor of Python is used to synthesize the 3D image of each frame into a video according to the original number of frames and the packaging format.

Step S60, merging the separated audio data and the generated 3D video data to obtain a 3D video;

Wherein, in one embodiment of the present invention, as described in step S10, when the converted video contains audio, the separated audio data and the generated 3D video data are combined to obtain a 3D video, that is, directly in the above-mentioned The separated audio data is imported into the 3D video data. At this time, since all time parameters are the same as those of the original video, there is no need for audio and video synchronization, and the 3D video can be directly generated.

And when the video to be converted does not contain audio and does not need audio data, the three-dimensional video can be obtained directly in the above-mentioned step S50; And the audio data is adjusted to obtain a 3D video adapted to the 3D video data.

Correspondingly, the converted 3D video can be applied to the following scenarios:

1. For scenes that need to be converted from platform video to naked-eye 3D video, ordinary video can be processed to generate a virtual right image of each frame with a virtual right-eye perspective through the video processing method provided in the embodiment of the present invention, and then by converting After splicing each frame of the image to be converted from the frame division of the ordinary video with the corresponding virtual right image, a 3D video source in the form of left and right in naked-eye 3D technology can be generated and used for projection.

2. For the 3D video source required by the VR box, provide technical support for the conversion of some 3D video sources through the video processing method provided in the embodiment of the present invention.

To sum up, the video processing method in the above-mentioned embodiments of the present invention firstly separates the audio and video from the original video to be converted into naked-eye 3D video, so that the audio data and video data are obtained first, and then the video data is processed frame by frame. Framing processing, so as to obtain the image to be converted in each frame, and then use visual processing technology to convert the image to be converted into a virtual right image of a specific 3D right image virtual perspective, and stitch the image to be converted with the virtual right image to generate a three-dimensional image, and then by merging the three-dimensional images of all frames to obtain three-dimensional video data, and incorporating the separated audio data into the three-dimensional video data, the final three-dimensional video is generated, so that the process of converting the original image into a naked-eye 3D image can be automated , automatically convert the original video into the corresponding naked-eye 3D video, thereby greatly reducing the high labor cost of naked-eye 3D video production, improving the production of naked-eye 3D video, which is conducive to the popularization of naked-eye 3D, and solves the problem of existing 3D video production The problem of high cost and low efficiency.

Embodiment two

Please refer to FIG. 2 , which is a schematic flowchart of a video processing method provided by the second embodiment of the present invention. For the convenience of description, only the parts related to the embodiment of the present invention are shown. The second embodiment is the same as the first embodiment The methods are roughly the same, for a brief description, for the parts not mentioned in this embodiment, you can refer to the corresponding content in the first embodiment, specifically the method includes:

Step S11, separating the acquired audio signal and video signal in the video to be converted to obtain audio data and video data.

Step S21 , splitting the video data frame by frame to obtain an image to be converted for each frame.

Wherein, in one embodiment of the present invention, it may also include after step S21:

Add text subtitles to each frame of the image to be converted.

That is to say, when the video to be converted only includes audio information and video information, but does not include subtitle information, it can correspondingly carry out the video processing method described in the embodiment of the present invention, in the split to be converted Convert the image to add text and subtitles, and the text and subtitles of each frame need to correspond to the audio in the audio data, so that the added text and subtitles can also be processed accordingly, so that the final image with subtitles can be generated 3D video.

Step S31, performing depth processing on each frame of the image to be converted to obtain a corresponding depth map;

Wherein, in one embodiment of the present invention, the aforementioned depth processing of each frame of the image to be converted to obtain the corresponding depth map can be achieved through the following steps:

The image to be converted of each frame is converted into a matrix containing information of three primary colors respectively, and the information of three primary colors includes three color channel information of red, green and blue;

Input the matrix of the three primary color information included in each frame into the depth processing algorithm for algorithm processing to obtain the corresponding depth map.

Specifically, the image to be converted of each frame is first converted into a matrix containing RGB (red (R), green (G), blue (B)) information, and then the converted matrix of each frame is input to the depth processing algorithm In this case, the depth processing algorithm will use the trained data set to identify the distance of different objects under normal circumstances, and combine the conditional random field to optimize the calculation speed, so as to quickly identify the depth of the object, and finally obtain the corresponding depth map . It can be understood that, in other embodiments of the present invention, the manner of converting the image to be converted into a depth map may also be other existing manners, which are set according to actual use requirements, and are not specifically limited here.

Step S41, respectively dynamically converting the depth map of each frame into a corresponding mask;

Wherein, in one embodiment of the present invention, the above-mentioned dynamic conversion of the depth map of each frame into the corresponding mask can be realized through the following steps:

Improve and adjust the contrast ratio of the depth map of each frame;

Convert the depth map of each frame into a grayscale image respectively;

Binarize the depth image converted to grayscale image for each frame to obtain the corresponding mask.

Specifically, after performing depth processing on the image to be converted to obtain a depth map,

Specifically, it first performs image processing on the depth map of each frame to improve the contrast of the main part. Specifically, the Gama value (gamma value) and contrast ratio of the depth map can be adjusted, and then converted into a grayscale image, and then processed. The binarization process obtains a binary image (that is, a mask).

Step S51, segmenting the foreground and background from the corresponding image to be converted according to the mask of each frame;

Wherein, in one embodiment of the present invention, after the depth map is dynamically converted into a mask, the image segmentation technology is used to extract a certain part of the image separately, specifically, the mask and the image to be converted are bitwise operated ( Also known as bit operation), so that the foreground is separated from the background, and then intercepted and inverted according to the mask, so that everything except the foreground area is black. At this time, the main part of the foreground can be deducted and extracted.

Step S61, performing rotation stretching and left shifting transformation on the extracted foreground of each frame respectively, and filling the transformed result into the corresponding background;

Wherein, in one embodiment of the present invention, after the foreground and the background are segmented, after the foreground is extracted, all the pixels in the extracted foreground are rotated, stretched and shifted to the left based on the middle axis, so that Depth transformation that highlights the subject alone, then places all transformed pixels back into the background.

Step S71, performing image restoration on the image filled in the background of each frame to obtain the corresponding virtual right image;

Wherein, in one embodiment of the present invention, the image restoration of the image filled into the background of each frame above to obtain the corresponding virtual right image can be realized through the following steps:

Draw a mask based on the pixel positions of the image that is filled into the background each frame;

Fill the holes outside the mask to get the corresponding virtual right image.

Among them, since the foreground is moved to the left based on the depth, after the subsequent filling back to the background, there will be some holes in the generated image when the depth information undergoes a large transformation, and the blank part needs to be repaired at this time; specifically, according to The shifted image pixel position is used to draw a mask, and then the hole area is filled, so as to obtain a virtual right image with the same format and size as the original image to be converted, but part of the layer position is shifted. At present, for computing power considerations, the mean filling method or the nearest neighbor interpolation method is usually used, so that the overall image is finally smoothed and the small "cracks" are erased to generate a virtual right image of the virtual perspective of the 3d right image.

Step S81, performing image processing on the image to be converted and the corresponding virtual right image of each frame to generate a three-dimensional image;

Wherein, in one embodiment of the present invention, the above-mentioned processing of the image to be converted and the corresponding virtual right image of each frame to generate a three-dimensional image can be realized through the following steps:

Convert the image to be converted and the corresponding virtual right image of each frame into corresponding two sets of arrays;

Perform matrix broadcasting on the two sets of arrays in each frame, and merge the arrays;

Transform the merged arrays in each frame into corresponding 3D images respectively.

Among them, by first converting the two sets of images into multidimensional arrays, the images are converted into multidimensional arrays at this time, so that the problem of stitching two sets of images can be converted into a combination of two multidimensional arrays, and then the merged The multi-dimensional array is converted into a three-dimensional image, so that the two sets of images can be spliced into a three-dimensional image. At this time, if you observe the left and right views in the 3D image, you can find that the two pictures have a slight change, and this slight change is essentially similar to the difference between the left and right eyes of the human eye. Therefore, by splicing the original image with the newly generated virtual perspective, the perspective of the three-dimensional object usually seen by the human eye can be simulated.

Step S91, combining the 3D images of all frames to obtain 3D video data.

Step S101, combining the separated audio data and the generated 3D video data to obtain a 3D video.

To sum up, the video processing method in the above-mentioned embodiments of the present invention firstly separates the audio and video from the original video to be converted into naked-eye 3D video, so that the audio data and video data are obtained first, and then the video data is processed frame by frame. Framing processing, so as to obtain the image to be converted in each frame, generate a depth map based on the depth estimation technology of the image to be converted, and then dynamically convert the depth map into a mask, and then use the mask to extract all the pixels of the foreground in the image to be converted Rotate, stretch and move to the left, and based on the image repair technology, perform image repair on the hole generated by the movement, so that the virtual right image under the virtual perspective of the 3D right image is finally generated, and then the image to be converted is stitched with the virtual right image 3D images are generated, and the 3D video data is obtained by merging the 3D images of all frames, and the separated audio data is incorporated into the 3D video data, so that the 3D video is finally generated. At this time, the video processing method is automatically processed by software Therefore, the problems of high cost and low efficiency of existing 3D video production are solved.

Embodiment three

Please refer to FIG. 3, which is a schematic structural diagram of a video processing system provided by the third embodiment of the present invention. For ease of description, only the parts related to the embodiment of the present invention are shown. The system includes:

Separation module 11, for separating the audio signal and the video signal in the acquired video to be converted to obtain audio data and video data;

The splitting module 21 is used to split the video data frame by frame to obtain the image to be converted of each frame;

The first image processing module 31 is configured to perform depth processing on each frame of the image to be converted, and perform image processing on the image to be converted according to the result of the depth processing to generate a corresponding virtual right image, the virtual right image being the image to be converted The virtual viewpoint image processed by depth conversion;

The second image processing module 41 is used to respectively perform image processing on the image to be converted and the corresponding virtual right image of each frame to generate a three-dimensional image, and the three-dimensional image is that the image to be converted and the virtual right image are respectively located on the left and right sides Image;

The first merging module 51 is used for merging the three-dimensional images of all frames to obtain three-dimensional video data;

The second merging module 61 is configured to combine the separated audio data and the generated 3D video data to obtain 3D video.

Further, in one embodiment of the present invention, the first image processing module 31 includes:

A depth map processing unit, configured to perform depth processing on the image to be converted in each frame to obtain a corresponding depth map;

A mask conversion unit is used to dynamically convert the depth map of each frame into a corresponding mask;

The image segmentation unit is used to segment the foreground and the background from the corresponding image to be converted according to the mask of each frame;

The image transformation unit is used to perform rotation stretching and left shift transformation on the extracted foreground of each frame respectively, and fill the transformed result into the corresponding background;

The virtual right image processing unit is configured to respectively perform image restoration on the image filled in the background of each frame to obtain a corresponding virtual right image.

Further, in one embodiment of the present invention, the depth map processing unit is used for:

The image to be converted of each frame is converted into a matrix containing information of three primary colors respectively, and the information of three primary colors includes three color channel information of red, green and blue;

Input the matrix of the three primary color information included in each frame into the depth processing algorithm for algorithm processing to obtain the corresponding depth map.

Further, in one embodiment of the present invention, the mask conversion unit is used for:

Improve and adjust the contrast ratio of the depth map of each frame;

Convert the depth map of each frame into a grayscale image respectively;

Binarize the depth image converted to grayscale image for each frame to obtain the corresponding mask.

Further, in one embodiment of the present invention, the virtual right image processing unit is used for:

Draw a mask based on the pixel positions of the image that is filled into the background each frame;

Fill the holes outside the mask to get the corresponding virtual right image.

Further, in one embodiment of the present invention, the second image processing module 41 includes:

An array conversion unit, configured to convert the image to be converted and the corresponding virtual right image of each frame into corresponding two sets of arrays;

The array merging unit is used to broadcast the matrix to two groups of arrays in each frame respectively, and to merge the arrays;

The image conversion unit is used to respectively convert the combined array in each frame into a corresponding three-dimensional image.

Further, in one embodiment of the present invention, the system also includes:

The subtitle adding module is used for adding text subtitles to the image to be converted in each frame.

The implementation principles and technical effects of the video processing system provided by the embodiments of the present invention are the same as those of the foregoing method embodiments. For brief description, for the parts not mentioned in the device embodiments, reference may be made to the corresponding content in the foregoing method embodiments.

Embodiment Four

Another aspect of the present invention also proposes a device terminal, please refer to FIG. 4, which shows the device terminal in the fourth embodiment of the present invention, including a memory 20, a processor 10, and a device stored in the memory and capable of running on the processor. A program 30, when the processor 10 executes the program 30, implements the video processing method as described in Embodiment 1 or Embodiment 2 above.

Wherein, the processor 10 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chips in some embodiments, and is used to run the program codes stored in the memory 20 or Processing data, such as executing access restriction programs, etc.

Wherein, the memory 20 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 20 may be an internal storage unit of the device terminal in some embodiments, such as a hard disk of the device terminal. In other embodiments, the memory 20 can also be an external storage device of the equipment terminal, such as a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card) equipped on the equipment terminal. )wait. Further, the memory 20 may also include both an internal storage unit of the device terminal and an external storage device. The memory 20 can be used not only to store application software and various data installed in the device terminal, but also to temporarily store data that has been output or will be output.

To sum up, the device terminal in the above-mentioned embodiments of the present invention separates the original video to be converted into a naked-eye 3D video through audio and video separation, so that the audio data and video data are obtained first, and then the video data is divided frame by frame. Frame processing, so as to obtain the image to be converted for each frame, and then use visual processing technology to convert the image to be converted into a virtual right image of a specific 3D right image virtual perspective, and stitch the image to be converted with the virtual right image to generate a three-dimensional image , and then obtain 3D video data by merging the 3D images of all frames, and incorporate the separated audio data into the 3D video data, so that a 3D video is finally generated, so that the process of converting the original image into a naked-eye 3D image can be automated, Automatically convert the original video into the corresponding naked-eye 3D video, thereby greatly reducing the high labor cost of naked-eye 3D video production, improving the production of naked-eye 3D video, which is conducive to the popularization of naked-eye 3D and solving the existing 3D video production cost high and low efficiency issues.

An embodiment of the present invention also provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps of the video processing method described in the foregoing embodiments are implemented. The readable storage medium, such as: ROM/RAM, magnetic disk, optical disk, etc.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units or modules according to needs. The completion of the modules means that the internal structure of the storage device is divided into different functional units or modules, so as to complete all or part of the functions described above. Each functional unit and module in the implementation manner may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware It can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application.

Those skilled in the art will understand that the logic and/or steps shown in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, which can be specifically implemented in on any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute them), or Used in conjunction with these instruction execution systems, devices or equipment. As far as this specification is concerned, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit programs for instruction execution systems, devices or devices or use in conjunction with these instruction execution systems, devices or devices.

More specific examples (non-exhaustive list) of readable storage media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since it may be possible, for example, by optical scanning of the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. The program is processed electronically and then stored in memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

Those skilled in the art can understand that the composition structure shown in Figure 3 does not constitute a limitation to the video processing system of the present invention, and may include more or less components than those shown in the figure, or combine some components, or different components Arrangement, and the video processing method in Fig. 1-2 also adopts more or less components shown in Fig. 3, or some components are combined, or different component arrangements are implemented. The units, modules, etc. referred to in the present invention refer to a series of computer programs that can be executed by the processor (not shown in the figure) in the video processing system and can complete specific functions, which can all be stored in the video processing system. storage device (not shown).

Those skilled in the art can also understand that the composition structure shown in FIG. 4 does not constitute a limitation to the device terminal of the present invention, and may include more or less components than shown in the figure, or combine some components, or different components Arrangement, and the video processing method in Fig. 1-2 also adopts more or less components shown in Fig. 4, or some components are combined, or different component arrangements are implemented.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present invention. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims (10)

1. A video processing method, characterized in that the method comprises: Separating the acquired audio signal and video signal in the video to be converted to obtain audio data and video data; Split the video data frame by frame to obtain the image to be converted for each frame; Perform depth processing on the image to be converted for each frame, and perform image processing on the image to be converted according to the result of the depth processing to generate a corresponding virtual right image, the virtual right image is a virtual viewpoint image after the image to be converted has been processed by depth conversion ; Carrying out image processing on the image to be converted and the corresponding virtual right image of each frame respectively to generate a three-dimensional image, the three-dimensional image is an image in which the image to be converted and the virtual right image are respectively located on the left and right sides; Merging the three-dimensional images of all frames to obtain three-dimensional video data; Combining the separated audio data and the generated 3D video data to obtain 3D video. 2. The video processing method according to claim 1, wherein the image to be converted of each frame is subjected to in-depth processing, and the image to be converted is processed according to the result of the in-depth processing to generate a corresponding virtual right image The steps include: Depth processing is performed on the image to be converted for each frame to obtain a corresponding depth map; Dynamically convert the depth map of each frame into the corresponding mask; Separate the foreground and background from the corresponding image to be converted according to the mask of each frame; The extracted foreground of each frame is rotated, stretched and left-shifted respectively, and the transformed result is filled into the corresponding background; The images filled in the background of each frame are respectively image restored to obtain the corresponding virtual right image. 3. The video processing method according to claim 2, wherein the step of performing depth processing on the image to be converted of each frame to obtain a corresponding depth map comprises: The image to be converted of each frame is converted into a matrix containing information of three primary colors respectively, and the information of three primary colors includes three color channel information of red, green and blue; Input the matrix of the three primary color information included in each frame into the depth processing algorithm for algorithm processing to obtain the corresponding depth map. 4. The video processing method according to claim 2, wherein the step of dynamically converting the depth map of each frame into a corresponding mask respectively comprises: Improve and adjust the contrast ratio of the depth map of each frame; Convert the depth map of each frame into a grayscale image respectively; Binarize the depth image converted to grayscale image for each frame to obtain the corresponding mask. 5. The video processing method according to claim 2, wherein the steps of performing image repair on the images filled into the background by each frame respectively to obtain the corresponding virtual right image include: Draw a mask based on the pixel positions of the image that is filled into the background each frame; Fill the holes outside the mask to get the corresponding virtual right image. 6. The video processing method according to claim 1, wherein the step of performing image processing on the image to be converted of each frame together with the corresponding virtual right image to generate a three-dimensional image comprises: Convert the image to be converted and the corresponding virtual right image of each frame into corresponding two sets of arrays; Perform matrix broadcasting on the two sets of arrays in each frame, and merge the arrays; Transform the merged arrays in each frame into corresponding 3D images respectively. 7. video processing method as claimed in claim 1, is characterized in that, described video data is carried out frame by frame and obtains the step of the image to be converted of each frame and also comprises after: Add text subtitles to each frame of the image to be converted. 8. A video processing system, characterized in that the system comprises: A separation module, configured to separate the acquired audio signal and video signal in the video to be converted to obtain audio data and video data; A splitting module is used to split the video data frame by frame to obtain an image to be converted for each frame; The first image processing module is used to perform in-depth processing on the image to be converted in each frame, and perform image processing on the image to be converted according to the result of the in-depth processing to generate a corresponding virtual right image, the virtual right image is the image to be converted through image processing The virtual viewpoint image after depth conversion processing; The second image processing module is used to respectively perform image processing on the image to be converted and the corresponding virtual right image of each frame to generate a three-dimensional image, and the three-dimensional image is the image to be converted and the virtual right image respectively located on the left and right sides image; The first combining module is used to combine the three-dimensional images of all frames to obtain three-dimensional video data; The second merging module is used for merging the separated audio data and the generated 3D video data to obtain 3D video. 9. A readable storage medium storing a program, characterized in that, when the program is executed by a processor, the video processing method according to any one of claims 1 to 8 is realized. 10. A device terminal, comprising a memory, a processor, and a program stored on the memory and operable on the processor, characterized in that, when the processor executes the program, any one of claims 1 to 8 is implemented The video processing method described.

Images