CN107426507A - Video image splicing apparatus and its joining method - Google Patents

Abstract

The present invention provides a kind of video image splicing apparatus and its joining method, video image splicing apparatus includes image capture module, for obtaining target video image and video image to be transformed, the target video image is overlapping with the subregion of the video image to be transformed；Image pre-processing module, for carrying out image preconditioning to the video image to be transformed of reception, to obtain preprocessed video image；Image mosaic accelerating module, for the overlapping region of the target video image of reception and the preprocessed video image to be carried out into acceleration splicing, to obtain spliced video image.Video image splicing apparatus proposed by the present invention includes image mosaic accelerating module, can realize that the video of embedded architecture gathers in real time, splicing and display.

Description

Video image splicing device and splicing method thereof

technical field

The invention relates to the field of video image processing, in particular to a video image splicing device and a splicing method thereof.

Background technique

With the rapid development of computer technology, the research and application of digital image processing technology in various fields has become increasingly in-depth and extensive. Video is a continuous and dynamic collection of images, which has attracted great attention from academia and industry because it contains more detailed information. In the field of video processing, in order to obtain high-resolution large scene information, video stitching is usually used to stitch multiple videos with overlapping areas. However, there are many challenges in video stitching technology, mainly real-time Handling, including dynamic objects will produce ghosting and other challenges.

At present, two solutions are mainly adopted to face these challenges: one is to consider more restrictive factors in terms of algorithm implementation; the other is to achieve through hardware acceleration, such as using powerful servers and GPUs. Although both schemes can improve the efficiency or quality of video processing in some aspects, the first scheme is difficult to meet the real-time requirements due to the complexity of the algorithm; the second scheme relies too much on hardware and is not suitable for Embedded and other application scenarios with limited resources.

Contents of the invention

In order to solve the above problems, the present invention proposes a video image splicing device and a splicing method thereof, which can realize real-time video acquisition, splicing processing and display of embedded architecture.

The specific technical solution proposed by the present invention is to provide a video image splicing device, including: an image acquisition module for acquiring a target video image and a video image to be transformed, the target video image and the partial area of the video image to be transformed Overlapping; image preprocessing module, used to perform image conversion preprocessing on the received video image to be converted, to obtain preprocessed video image; image splicing acceleration module, used to receive the target video image and the preprocessed video image Process overlapping areas of video images for accelerated stitching to obtain stitched video images.

Further, the image splicing acceleration module includes: a first image format conversion unit, configured to perform format conversion on the target video image and the pre-processed video image respectively; The video image and the pre-processed video image are filtered to obtain a first filtered image and a second filtered image; a downsampling unit is configured to perform downsampling processing on the first filtered image and the second filtered image respectively to construct The first Gaussian pyramid and the second Gaussian pyramid; a difference unit, which is used to perform differential processing between adjacent layers of the first Gaussian pyramid and between adjacent layers of the second Gaussian pyramid, so as to obtain the first Gaussian pyramid Placian Pyramid and the second Laplacian Pyramid; a calculation unit for calculating the value corresponding to the overlapping area of the first Laplacian Pyramid and the corresponding overlapping area of the second Laplacian Pyramid The numerical value is weighted and averaged layer by layer to obtain the Laplacian pyramid after the weighted average; the upsampling unit is used to perform upsampling processing on the Laplacian pyramid after the weighted average to obtain the Laplacian pyramid after the format inverse conversion Video image; a second image format converting unit, configured to perform format inverse conversion on the video image to be format inversely converted to obtain a spliced video image.

Further, the image preprocessing module includes: a feature extraction unit, used to extract the feature points of the target video image and the video image to be transformed; a feature matching unit, used to compare the target video image according to the Hamming distance Match the feature points of the video image to be transformed with the feature points of the video image to be transformed and calculate a homography matrix according to the matching relationship; the image transformation unit is used to use the homography matrix to perform image transformation preprocessing on the video image to be transformed, to Obtain preprocessed video images.

Further, it also includes a storage unit (DDR) connected between the image preprocessing module and the image splicing acceleration module, and the DDR is used for storing target video images and preprocessed video images.

Further, it also includes a display module connected to the image splicing acceleration module, and the display module is used for displaying the spliced video images.

The present invention also provides a splicing method of video images, comprising:

Acquiring the target video image and the video image to be transformed; wherein, the target video image and the partial area of the video image to be transformed overlap;

Perform image transformation preprocessing on the video image to be transformed to obtain a preprocessed video image;

The overlapping regions of the target video image and the preprocessed video image are spliced to obtain a spliced video image.

Further, the method for splicing the overlapping regions of the target video image and the preprocessed video image includes:

Perform format conversion on the target video image and the preprocessed video image respectively;

Filtering the format-converted target video image and the preprocessed video image respectively by using a Gaussian kernel function to obtain a first filtered image and a second filtered image;

respectively performing downsampling processing on the first filtered image and the second filtered image to construct a first Gaussian pyramid and a second Gaussian pyramid;

Perform difference processing between adjacent layers of the first Gaussian pyramid and between adjacent layers of the second Gaussian pyramid to obtain the first Laplacian pyramid and the second Laplacian pyramid;

Carry out layer-by-layer weighted average to the numerical value corresponding to the overlapping area of the first Laplacian pyramid and the numerical value corresponding to the overlapping area of the second Laplacian pyramid, to obtain the Laplacian pyramid after the weighted average;

Perform upsampling processing on the weighted average Laplacian pyramid to obtain the video image to be inversely converted;

The format of the video image to be reverse-converted is performed to obtain the spliced video image.

Further, performing image transformation preprocessing on the video image to be transformed to obtain a preprocessed video image includes:

Extract the feature points of the target video image and the video image to be transformed;

Matching the feature points of the target video image and the feature points of the video image to be transformed according to the Hamming distance;

Calculate the homography matrix according to the matching relationship;

Perform image transformation preprocessing on the video image to be transformed by using the homography matrix, so as to obtain the preprocessed video image.

Further, after performing image conversion preprocessing on the video image to be converted to obtain the preprocessed video image, the target video image and the preprocessed video image are stored in the DDR;

The target video image and the pre-processed video image are read from the DDR before splicing the overlapping area of the target video image and the pre-processed video image.

Further, after the spliced video image is obtained, the spliced video image is displayed.

The video image splicing device and the splicing method thereof proposed by the present invention, the video image splicing device includes an image splicing acceleration module, the image splicing acceleration module is a user-defined IP core, and the image splicing acceleration module can be used according to the user-defined image splicing algorithm. Video images are spliced, and the VDMA unit, image splicing acceleration module and output unit are realized through FPGA, and through software and hardware collaborative design, real-time video acquisition, splicing processing and display of embedded architecture are realized.

Description of drawings

The above and other aspects, features and advantages of embodiments of the present invention will become more apparent through the following description in conjunction with the accompanying drawings, in which:

Fig. 1 is a schematic diagram of the circuit structure of a video image splicing device;

Fig. 2 is a schematic diagram of the circuit structure of the image preprocessing module;

Fig. 3 is a schematic structural diagram of the programmable system PL circuit;

Fig. 4 is a schematic diagram of the circuit structure of the image splicing acceleration module;

Fig. 5 is a schematic flow chart of a splicing method for video images;

Fig. 6 is a schematic flow chart of step S2;

FIG. 7 is a schematic flow chart of step S4.

detailed description

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. This invention may, however, be embodied in many different forms and should not be construed as limited to the specific embodiments set forth herein. Rather, the embodiments are provided to explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to particular intended uses.

The video image splicing device provided by the present invention is an embedded video image splicing device based on Xilinx ZYNQ platform, which includes a processing system (Processing System, PS) and a programmable system (Programmable Logic, PL), where PS is an ARM processor. In the present invention, PS and PL are integrated on one chip.

With reference to Fig. 1, the video image splicing device that the present embodiment provides comprises image acquisition module 10, PS, PL and is connected to the storage unit (DDR) 30 between described PS and PL, and preferably, DDR 30 is double rate synchronous DRAM. The PS includes an image preprocessing module 20 , and the PL includes an image splicing acceleration module 40 and a display module 50 .

The image acquisition module 10 is used to obtain the target video image and the video image to be converted, the target video image overlaps with the partial area of the video image to be transformed, for example, the image acquisition module 10 is a camera image acquisition module 10 and an image preprocessing module 20 connection, for example, the image acquisition module 10 is connected to the image preprocessing module 20 through a USB interface, and the image preprocessing module 20 is used to perform image conversion preprocessing on the received video image to be converted to obtain a preprocessed video image. The DDR 30 is connected between the image preprocessing module 20 and the image splicing acceleration module 40, and is used for storing target video images and preprocessed video images. The image splicing acceleration module 40 is connected with the DDR 30 and then connected with the display module 50. The image splicing acceleration module 40 is used to splice the overlapping areas of the standard video image and the pre-processed video image to obtain the spliced video image. Wherein, the image splicing acceleration module 40 is a user-defined IP core. The display module 50 is used for displaying the spliced video images. In order to obtain a video image with good splicing quality, the area of the overlapping region is 20% to 80% of the smaller area of the target video image or the video image to be transformed; simultaneously, considering the amount of calculation, preferably, the area of the overlapping region It is 40% to 60% of the smaller area of the target video image or the video image to be transformed. Referring to FIG. 2 , the image preprocessing module 20 includes a feature extraction unit 21 , a feature matching unit 22 and an image transformation unit 23 connected in sequence. The feature extraction unit 21 is connected with the image acquisition module 10, the feature extraction unit 21 is used to receive the target video image and the video image to be transformed and extracts the feature points of the target video image and the video image to be transformed respectively, and the feature matching unit 22 is used for according to Hamming The distance matches the feature points of the target video image and the feature points of the video image to be transformed and calculates a homography matrix according to the matching relationship, and the image transformation unit 23 is used to utilize the homography matrix to transform the video image Perform image transformation preprocessing to obtain preprocessed video images. The image conversion unit 23 is also connected to the DDR 30, and is used to send the pre-processed video image to the DDR 30 for storage.

Referring to FIG. 3 , the PL system further includes a VDMA unit 41 and an output unit 42 . VDMA unit 41 is connected between DDR 30 and image splicing acceleration module 40, and VDMA unit 41 carries out data communication between DDR 30 by HP interface, and VDMA unit 41 is used for extracting target video image and preprocessing video image from DDR 30 and Data transmission is realized between the AXI4-Stream interface and the image splicing acceleration module 40 . The output unit 42 is connected between the image splicing acceleration module 40 and the display module 50, and the output unit 42 is used to convert the format of the spliced video image, so that the format of the spliced video image matches the display format of the display module 50 . For example, the display module 50 is an HDMI display, and the color mode of the spliced video image received by the output unit 42 from the image splicing acceleration module 40 is RGB, and the output unit 42 converts the color mode of the spliced video image into YCbCr, and then displays Module 50 is displayed.

In this embodiment, the VDMA unit 41, the image splicing acceleration module 40 and the output unit 42 are realized by FPGA, of course, they can also be realized by other ways, which are shown here as an example and not intended to limit the present invention.

Referring to Fig. 4, image mosaic acceleration module 40 comprises the first image format conversion unit 100, filter unit 101, down-sampling unit 102, difference unit 103, calculation unit 104, up-sampling unit 105 and the second image format conversion unit 106 connected in sequence . The first image format conversion unit 100 is also connected to the VDMA unit 41 , and the second image format conversion unit 106 is connected to the up-sampling unit 105 and then connected to the output unit 42 .

The first image format conversion unit 100 is used to convert the target video image in AXIvideo format and the preprocessed video image into Mat format respectively, and the filtering unit 101 is used to filter the target video image in Mat format and the preprocessed video image respectively, To obtain the first filtered image and the second filtered image, the downsampling unit 102 is configured to respectively downsample the first filtered image and the second filtered image to construct a first Gaussian pyramid and a second Gaussian pyramid. The difference unit 103 is used to perform differential processing between adjacent layers of the first Gaussian pyramid and between adjacent layers of the second Gaussian pyramid, so as to obtain the first Laplacian pyramid and the second Laplacian pyramid, and calculate Unit 104 is used to carry out layer-by-layer weighted average of the value corresponding to the overlapping region of the first Laplacian pyramid and the value corresponding to the overlapping region of the second Laplacian pyramid, so as to obtain the weighted average For the Laplacian pyramid, the upsampling unit 105 is used to perform upsampling processing on the received Laplacian pyramid after the weighted average, so as to obtain a video image to be inversely converted in format, and the second image format conversion unit 106 is used for Carrying out format inverse conversion on the video image to be inversely formatted to obtain a spliced video image, that is, converting the spliced video image in Mat format into a video image in AXIvideo format.

Referring to FIG. 5 , this embodiment also provides a splicing method of the video image splicing device, the method comprising:

Step S1. Obtain the target video image and the video image to be converted; wherein, the target video image and the video image to be converted partially overlap. In order to obtain a video image with good splicing quality, the area of the overlapping region is 20% to 80% of the smaller area of the target video image or the video image to be transformed; simultaneously, considering the amount of calculation, preferably, the area of the overlapping region It is 40% to 60% of the smaller area of the target video image or the video image to be transformed.

Step S2, performing image transformation preprocessing on the video image to be transformed to obtain a preprocessed video image.

Referring to FIG. 6, specifically, step S2 includes:

S21. Extract feature points of the target video image and the video image to be transformed, wherein the feature point extraction algorithm adopts ORB (ORiented Brief) algorithm.

S22. Match the feature points of the target video image and the feature points of the video image to be transformed according to the Hamming distance.

S23. Calculate and obtain a homography matrix according to the matching relationship.

S24. Perform image transformation preprocessing on the video image to be transformed by using the homography matrix to obtain a preprocessed video image.

Referring again to FIG. 5 , step S3 , storing the target video image and the preprocessed video image in the DDR 30 .

Step S4, splicing the overlapping area of the target video image and the pre-processed video image to obtain a spliced video image.

Referring to FIG. 7, specifically, step S4 includes:

S41. Read the target video image and the pre-processed video image from the DDR 30, and perform format conversion on the target video image and the pre-processed video image respectively, wherein the format conversion is for example converting the target video image in AXIvideo format and the pre-processed video image to The target video image in Mat format and the preprocessed video image. S42. Filter the format-converted target video image and the preprocessed video image respectively by using a Gaussian kernel function to obtain a first filtered image and a second filtered image.

S43. Perform downsampling processing on the first filtered image and the second filtered image respectively, so as to construct a first Gaussian pyramid and a second Gaussian pyramid.

S44. Perform differential processing on adjacent layers of the first Gaussian pyramid and adjacent layers of the second Gaussian pyramid, to obtain the first Laplacian pyramid and the second Laplacian pyramid.

S45. Perform layer-by-layer weighted averaging of the value corresponding to the overlapping area of the first Laplacian pyramid and the value corresponding to the overlapping area of the second Laplacian pyramid to obtain a weighted-averaged Laplacian pyramid.

S46. Perform up-sampling processing on the weighted-averaged Laplacian pyramid to obtain a video image to be inversely converted, wherein the video image to be inversely converted is in Mat format.

S47. Perform format inverse conversion on the video image to be format inverse converted to obtain a spliced video image, wherein the spliced video image is in AXIvideo format.

Referring to FIG. 5 again, step S5, displaying the spliced video images.

The video image splicing device and the splicing method thereof proposed by the present invention, the video image splicing device includes an image splicing acceleration module, the image splicing acceleration module is a user-defined IP core, and the image splicing acceleration module can be used according to the user-defined image splicing algorithm. Video images are spliced, and the VDMA unit, image splicing acceleration module and output unit are realized through FPGA, and through software and hardware collaborative design, real-time video acquisition, splicing processing and display of embedded architecture are realized.

The above description is only the specific implementation of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present application, some improvements and modifications can also be made. It should be regarded as the protection scope of this application.

Claims (10)

1. A video image stitching device, characterized in that, comprising: An image acquisition module, configured to acquire a target video image and a video image to be converted, where the target video image overlaps with a part of the video image to be converted; An image preprocessing module, configured to perform image transformation preprocessing on the received video image to be transformed, so as to obtain a preprocessed video image; An image splicing acceleration module, configured to splice the overlapping areas of the received target video image and the preprocessed video image to obtain a spliced video image. 2. video image splicing device according to claim 1, is characterized in that, described image splicing acceleration module comprises: A first image format conversion unit, configured to perform format conversion on the target video image and the pre-processed video image respectively; a filtering unit, configured to respectively filter the format-converted target video image and the preprocessed video image to obtain a first filtered image and a second filtered image; a downsampling unit, configured to downsample the first filtered image and the second filtered image, respectively, to construct a first Gaussian pyramid and a second Gaussian pyramid; A differential unit, configured to perform differential processing between adjacent layers of the first Gaussian pyramid and between adjacent layers of the second Gaussian pyramid, so as to obtain the first Laplacian pyramid and the second Laplacian pyramid Pyramid of Sri Lanka; A calculation unit, configured to carry out layer-by-layer weighted averaging of the value corresponding to the overlapping region of the first Laplacian pyramid and the value corresponding to the overlapping region of the second Laplacian pyramid, to obtain a weighted average the Laplace Pyramid; An upsampling unit, configured to perform upsampling processing on the weighted and averaged Laplacian pyramid, so as to obtain a video image to be format reverse-converted; The second image format conversion unit is configured to perform inverse format conversion on the video image to be inversely converted to obtain a spliced video image. 3. video image splicing device according to claim 1, is characterized in that, described image preprocessing module comprises: A feature extraction unit for extracting feature points of the target video image and the video image to be transformed; A feature matching unit, configured to match the feature points of the target video image and the feature points of the video image to be transformed according to the Hamming distance and calculate a homography matrix according to the matching relationship; An image transformation unit, configured to use the homography matrix to perform image transformation preprocessing on the video image to be transformed, so as to obtain a preprocessed video image. 4. The video image splicing device according to claim 1, further comprising a storage unit connected between the image preprocessing module and the image splicing acceleration module, the storage unit is used to store the target video images and preprocess video images. 5. The video image splicing device according to claim 1, further comprising a display module connected to the image splicing acceleration module, the display module being used to display the spliced video images. 6. A splicing method of video images, characterized in that, comprising: Acquiring the target video image and the video image to be transformed; wherein, the target video image and the partial area of the video image to be transformed overlap; Perform image transformation preprocessing on the video image to be transformed to obtain a preprocessed video image; The overlapping regions of the target video image and the preprocessed video image are spliced to obtain a spliced video image. 7. the splicing method of video image according to claim 6, is characterized in that, the method for splicing the overlapping area of target video image and preprocessing video image comprises: Perform format conversion on the target video image and the preprocessed video image respectively; Filtering the format-converted target video image and the preprocessed video image respectively by using a Gaussian kernel function to obtain a first filtered image and a second filtered image; respectively performing downsampling processing on the first filtered image and the second filtered image to construct a first Gaussian pyramid and a second Gaussian pyramid; Respectively performing differential processing between adjacent layers of the first Gaussian pyramid and between adjacent layers of the second Gaussian pyramid to obtain the first Laplacian pyramid and the second Laplacian pyramid; Carry out layer-by-layer weighted average to the numerical value corresponding to the overlapping area of the first Laplacian pyramid and the numerical value corresponding to the overlapping area of the second Laplacian pyramid, to obtain the Laplacian pyramid after the weighted average; Perform upsampling processing on the weighted average Laplacian pyramid to obtain the video image to be inversely converted; The format of the video image to be reverse-converted is performed to obtain the spliced video image. 8. The splicing method of video images according to claim 6, wherein the video image to be transformed is subjected to image transformation preprocessing, so that the method for obtaining the preprocessing video image comprises: Extract the feature points of the target video image and the video image to be transformed; Matching the feature points of the target video image and the feature points of the video image to be transformed according to the Hamming distance; Calculate the homography matrix according to the matching relationship; Perform image transformation preprocessing on the video image to be transformed by using the homography matrix, so as to obtain the preprocessed video image. 9. The splicing method of video images according to claim 6, characterized in that, after the video images to be transformed are subjected to image conversion pre-processing to obtain the pre-processing video images, the target video images and the pre-processing video images are stored in the storage in the unit; Before splicing the overlapping area of the target video image and the pre-processing video image, the target video image and the pre-processing video image are read from the storage unit. 10. The method for splicing video images according to claim 6, characterized in that after the spliced video images are obtained, the spliced video images are displayed.