CN107333064B - Spherical panoramic video splicing method and system - Google Patents

Abstract

The invention discloses a spherical panoramic video splicing method, which comprises the following steps: acquiring a video pair shot by the fisheye lens; respectively taking the first image and the second image which have the one-to-one correspondence relationship as image pairs to execute splicing operation to form a spliced video; namely, the panoramic video is automatically spliced by carrying out proper coordinate transformation on the video shot by the fisheye lens. In the solving process of the transformation matrix, angular point detection, empirical region search and HSV color space matching are required to realize automatic detection of matching points; the method has the advantages of small shooting quantity, high splicing speed and no need of manual adjustment and intervention after the splicing process is started. The generated video has good splicing effect, the splicing of the middle joint position is accurate, and the transition is natural. From the whole video playing, the image jitter and the dislocation are avoided. The invention also discloses a splicing system of the spherical panoramic video, which has the beneficial effects.

Description

Method and system for splicing spherical panoramic video

technical field

The invention relates to the technical field of image processing, in particular to a method and system for splicing spherical panoramic videos.

Background technique

Panoramic video is a new type of video. Compared with traditional video, which can only see information in a certain direction and range in space, panoramic video records all-round information in 360-degree space, and can interact with users, such as changing the viewing direction and switching scenes. It is assembled from multiple ordinary videos shot in different directions. Video splicing has important applications in military, medical, computer vision, video conferencing, security and other fields. Panoramic video is developed from panorama. The stitching of panoramas has been studied a lot.

According to the different technical methods, the most commonly used methods can be divided into three categories: frequency domain-based methods, feature-based methods, and gray gradient-based methods. Among them, the method based on frequency domain is represented by the improved method of fractional Fourier transform, which can well solve the registration problem of translation, rotation and uniform scaling, but cannot solve the registration problem of perspective projection transformation. allow. The feature-based method is represented by the Scale Invariant Feature Transform (SIFT) algorithm, which can well solve the registration problem of various transformations including perspective transformation, but the time complexity of this method is very high, and it is not suitable for the matching of texture images. The quasi-results are not ideal either. The method based on gray gradient is to solve the projection transformation parameters by minimizing the image gray difference, and use the LM optimization technique to obtain the perspective projection parameters, but these methods are easily affected by illumination changes.

According to the different construction methods, panoramic images can be divided into three types: cylindrical, square and spherical (the same is true for panoramic videos). Among them, column panorama has been studied more, and various panorama related technologies are also pioneered by it. However, the column panorama has an inherent disadvantage. It cannot completely contain the spatial information, and the spatial information of the top and bottom is not available. The spatial information of spherical panorama is complete, but it is more difficult than that of cylindrical panorama, and the research reports are much less. On the one hand, the shooting scheme is cumbersome, requiring dozens or even dozens of photos to be taken; on the other hand, so many photo errors will accumulate so much that automatic stitching cannot be performed in the end, so it is a semi-automatic method. Manual adjustments are often required. Therefore, how to provide a spherical panorama video stitching method with a small number of shots, fully automatic, and high stitched image quality, so as to make panorama video shooting easier and improve the shooting quality, is a technical problem to be solved by those skilled in the art.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and system for splicing spherical panoramic videos, which can make the splicing of panoramic videos simple and efficient, and the image quality can be improved.

In order to solve the above-mentioned technical problems, the present invention provides a method for splicing spherical panoramic videos, the method comprising:

Acquire a video pair captured by a fisheye lens; wherein the video pair includes a first video with N frames of first images and a second video with N frames of second images; and N frames of first images and N frames of second images have a one-to-one correspondence;

The first image and the second image with the one-to-one correspondence are respectively used as image pairs to perform a splicing operation to form a spliced video; wherein, the process of the splicing operation includes:

determining the corner corresponding to the first image in the image pair;

Search for the experience area corresponding to each of the corner points in the second image in the image pair, and use the HSV algorithm to check the search result to determine the matching point corresponding to each of the corner points;

Calculate the transformation matrix of the image pair according to the selected corner points and the corresponding matching points;

Use the conversion matrix to determine the color value of each pixel in the non-overlapping area in the image pair, and perform pixel fusion in the overlapping area according to the weight coefficient to determine the color value of each pixel in the overlapping area;

All the color values are filled into the corresponding positions in the stitched video.

Optionally, determining the corner points corresponding to the first image in the image pair, including:

determining whether the first image in the image pair is the first frame image of the first video;

If it is the first frame of image, performing corner detection on the first frame of image to determine the corner corresponding to the first frame of image;

If it is not the first frame of image, determine whether the image pair is an image pair separated by a predetermined number of frames;

If it is not an image pair separated by a predetermined number of frames, the corner point of the first image in the image pair of the previous frame is used as the corner point corresponding to the first image in the image pair;

If the image pairs are separated by a predetermined number of frames, then corner detection is performed on the selected overlapping area of the first image in the image pair to obtain shadow corners, and it is determined that the number of shadow corners is the same as that in the overlapping area of the latitude and longitude images. Whether the difference in the number of corner points is greater than a threshold; if so, perform corner point detection on the first image in the image pair to determine the corner point corresponding to the first image in the image pair.

Optionally, the process of corner detection includes:

进行角点检测，得到预选角点M；Use the formula

Perform corner detection to obtain preselected corner M;

Calculate the response function value R of the preselected corner point M by using the formula R=det(M)-k(trace(M)) ² ;

The response function values R are arranged in the order from high to bottom, and the preselected corner points M corresponding to the response function values R of the previous predetermined number are selected as the corner points selected for corner detection;

Wherein, I is the first video, I _x and I _y are the gradient images obtained by filtering the image by the difference operator in the horizontal and vertical directions, w(x, y) is a two-dimensional Gaussian function, det is the matrix determinant, trace is the trace of the matrix, and k is a constant.

Optionally, searching the experience area corresponding to each of the corner points in the second image in the image pair, including:

Determine the corresponding theoretical positions of the corner points selected by the corner detection in the second image in the image pair, and expand a predetermined range outward with each of the theoretical positions as the center to search for corresponding matching points as an empirical area.

Optionally, use the HSV algorithm to check the search results to determine the matching points corresponding to each of the corner points, including:

计算所述经验区域中每一个像素点与所述经验区域对应的角点的加权颜色差总和E_x,y；Use the formula separately

Calculate the weighted color difference sum E _x,y of each pixel in the experience area and the corner corresponding to the experience area;

Select the pixel point corresponding to the minimum weighted color difference sum E _{x, y} as the matching point of the corner point;

Among them, E _{x, y} is the sum of the color difference of the second image at (x, y) for the corner points of the first image, H' _{s+u, t+v} and S' _{s+u, t+v} are the first Hue and saturation at (s+u,t+v) in the image, H″ _x+u,y+v and S″ _x+u,y+v are the second image at (x+u,y +v) hue and saturation, k _h and k _s are two coefficients, w(u, v) is the weight function of the window pixel position, u, v are the pixel positions in the window, and l is the square window size.

Optionally, calculate the transformation matrix of the image pair according to the selected corner points and the corresponding matching points, including:

From the corner points selected by the corner point detection, 5 corner points are selected as the selected corner points according to the distance threshold;

计算所述图像对的转换矩阵；According to the selected corner points and matching points, use the formula

calculating a transformation matrix for the pair of images;

Among them, a ₁₁ , a ₁₂ . . . a ₄₃ are 15 unknowns in total, and φ _L , θ _L , φ _R , and θ _R are the latitude and longitude of the first image and the second image after the shooting coordinate system is converted into a spherical coordinate system, respectively.

Optionally, the distance threshold _rt is specifically: _rt =k _t r ₀ , where r ₀ is the turning radius, and k _t is a coefficient.

Optionally, use the conversion matrix to determine the color value of each pixel in the non-overlapping area in the image pair, and perform pixel fusion in the overlapping area according to the weight coefficient to determine the color value of each pixel in the overlapping area, including:

Traverse the longitude and latitude in the transformation matrix, and determine the color value c ₁ of each position in the first image and the color value c ₂ of each position in the second image;

确定所述图像对拼接后的图像中各位置的颜色值；Use color values to determine formulas

Determine the color value of each position in the image after the image pair is spliced;

Wherein, c is the color value of each position in the image after the image pair is spliced, and k ₁ and k ₂ are weight coefficients.

The present invention also provides a splicing system for spherical panoramic video, the system comprising:

A video acquisition module for acquiring a video pair captured by a fisheye lens; wherein the video pair includes a first video with N frames of first images and a second video with N frames of second images; and N frames of first images There is a one-to-one correspondence with the N frames of the second image;

a splicing module, configured to perform a splicing operation with the first image and the second image having the one-to-one correspondence as an image pair, respectively, to form a spliced video;

A stitching operation module, used to determine the corner points corresponding to the first image in the image pair; search for the experience area corresponding to each of the corner points in the second image in the image pair, and use the HSV algorithm to search results Carry out inspection to determine the matching point corresponding to each of the corner points; calculate the transformation matrix of the image pair according to the selected corner point and the corresponding matching point; use the transformation matrix to determine each pixel in the non-overlapping area of the image pair and perform pixel fusion in the overlapping area according to the weight coefficient to determine the color value of each pixel in the overlapping area; fill all the color values into the corresponding positions in the spliced video.

A method for splicing spherical panoramic videos provided by the present invention includes: acquiring video pairs captured by a fisheye lens; respectively performing a splicing operation on a first image and a second image having the one-to-one correspondence as an image pair to form splicing videos; wherein, the process of the splicing operation includes: determining the corner points corresponding to the first image in the image pair; searching for the experience area corresponding to each of the corner points in the second image in the image pair, And use the HSV algorithm to check the search result to determine the matching point corresponding to each of the corner points; calculate the transformation matrix of the image pair according to the selected corner point and the corresponding matching point; use the transformation matrix to determine the image pair. The color value of each pixel in the non-overlapping area is determined, and the pixel fusion of the overlapping area is performed according to the weight coefficient to determine the color value of each pixel in the overlapping area; all the color values are filled into the corresponding positions in the spliced video.

It can be seen that the automatic stitching of panoramic videos is realized by performing appropriate coordinate transformation on the video pairs captured by the fisheye lens. Among them, corner detection, empirical region search and HSV color space matching are required in the process of solving the transformation matrix to realize automatic detection of matching points; this method has a small number of shots, fast stitching, and no manual adjustment and intervention after the stitching process is started. The stitching effect of the generated video is good, the stitching of the middle seam is accurate, and the transition is natural. From the entire video playback, there is no image jitter and misalignment. The present invention also provides a splicing system for spherical panoramic video, which has the above beneficial effects, and will not be repeated here.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

1 is a flowchart of a method for splicing spherical panoramic videos according to an embodiment of the present invention;

2 is a schematic diagram of the calculation principle of latitude and longitude and turning radius in a fisheye image provided by an embodiment of the present invention;

3 is a schematic diagram of a shooting direction and a coordinate system of a spherical panoramic video provided by an embodiment of the present invention;

4 is a schematic diagram of the conversion of Cartesian coordinates to latitude and longitude provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of conversion between longitude and latitude and pixel coordinates of a panoramic unfolded image provided by an embodiment of the present invention;

6 is a schematic diagram of calculating a color fusion weight provided by an embodiment of the present invention;

7 is a schematic diagram of a splicing overlapping area provided by an embodiment of the present invention;

8 is a screenshot of a specific splicing video provided by an embodiment of the present invention;

9 is a schematic flowchart of a specific spherical panoramic video stitching method provided by an embodiment of the present invention;

FIG. 10 is a structural block diagram of a splicing system for spherical panoramic videos provided by an embodiment of the present invention.

Detailed ways

The core of the present invention is to provide a method and system for splicing spherical panoramic videos, which can make the splicing of panoramic videos simple and efficient, and the image quality can be improved.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1, which is a flowchart of a method for splicing spherical panoramic videos according to an embodiment of the present invention; the method may include:

S100. Acquire a video pair captured by a fisheye lens; wherein the video pair includes a first video having N frames of first images and a second video having N frames of second images; and N frames of first images and N frames of second images have a one-to-one correspondence.

Specifically, in this embodiment, each frame of image pair in the video pair captured by the fisheye lens is spliced, so as to complete the splicing of the entire video pair. That is, this embodiment does not limit the fisheye lens itself, as long as the video pairs that can be shot are spliced together to form a panoramic video. The first video and the second video here are only to distinguish the two videos in the video pair, and they are not limited, that is, any video in the video pair can be the first video, and the corresponding other video is the first video. Two videos. The corresponding first image and second image are only for distinguishing frame images in the two videos. This embodiment does not limit the value of N, which can be any integer greater than 0. The frames of the videos in the two video pairs are the same and have a one-to-one correspondence, that is, they correspond to the frames of the videos in the shooting itinerary. . For example, the first video has 5 frames of first images (A1, A2, A3, A4, A5), and the second video has 5 frames of second images (B1, B2, B3, B4, B5), then A1 and B1 have Image pairs with one-to-one correspondence, A2 and B2 are image pairs with one-to-one correspondence.

S110. Perform a splicing operation with the first image and the second image having a one-to-one correspondence as an image pair to form a spliced video.

Specifically, after the splicing operation is performed on each frame image pair in the video pair, the splicing work of the entire video pair is completed. That is, operate on each frame of the video and perform appropriate optimization processing, and finally generate a spliced video. This embodiment does not limit the processing form of performing the splicing operation on each pair of image pairs, which may be splicing each image pair at the same time, or may perform the splicing operation on each frame image pair in sequence according to the frame sequence. Users can choose according to the actual situation. The process of the corresponding splicing operation for each frame of image pair is as follows:

Among them, the process of the splicing operation includes:

Step 1. Determine the corner corresponding to the first image in the image pair;

Specifically, in order to obtain the subsequent transformation matrix, the image corners need to be determined. In this embodiment, the corner points of the first image are first determined, and then matching points corresponding to each corner point are determined through matching.

This embodiment does not limit the manner of determining the corner points of the first image in each frame of image pair, as long as the corner points of each first image can be determined. For example, the first image in each frame of image pair uses the method of corner detection to calculate the corresponding corners, or if the corners corresponding to the frames before the current frame are more accurate, the corners of the first image of the previous frame can be used. The corner point corresponding to the first image in the image pair of the current frame. Preferably, in order to save computing resources and speed up splicing; determining the corner corresponding to the first image in the image pair may include:

Determine whether the first image in the image pair is the first frame image of the first video;

If it is the first frame image, perform corner detection on the first frame image to determine the corner corresponding to the first frame image;

If it is not the first frame of image, then determine whether the image pair is an image pair separated by a predetermined number of frames;

If it is not an image pair separated by a predetermined number of frames, the corner point of the first image in the image pair of the previous frame is used as the corner point corresponding to the first image in the image pair;

If the image pairs are separated by a predetermined number of frames, corner detection is performed on the selected overlapping area of the first image in the image pair to obtain shadow corner points, and the difference between the number of shadow corner points and the corner points in the overlapping area of the latitude and longitude images is determined. Whether it is greater than the threshold; if so, perform corner detection on the first image in the image pair to determine the corner corresponding to the first image in the image pair; point as the corner corresponding to the first image in the image pair.

Specifically, the corner point detection can be performed only on the first image (ie, the first frame of image) in the image pair of the first frame of video by the method for determining the corner point. The corner point correction process may then be performed only on the first image in the image pair spaced apart by a predetermined number of frames. For example, corner correction is performed every 10 frames. Such a processing process can improve the computational efficiency while ensuring the accuracy of the corner points.

Further, this embodiment does not limit the size of the selected overlapping area, that is, the shadow corner points may be calculated in the entire overlapping area, or may be calculated in a part of the overlapping area. But the latter is more efficient because the retrieved area is smaller. This embodiment does not limit the selection manner of the selected overlap region.

进行角点检测，得到预选角点M；利用公式R＝det(M)-k(trace(M))²计算预选角点M的响应函数值R；将响应函数值R按照从高到底的顺序进行排列，并选取前预定数量的响应函数值R对应的预选角点M作为角点检测选定的角点；Optionally, the process of corner detection can be: using the formula

Perform corner detection to obtain a preselected corner M; use the formula R=det(M)-k(trace(M)) ² to calculate the response function value R of the preselected corner M; set the response function value R in the order from high to bottom Arrange, and select the preselected corner point M corresponding to the response function value R of the previous predetermined number as the corner point selected for the corner point detection;

Wherein, I is the first video, I _x and I _y are the gradient images obtained by filtering the image by the difference operator in the horizontal and vertical directions, w(x, y) is a two-dimensional Gaussian function, det is the matrix determinant, trace is the trace of the matrix, and k is a constant (its value can be 0.04-0.06). When the value of R is large, it is a corner; when R<0, it is an edge, and when |R| is small, it is a flat area. There may be many positions judged as corner points in the whole image, and the n with the largest R value (for example, n=30) can be selected for further screening. That is, the present embodiment does not limit the numerical value of the first predetermined number.

Every certain number of frames (for example, every 10 frames), corner detection is performed on the overlapping area. See Figure 7 for an illustration of the overlapping area. The transformation matrix changes due to slight camera shake and changes in the distance of the scene. At this time, it is necessary to re-designate the matching points and re-calculate the transformation matrix. It can be known by checking the stitching of the overlapping areas of the images. As shown in Figure 7, for a 2×180° shot, the overlapping area is shown by the shaded area, which is a narrow rectangle in the middle in the latitude and longitude images. If the image is not stitched well, there must be a lot of misalignment of lines in this rectangle, and corners will be formed at the misaligned place. Performing corner detection can detect them. Therefore, for a certain frame of image, it can be checked whether the number of corner points in the shadow area of the fisheye image (eg, the first fisheye image) is approximately equal to the number of corner points in the shadow area of the latitude and longitude image. If they are roughly equal, the stitching effect is good; if the number of shadow areas in the latitude and longitude images is much larger than that in the fisheye image, it means that the stitching effect has declined, and 5 sets of matching points need to be recalculated to recalculate the transformation matrix.

You don't need to do this every frame of the image, just check it every so often (e.g. every 10 frames). The inspected image area does not need to take all the overlapping areas. In the fisheye image, you only need to expand n (for example, n=5) pixels outward according to the turning radius, shrink n pixels inward, and take the circles included between them. In the latitude and longitude image, just move n pixels to the left and n pixels to the right along the center line (dotted line in the figure), and take the rectangular area included in between. In fact, the corners of the dislocation are all at the midline. Check the detected number of corner points in the overlap area of the fisheye image and the number of corner points in the overlap area of the latitude and longitude images. If the difference between the two numbers is large, the corner detection is performed again.

Step 2, in the second image in the image pair, search the experience area corresponding to each corner point, and use the HSV algorithm to check the search result to determine the matching point corresponding to each corner point;

Specifically, when shooting a panorama, if the nodes of the camera can be guaranteed to remain unchanged and the shooting angle is accurately known, any point in the first image can actually be calculated in the second image. For example, when the shooting mode is 2×180°, in the first image where the polar coordinates are (r′ _p , θ), the matching point position in the second image should be (2r ₀ -r′ _p ,π-θ' ), where r ₀ is the turning radius. However, due to the fact that the camera node cannot strictly ensure the invariance, the error of the shooting angle, etc., the position of the matching point will be offset. And due to the vibration during the panorama video shooting and the dramatic changes in the distance of the scene, the matching point needs to be readjusted. Therefore, the empirical regions corresponding to the corner points in the second image of the image pair are searched. This embodiment does not limit the determination form of a specific experience area (which may also be referred to as an experience location), nor does it limit the size of the specific experience area. For example, this method can be used: take the theoretical position of the matching point as the initial empirical position, and use this as the center to specify a range, for example, search within a rectangular range of l×l (the value of l can be 0.1-0.3 times r ₀ ), Find the true match point location. When splicing a new frame of the video, when it is found that the position of the matching point has shifted, take the original matching position as the empirical position, take it as the center, and then search again within the 1×1 rectangular range. The method of determining the search range based on the empirical position does not need to search the entire image, which greatly reduces the search range and greatly saves the search time. Alternatively, determine the corresponding theoretical positions of the corners selected by the corner detection in the second image in the image pair, and expand a predetermined range outward with each theoretical position as the center to search for the corresponding matching points as the empirical area.

The process of determining the matching point based on HSV search for the second image can be: for two images, given a small window, the center of the window of the first image is designated as the corner point being checked, and the center of the window of the second image is equal to l× All pixels within the rectangular range of l are tried to traverse, and the sum of the weighted color difference between each position and the first image window is calculated.

For color comparison and matching, it is more appropriate to use the HSV color model than the RGB color model. Therefore, the first image and the second image need to be converted into HSV color representation. The sum of the weighted color differences for the window is calculated as:

计算经验区域中每一个像素点与经验区域对应的角点的加权颜色差总和E_x,y；using the formula

Calculate the weighted color difference sum E _x,y of the corners corresponding to each pixel in the experience area and the experience area;

Select the pixel point corresponding to the minimum weighted color difference sum E _{x, y} as the matching point of the corner point;

Among them, E _{x, y} is the sum of the color difference of the second image at (x, y) for the corner points of the first image, H' _{s+u, t+v} and S' _{s+u, t+v} are the first Hue and saturation at (s+u,t+v) in the image, H″ _x+u,y+v and S″ _x+u,y+v are the second image at (x+u,y The hue and saturation at + _v ), kh and _ks are two coefficients, which can set the importance of hue and saturation to the calculation result. It is desirable that _kh = 0.8 and _ks = 0.2, ie hue is 4 times more important than saturation. w(u, v) is the weight function of the pixel position of the window, which can be a two-dimensional Gaussian function, u, v are the positions of the pixels in the window, respectively, and l is the size of the square window.

Step 3. Calculate the transformation matrix of the image pair according to the selected corner point and the corresponding matching point;

Specifically, this embodiment does not limit the number of selected corner points and corresponding matching points, but in order to improve calculation efficiency, since there are 15 unknowns in the transformation matrix, at least 5 pairs of matching points can be calculated. Preferably, calculating the transformation matrix of the image pair according to the selected corner points and the corresponding matching points may include:

Select 5 corner points from the corner points selected by the corner point detection according to the distance threshold as the selected corner points;

计算选定的角点以及对应的匹配点计算图像对的转换矩阵；Use the formula

Calculate the selected corner points and the corresponding matching points to calculate the transformation matrix of the image pair;

Among them, a ₁₁ , a ₁₂ . . . a ₄₃ are 15 unknowns in total, and φ _L , θ _L , φ _R , and θ _R are the latitude and longitude of the first image and the second image after the shooting coordinate system is converted into a spherical coordinate system, respectively.

Specifically, since only 5 points are really needed. These 5 pairs of matching points not only hope to be corner points with distinct color changes, but also hope that the distance between them is relatively large, otherwise there will be some errors in the matrix solved later. Therefore, it is necessary to select 5 corner points according to the distance from the above n corner points. Use the distance threshold for selection. The distance threshold _rt is specifically: _rt =k _t r ₀ ; wherein, r ₀ is the turning radius, and k _t is a coefficient, which may be 0.3-0.6. Traverse the n corners, check the mutual distance, until you find 5 corners whose mutual distance is greater than _rt .

Wherein, the turning radius is defined as: in the coordinate system of the first image, the corresponding radius when the viewing angle of the camera is 180°. It can be proved that when the two images are taken with the camera rotated by 180°, the turning radius is equal to the average value of the distances from the corresponding matching points of the two fisheye images image1 and image2 (please refer to Figure 2) to their origin, namely:

Among them, r' _p is the distance from the image center to the pixel point P' in the image 1, and x' _p and y' _p are the rectangular coordinates of the pixel point P' relative to the image center. r" _p is the distance from the image center to the pixel point P" in image 2, and x" _p and y" _p are the Cartesian coordinates of the pixel point P" relative to the image center.

是求解全景图像变换中最重要的方程组。有a₁₁、a₁₂…a₄₃共为15个未知数，因此只需有5对匹配点就可以把矩阵元素都计算出来。可以用列主元高斯消去法等方法来解此方程组。式中φ_L、θ_L、φ_R、θ_R分别是第一图像和第二图像的拍摄坐标系转换成球坐标系后的经纬度。要理解和计算他们，需要理解下图的坐标系和转换计算方法：due to the equation

It is the most important system of equations in solving panoramic image transformation. There are a ₁₁ , a ₁₂ ... a ₄₃ for a total of 15 unknowns, so only 5 pairs of matching points can be used to calculate the matrix elements. This system of equations can be solved by methods such as column pivoting Gaussian elimination. In the formula, φ _L , θ _L , φ _R , and θ _R are the latitude and longitude of the first image and the second image after the shooting coordinate system is converted into a spherical coordinate system, respectively. To understand and calculate them, you need to understand the coordinate system and transformation calculation method of the following figure:

The formation of the panorama can actually be imagined as follows: a person sits in a huge glass sphere and looks at the surrounding scene, one eye of the person is exactly at the center of the sphere, and the light from the scene hitting the eye must pass through a certain point on the glass sphere and imagine that it is there. image at the point. In the end, all the scenes are imaged on the glass sphere, and this glass sphere with the image is the panorama of the scene. A fisheye photo is an isometric plane image, and a panoramic image is a latitude and longitude image. To turn the fisheye image into a panoramic image, it can be realized by the method of coordinate system transformation.

Figure 3 shows the situation of two fisheye images. The arrow shoot _L represents the shooting direction of the camera to shoot the first video, and the arrow shoot _R represents the shooting direction of the camera to shoot the second video. There are five coordinate systems involved: (1) The world coordinate system XYZ, which is the coordinate system of the three-dimensional world, and the latitude and longitude of the panoramic image will also be converted from this coordinate system. (2) The shooting coordinate system oxlylzl of the left hemisphere part. The shooting direction shoot _L of the first image is opposite to its zl axis direction. (3) The image coordinate system o'x'y' of the left photo image (image1). This is the plane coordinate system. This coordinate system is used when viewing the image (image1). (4) The shooting coordinate system oxryrzr of the right hemisphere part. Its shooting direction shoot _R is opposite to its zr axis. (5) The image coordinate system o"x"y" of the photo image (image2) on the right. This is a plane coordinate system. This coordinate system is used when observing the image (image2). In addition to these five coordinate systems, there is also a global coordinate system. The unwrapped image coordinate system of the coordinate sphere is expanded according to latitude and longitude. Besides these five coordinate systems, there is also an unwrapped image coordinate system expanded by the global coordinate sphere according to latitude and longitude.

The part of the right hemisphere, although mostly not imaged in the first image, can be represented in the scalar system oxlylzl. The coordinate system oxryrzr can be rotated to obtain the coordinate system oxlylzl. The coordinates in oxryrzr of each point on the right hemisphere can be read out in the second image. The same point can be multiplied by a certain matrix and can be transformed into oxlylzl, denoted as:

其中，(x_L,y_L,z_L)表示坐标系OX_LY_LZ_L中的坐标，(x_R,y_R,z_R)表示坐标系OX_RY_RZ_R的。

Among them, (x _L , y _L , z _L ) represents the coordinates in the coordinate system OX _L Y _L Z _L , and (x _R , y _R , z _R ) represents the coordinate system OX _R Y _R Z _R.

和直角坐标(x,y,z)存在以下关系：On the other hand, consider the conversion between spherical coordinate system and rectangular coordinate system, as shown in Figure 4. The radius of the sphere in Figure 3 and Figure 4 can be arbitrary and does not affect the imaging, and can be assumed to be 1. So latitude and longitude

and Cartesian coordinates (x, y, z) have the following relationship:

and the reverse relationship:

和

就可以改写成

Depend on

and

can be rewritten as

Step 4. Determine the color value of each pixel in the non-overlapping area in the image pair by using the conversion matrix, and perform pixel fusion in the overlapping area according to the weight coefficient to determine the color value of each pixel in the overlapping area;

查找两个鱼眼视频对应帧图像对应位置的颜色值，然后计算融合颜色，填入生成视频的对应帧图像中。方法如下：Specifically, the panoramic image is an image with an aspect ratio of 2:1, in which the width direction represents longitude and the height direction represents latitude. Therefore, to generate one frame of panoramic video, it is necessary to traverse 0-360° longitude and -90°-90° latitude. So iterate over longitude theta and latitude

Find the color value of the corresponding position of the corresponding frame image of the two fisheye videos, then calculate the fusion color, and fill in the corresponding frame image of the generated video. Methods as below:

The final stitched panoramic image is actually an image obtained by expanding the spherical surface in the global coordinate system according to latitude and longitude. The longitude and latitude can be converted from the pixel coordinates in the expanded image, and the global coordinates can be converted from the longitude and latitude. For example, referring to Figure 5, the latitude and longitude of the pixel in row i and column j is calculated as follows:

可换算出全局坐标系oxyz中的直角坐标(x,y,z)。Then use the latitude and longitude by the formula

The Cartesian coordinates (x, y, z) in the global coordinate system oxyz can be converted.

For that pixel, find its corresponding location in the left shot coordinate system. To this end, the relationship between the shooting coordinate system and the global coordinate system needs to be considered first: consider the transformation of the coordinate system oxlylzl to oxyz in Figure 3. The solved matrix a is a matrix transformed from the coordinate system oxryrzr and to the oxlylzl coordinate system. The global coordinate system oxyz is different from these two coordinate systems (see Figure 2). And oxlylzl can be transformed into oxyz by the following transformation: rotate 180° around the yl axis, and then rotate 90° around the new xl axis. which is:

Transform the above formula to get:

The position of (x, y, z) in the left shooting coordinate system can be calculated from the above formula. At the same time, it corresponds to the right shooting coordinate system:

x'＝r'cos θ'，y'＝r'sin θ'就可算出照片图像中的像素坐标(x’,y’)和(x”,y”)，就可以查到像素颜色了。式中r₀为转向半径，(x’,y’)为鱼眼第一图像的图像坐标，把(x’,y’)和

换成(x”,y”)和

可算得鱼眼第二图像的图像坐标(x”,y”)。各参数和关系参见图2。-1 is the matrix inversion operation. The shooting coordinate system is the same as the latitude and longitude of the corresponding photo coordinate system, so it can be used according to

x'=r'cos θ', y'=r'sin θ', the pixel coordinates (x', y') and (x", y") in the photo image can be calculated, and the pixel color can be checked. In the formula, r ₀ is the turning radius, (x', y') is the image coordinate of the first fisheye image, and the (x', y') and

replaced by (x", y") and

The image coordinates (x", y") of the second fisheye image can be calculated. See Figure 2 for parameters and relationships.

确定图像对拼接后的图像中各位置的颜色值；其中，c为图像对拼接后的图像中各位置的颜色值，k₁、k₂是权值系数。At least one of (x', y') of the first fisheye image and (x", y") of the second fisheye image is colored. For the points examined in the unfolded images, sometimes only the first image is imaged, sometimes only the second image, and sometimes both images. Use color values to determine formulas

Determine the color value of each position in the image after the image pair is spliced; wherein, c is the color value of each position in the image after the image pair is spliced, and k ₁ and k ₂ are weight coefficients.

where k ₁ +k ₂ =1. It can be taken as follows: see Figure ₆ , P1 and P2 are matching point pairs, _P2e is the intersection of _{P2 at the fisheye boundary along the radial direction, P2e1 is the corresponding point of P2e in the first} fisheye image ( Calculated by multiplying P _2e by the transformation matrix). Similar to P _2e , P _1e is the intersection of P ₁ at the fisheye boundary in the radial direction. k ₁ and k ₂ can be calculated according to the distance relationship between P ₁ and P _1e and P _2e1 :

Step 5. Fill all the color values to the corresponding positions in the spliced video.

At this point, a panoramic video is generated. See Figure 8 for an example of a screenshot of a generated panoramic video case played using a normal video player. Use the dedicated panoramic video player for 360-degree VR.

The above solution is described below through a specific example, and two video materials with a field of view greater than 180 degrees are shot by using a camera and a fisheye lens for the scene. Use the following steps to process and generate a new video. Please refer to Figure 9.

1. Start Start doing the following for each frame of the two videos.

2. If it is the first frame, or jump from 11 to this step, and operate from 3 down, otherwise directly operate from 9 down.

3. Perform corner detection on image 1 (ie, the first image).

4. Search image 2 (ie, the second image) based on the empirical range.

5. Determine matching points for image 2 based on HSV search.

6. Select 5 pairs from the above n pairs of matching points.

7. If it is the first frame, calculate the turning radius. If it is not the first frame, skip this step. The turning radius calculated in the first frame is used for the calculation in subsequent frames.

8. Solve the system of equations to calculate the transformation matrix.

查找两个鱼眼视频对应帧图像对应位置的颜色值，然后计算融合颜色，填入生成视频的对应帧图像中。9. A panoramic image is an image with an aspect ratio of 2:1, where the width direction represents longitude and the height direction represents latitude. Therefore, to generate one frame of panoramic video, it is necessary to traverse 0-360° longitude and -90°-90° latitude. So iterate over longitude theta and latitude

Find the color value of the corresponding position of the corresponding frame image of the two fisheye videos, then calculate the fusion color, and fill in the corresponding frame image of the generated video.

10. Every certain number of frames (for example, every 10 frames), perform corner detection on the overlapping area.

11. Check the number of corner points in the overlap area of the fisheye image detected by 10 and the number of corner points in the overlap area of the latitude and longitude images. If the difference between the two numbers is large, return 2, otherwise, jump to 1 until all frames are processed.

Based on the above technical solutions, the method for splicing spherical panoramic videos provided by the embodiments of the present invention is suitable for the automatic splicing method of spherical panoramic videos in a 2×180° shooting mode; The images in the video are detected at the corners, and then the corresponding empirical positions in the corresponding frame images of the second video are searched and compared, and then each matching point in the video 2 is obtained by the HSV test and comparison method, and 5 pairs of matching points are used. , solve the equation to calculate the transformation matrix of the video image. The color value of each corresponding pixel is copied into the generated video using the transformation matrix, and the pixel fusion of the overlapping area is performed according to the coefficient. Each frame of the video is manipulated and optimized appropriately, resulting in a stitched video. For panorama video splicing, the method provided in this embodiment has a small number of shots, fast splicing speed, and does not require manual adjustment and intervention after the splicing process is started. The stitching effect of the generated video is good, the stitching of the middle seam is accurate, and the transition is natural. From the entire video playback, there is no image jitter and misalignment.

The following describes the splicing system for spherical panoramic videos provided by the embodiments of the present invention. The splicing system for spherical panoramic videos described below and the splicing method for spherical panoramic videos described above may refer to each other correspondingly.

Please refer to FIG. 10. FIG. 10 is a structural block diagram of a spherical panoramic video splicing system provided by an embodiment of the present invention, and the system may include:

A video acquisition module 100 is configured to acquire a video pair captured by a fisheye lens; wherein the video pair includes a first video with N frames of first images and a second video with N frames of second images; and the N frames of first images and N frames of second images have a one-to-one correspondence;

The splicing module 200 is used to perform a splicing operation with the first image and the second image having a one-to-one correspondence as an image pair to form a spliced video;

The stitching operation module 300 is used to determine the corner points corresponding to the first image in the image pair; in the second image in the image pair, the experience area corresponding to each corner point is searched, and the HSV algorithm is used to check the search results to determine each corner The matching point corresponding to the point; calculate the transformation matrix of the image pair according to the selected corner point and the corresponding matching point; use the transformation matrix to determine the color value of each pixel in the non-overlapping area of the image pair, and perform the overlapping area according to the weight coefficient Pixel fusion determines the color value of each pixel in the overlapping area; fills the entire color value into the corresponding position in the stitched video.

The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the possibilities of hardware and software. Interchangeability, the above description has generally described the components and steps of each example in terms of functionality. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The steps of a method or algorithm described in conjunction with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

The method and system for splicing spherical panoramic videos provided by the present invention are described above in detail. The principles and implementations of the present invention are described herein by using specific examples, and the descriptions of the above embodiments are only used to help understand the method and the core idea of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (6)

1. A spherical panoramic video splicing method is characterized by comprising the following steps:

acquiring a video pair shot by the fisheye lens; wherein the video pair comprises a first video having N frames of a first image and a second video having N frames of a second image; the N frames of first images and the N frames of second images have one-to-one correspondence;

respectively taking the first image and the second image which have the one-to-one correspondence relationship as image pairs to execute splicing operation to form a spliced video; wherein the process of the splicing operation comprises:

determining an angular point corresponding to a first image in the image pair;

searching an experience area corresponding to each angle point in a second image in the image pair, and checking a search result by using an HSV (hue, saturation and value) algorithm to determine a matching point corresponding to each angle point;

calculating a transformation matrix of the image pair according to the selected corner points and the corresponding matching points;

determining the color value of each pixel in the non-overlapped region in the image pair by using the conversion matrix, and performing pixel fusion of the overlapped region according to the weight coefficient to determine the color value of each pixel in the overlapped region;

filling all the color values into corresponding positions in the spliced video;

the determining the corner point corresponding to the first image in the image pair includes:

judging whether a first image in the image pair is a first frame image of the first video;

if the image is a first frame image, performing corner detection on the first frame image, and determining a corner corresponding to the first frame image;

if the image pair is not the first frame image, judging whether the image pair is the image pair separated by the preset frame number; wherein the interval is an interval between the image pair and a previous detection image pair;

if the image pair is not the image pair separated by the preset frame number, taking the corner point of the first image in the previous frame image pair as the corner point corresponding to the first image in the image pair;

if the images are spaced by a preset frame number, carrying out corner detection on a selected overlapping area of a first image in the image pair to obtain shadow corners, and judging whether the difference between the number of the shadow corners and the number of corners in a longitude and latitude image overlapping area corresponding to the images shot by the fisheye lens is larger than a threshold value or not; if so, performing corner detection on the first image in the image pair, and determining a corner corresponding to the first image in the image pair; wherein the shadow corner points are corner points in the overlapping region;

utilizing HSV algorithm to check the search result and determine the matching point corresponding to each angle point, including:

respectively using formulas

Calculating the weighted color difference sum E of each pixel point in the empirical region and the corner point corresponding to the empirical region_x,y；

Selecting the smallest weighted color difference sum E_x,yThe corresponding pixel points are used as matching points of the angular points;

wherein E is_x,yIs the sum of color differences, H ', of the second image at (x, y) for corner points in the first image'_s+u,t+vAnd S'_s+u,t+vIs the hue and saturation, H ″, at (s + u, t + v) in the first image_x+u,y+vAnd S ″)_x+u,y+vIs the hue and saturation at (x + u, y + v), k, in the second image_hAnd k_sFor two coefficients, w (u, v) is a weighting function of the pixel position of the window, u, v are the positions of the pixels within the window, respectively, and l is the size of the square window.

2. The method according to claim 1, wherein the corner point detection process comprises:

using formulas

Carrying out angular point detection to obtain a preselected angular point M;

using the formula R ═ det (M) -k (trace (M))²Calculating a response function value R of the preselected angular point M;

arranging the response function values R in a sequence from top to bottom, and selecting preselected angular points M corresponding to a preset number of response function values R as angular points selected by angular point detection;

wherein I is a first video, I_x、I_yThe gradient image is obtained by filtering the image through a difference operator in the horizontal direction and the vertical direction, w (x, y) is a two-dimensional Gaussian function, det is a matrix determinant, trace is a trace of the matrix, and k is a constant.

3. The method of claim 2, wherein searching for an empirical region in the second image of the pair of images corresponding to each of the corner points comprises:

and determining the corresponding theoretical position of the corner point selected by the corner point detection in the second image in the image pair, and taking each theoretical position as a center to expand a preset range outwards to serve as an empirical region to search for a corresponding matching point.

4. The method of claim 3, wherein computing a transformation matrix for the image pair from the selected corner points and corresponding matching points comprises:

selecting 5 corner points from the corner points detected and selected according to a distance threshold value as selected corner points; wherein the distances between the selected 5 corner points are all larger than the distance threshold;

according to the selected corner points and the corresponding matching points, using a formula

Computing a transformation matrix for the pair of images;

wherein, a₁₁、a₁₂…a₄₃A total of 15 unknowns, phi_LA point in the first image, which is a pair of matching points, whose imaging coordinate system is converted into a spherical coordinate system, is referred to as a calculated latitude θ_LThen the calculated longitude after the shooting coordinate system is converted into a spherical coordinate system is referred to; phi is a_RThe calculated latitude of the point of the second image in the pair of matching points after the shooting coordinate system is converted into the spherical coordinate system; theta_RThen the calculated longitude after the shooting coordinate system is converted into a spherical coordinate system is referred to; the distance threshold value r_tThe method specifically comprises the following steps: r is_t＝k_tr₀(ii) a Wherein r is₀To the turning radius, k_tIs a coefficient.

5. The method of claim 4, wherein determining the color value of each pixel in the non-overlapping region of the image pair using the transformation matrix and performing pixel fusion of the overlapping region according to the weight coefficients to determine the color value of each pixel in the overlapping region comprises:

go throughLongitude and latitude in the conversion matrix, and determining color value c of each position in the first image₁And color values c of locations in the second image₂；

Determining a formula using color values

Determining color values of all positions in the images after the image pair splicing;

wherein c is a color value of each position in the image after the image pair splicing, k₁、k₂Are the weight coefficients.

6. A system for stitching spherical panoramic video, the system comprising:

the video acquisition module is used for acquiring a video pair shot by the fisheye lens; wherein the video pair comprises a first video having N frames of a first image and a second video having N frames of a second image; the N frames of first images and the N frames of second images have one-to-one correspondence;

the splicing module is used for respectively taking the first image and the second image which have the one-to-one correspondence relationship as image pairs to execute splicing operation to form a spliced video;

the splicing operation module is used for determining an angular point corresponding to a first image in the image pair; searching an experience area corresponding to each angle point in a second image in the image pair, and checking a search result by using an HSV (hue, saturation and value) algorithm to determine a matching point corresponding to each angle point; calculating a transformation matrix of the image pair according to the selected corner points and the corresponding matching points; determining the color value of each pixel in the non-overlapped region in the image pair by using the conversion matrix, and performing pixel fusion of the overlapped region according to the weight coefficient to determine the color value of each pixel in the overlapped region; filling all the color values into corresponding positions in the spliced video; wherein the determining the corner point corresponding to the first image in the image pair comprises: judging whether a first image in the image pair is a first frame image of the first video; if the image is the first frame image, the first frame image is processedDetecting corners of the image, and determining corners corresponding to the first frame of image; if the image pair is not the first frame image, judging whether the image pair is the image pair separated by the preset frame number; wherein the interval is an interval between the image pair and a previous detection image pair; if the image pair is not the image pair separated by the preset frame number, taking the corner point of the first image in the previous frame image pair as the corner point corresponding to the first image in the image pair; if the images are spaced by a preset frame number, carrying out corner detection on a selected overlapping area of a first image in the image pair to obtain shadow corners, and judging whether the difference between the number of the shadow corners and the number of corners in a longitude and latitude image overlapping area corresponding to the images shot by the fisheye lens is larger than a threshold value or not; wherein the shadow corner points are corner points in the overlapping region; if so, performing corner detection on the first image in the image pair, and determining a corner corresponding to the first image in the image pair; utilizing HSV algorithm to check the search result and determine the matching point corresponding to each angle point, including: respectively using formulas

Calculating the weighted color difference sum E of each pixel point in the empirical region and the corner point corresponding to the empirical region_x,y(ii) a Selecting the smallest weighted color difference sum E_x,yThe corresponding pixel points are used as matching points of the angular points; wherein E is_x,yIs the sum of color differences, H ', of the second image at (x, y) for corner points in the first image'_s+u,t+vAnd S'_s+u,t+vIs the hue and saturation, H ″, at (s + u, t + v) in the first image_x+u,y+vAnd S ″)_x+u,y+vIs the hue and saturation at (x + u, y + v), k, in the second image_hAnd k_sFor two coefficients, w (u, v) is a weighting function of the pixel position of the window, u, v are the positions of the pixels within the window, respectively, and l is the size of the square window.

Images