KR102587298B1 - Real-time omnidirectional stereo matching method using multi-view fisheye lenses and system therefore - Google Patents

Abstract

A real-time omnidirectional stereo matching method and system using multi-view fisheye lenses are disclosed. The real-time omnidirectional stereo matching method according to an embodiment of the present invention includes a first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to photograph opposite directions, and a second pair of fisheye cameras arranged to photograph opposite directions. A real-time omnidirectional stereo matching method in a camera system comprising a second pair of fisheye cameras including three fisheye cameras and a fourth fisheye camera, wherein the first pair of fisheye cameras and the second pair of fisheye cameras are arranged vertically. , receiving fisheye images of a subject captured through the first to fourth fisheye cameras; selecting one of the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates; generating a distance map for all pixels using the reference fisheye image and the fisheye image of the selected fisheye camera; and performing real-time stereo matching on the received fisheye images using the generated distance map.

Description

Real-time omnidirectional stereo matching method and system using multi-view fisheye lenses {REAL-TIME OMNIDIRECTIONAL STEREO MATCHING METHOD USING MULTI-VIEW FISHEYE LENSES AND SYSTEM THEREFORE}

The present invention relates to a real-time omnidirectional stereo matching technology using multi-view fisheye lenses. More specifically, a real-time omnidirectional stereo matching method and system that can be performed directly on a multi-view fisheye image without additional spherical correction using multi-view fisheye lenses. It's about.

Efficient and accurate understanding of the appearance and structure of a 3D scene is an essential capability of computer vision used in many applications such as autonomous vehicles, robots, augmented/mixed reality, etc. Existing stereo cameras using regular lenses have a narrow field of view, which is not sufficient to capture the scene in all directions.

Using fewer fisheye lenses to reduce the number of cameras while covering all directions is a natural choice. However, when multiple fisheye lenses are used, the epipolar geometry is not maintained. This means that fast scan line stereo matching is not applicable to this non-pinhole camera. Therefore, omnidirectional camera configurations using multiple fisheye lenses suffer from the inevitable dilemma between performance and accuracy when calculating 360-degree panoramas and distances due to the following optical properties of fisheye lenses.

First, the existing pinhole camera model is not effective as a fisheye lens even after lens correction. Second, in stereo matching, it is necessary to use equirectangular or latitude-longitude projection as a digital representation of the fisheye image. It introduces severe image distortion into these representations, and thus additionally requires warp-aware correspondence search with spatial changes for accurate stereo matching, which has significant computational cost. Lastly, without a 360-degree dense distance map, multi-view fisheye images cannot be accurately merged into a 360-degree panoramic image, and a 360-degree dense distance map cannot be obtained by filtering. A chicken-and-egg problem arises when combining multi-view fisheye images with 360-degree RGB-D images with high accuracy.

Embodiments of the present invention provide a real-time omnidirectional stereo matching method and system that can be performed directly on a multi-view fisheye image without additional spherical correction using multi-view fisheye lenses.

A real-time omnidirectional stereo matching system according to an embodiment of the present invention includes a first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to capture images in opposite directions; and a second pair of fisheye cameras including a third fisheye camera and a fourth fisheye camera arranged to capture images in opposite directions, wherein the first pair of fisheye cameras and the second pair of fisheye cameras have different heights. It is placed vertically and is characterized by creating a distance map and projecting a fisheye image along 3D coordinates.

The first pair of fisheye cameras and the second pair of fisheye cameras may be arranged at different heights to create a partial overlap of the first to fourth fisheye cameras.

Furthermore, a real-time omnidirectional stereo matching system according to an embodiment of the present invention includes a receiving unit that receives fisheye images of a subject captured through the first to fourth fisheye cameras; a selection unit that selects one fisheye camera among the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates; a generator that generates a distance map for all pixels using the reference fisheye image and the fisheye image of the selected fisheye camera; and a matching unit that performs real-time stereo matching on the received fisheye images using the generated distance map.

The selection unit may select a fisheye camera having the highest distance discrimination power for each pixel of the reference fisheye image as any one of the fisheye cameras.

The selection unit may select, for each pixel of the reference fisheye image, a fisheye camera having the largest angle change between the first and last distance candidates among the distance candidates as the fisheye camera.

The generator may generate a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of any one selected fisheye camera.

The generator may generate a 360-degree color image in real time using the generated distance map and pixel values of the fisheye images.

The generator can generate a final 360-degree color image in real time by inpainting the missing area in the 360-degree color image using the background of the 360-degree color image.

The generator determines an inpainting direction by determining a foreground direction and a background direction in the 360-degree color image, calculates an inpainting kernel according to the determined inpainting direction and the occlusion direction of the missing area, and calculates the calculated inpainting kernel. By applying an inpainting kernel to the distance map, the missing area can be inpainted using the depth value for the background of the 360-degree color image.

The real-time omnidirectional stereo matching method according to an embodiment of the present invention includes a first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to photograph opposite directions, and a second pair of fisheye cameras arranged to photograph opposite directions. Real-time omnidirectional view in a camera system comprising a second pair of fisheye cameras including three fisheye cameras and a fourth fisheye camera, wherein the first pair of fisheye cameras and the second pair of fisheye cameras are vertically arranged to have different heights. A stereo matching method, comprising: receiving fisheye images of a subject captured through the first to fourth fisheye cameras; selecting one of the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates; generating a distance map for all pixels using the reference fisheye image and the fisheye image of the selected fisheye camera; and performing real-time stereo matching on the received fisheye images using the generated distance map.

In the step of selecting any one fisheye camera, for each pixel of the reference fisheye image, the fisheye camera having the highest distance discrimination ability may be selected as the one fisheye camera.

The step of selecting any one fisheye camera includes, for each pixel of the reference fisheye image, selecting the fisheye camera having the largest angle change between the first and last distance candidates among the distance candidates as the one fisheye camera. You can.

The step of generating a distance map for all pixels may generate a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of any one of the selected fisheye cameras. there is.

Furthermore, the real-time omnidirectional stereo matching method according to an embodiment of the present invention may further include generating a 360-degree color image in real time using the generated distance map and pixel values of the fisheye images.

In the step of generating the 360-degree color image in real time, the final 360-degree color image can be generated in real time by inpainting areas missing from the 360-degree color image using the background of the 360-degree color image.

The step of generating the 360-degree color image in real time includes determining an inpainting direction by determining a foreground direction and a background direction in the 360-degree color image, and inpainting according to the determined inpainting direction and the occlusion direction of the missing area. By calculating a painting kernel and applying the calculated inpainting kernel to the distance map, the missing area can be inpainted using the depth value of the background of the 360-degree color image.

A real-time omnidirectional stereo matching method according to another embodiment of the present invention includes a first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to shoot in opposite directions, and a first pair of fisheye cameras arranged to shoot in opposite directions. A real-time omnidirectional stereo matching method in a camera system including a second pair of fisheye cameras including a third fisheye camera and a fourth fisheye camera, wherein the subject is photographed through the first to fourth fisheye cameras. Receiving fisheye images; selecting one of the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates; generating a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of the selected fisheye camera; and performing real-time stereo matching on the received fisheye images using the generated distance map.

In the step of selecting any one fisheye camera, for each pixel of the reference fisheye image, the fisheye camera having the highest distance discrimination ability may be selected as the one fisheye camera.

The step of selecting any one fisheye camera includes, for each pixel of the reference fisheye image, selecting the fisheye camera having the largest angle change between the first and last distance candidates among the distance candidates as the one fisheye camera. You can.

According to embodiments of the present invention, it is possible to provide an efficient real-time sphere-sweeping stereo technique that can be implemented directly on multi-view fisheye images without additional spherical correction using equirectangular projection or latitude-longitude projection.

According to embodiments of the present invention, a real-time omnidirectional 360-degree RGB-D camera can be implemented, which can be applied to robotics, autonomous driving, etc. where real-time omnidirectional 360-degree RGB-D cameras can be applied.

Figure 1 shows an example diagram for explaining a real-time omnidirectional stereo matching system according to an embodiment of the present invention.
Figure 2 shows an operation flowchart for a real-time omnidirectional stereo matching method according to an embodiment of the present invention.
Figure 3 shows an example diagram for explaining projection on a spherical swept volume.
FIG. 4 shows an example of camera selection for a ray angle given in a reference frame and an example diagram of a map in which a camera is selected according to a pixel position.
Figure 5 shows an example of a multiscale filtering process, and shows an example of bidirectional filtering between scales.
Figure 6 shows an example diagram for explaining a filter kernel of different edge preservation parameters.
Figure 7 shows an example diagram for explaining the distance estimation algorithm.
Figure 8 shows an example diagram to explain the inpainting process.
Figure 9 shows the configuration of a real-time omnidirectional stereo matching system according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms, but the present embodiments only serve to ensure that the disclosure of the present invention is complete and are within the scope of common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used herein, “comprises” and/or “comprising” refers to the presence of one or more other components, steps, operations and/or elements. or does not rule out addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. The same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

Camera sets with fisheye lenses have been used to capture large fields of view. Conventional scan line stereo algorithms based on epipolar geometry cannot be directly applied to this non-pinhole camera setup due to the optical properties of the fisheye lens. Therefore, existing full 360-degree RGB-D imaging systems cannot yet achieve real-time performance.

Embodiments of the present invention provide an efficient real-time sphere-sweeping stereo technique that can be implemented directly on multi-view fisheye images without additional spherical correction using equirectangular projection or latitude-longitude projection.

The main contributions of the present invention are as follows. First, we introduce an adaptive spherical matching method that can directly evaluate stereo matching in the fisheye image region by considering the local discriminative ability of distance in each fisheye image. Second, the present invention introduces an adaptive spherical matching method that can directly evaluate stereo matching in the fisheye image region. Second, the present invention provides By providing a high-speed cross-scale bilateral cost volume filtering method of O(n) with optimal complexity, we enable 360-degree dense distance estimation in real time in all directions while preserving edges. Finally, the fisheye color and distance images are perfectly combined into a full 360-degree RGB-D image through high-speed inpainting of the dense distance map.

As shown in FIG. 1, the present invention can be implemented using a plurality of fisheye cameras, for example, four fisheye cameras. The prototype of the invention is capable of capturing full 360-degree RGB-D video, including color and distance at every pixel, at a resolution of 2 megapixels at 29 fps. As a result, it can be seen that the real-time algorithm of the present invention surpasses the existing omnidirectional stereo and learning-based 360-degree stereo algorithms in terms of accuracy and performance.

Figure 2 shows an operation flowchart for a real-time omnidirectional stereo matching method according to an embodiment of the present invention. The first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to shoot in opposite directions. and a second pair of fisheye cameras including a third fisheye camera and a fourth fisheye camera arranged to capture images in opposite directions, wherein the first pair of fisheye cameras and the second pair of fisheye cameras are vertically positioned to have different heights. It shows a real-time omnidirectional stereo matching method performed in a camera system that generates a distance map and projects a fisheye image along 3D coordinates.

Referring to FIG. 2, the real-time omnidirectional stereo matching method according to an embodiment of the present invention receives fisheye images of a subject captured through the first to fourth fisheye cameras, and sweep volume for preset distance candidates. For each pixel of a preset reference fisheye image among the received fisheye images using (sweep volume), one fisheye camera among the remaining fisheye cameras is selected (S210, S220).

Here, in step S220, for each pixel of the reference fisheye image, the fisheye camera with the highest distance discrimination ability may be selected as any one fisheye camera.

Here, in step S220, for each pixel of the reference fisheye image, the fisheye camera with the largest angle change between the first and last distance candidates among the distance candidates may be selected as any one fisheye camera.

When one fisheye camera is selected in step S220, a distance map for all pixels is generated using the reference fisheye image and the fisheye image of the selected fisheye camera, and the received fisheye image is used using the generated distance map. Real-time stereo matching is performed on them (S230, S240).

Here, step S230 may generate a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of any one selected fisheye camera.

Furthermore, the real-time omnidirectional stereo matching method according to an embodiment of the present invention can generate a 360-degree color image in real time using the generated distance map and pixel values of the fisheye images.

At this time, the process of generating a 360-degree color image in real time involves inpainting areas missing from the 360-degree color image using the background of the 360-degree color image, thereby generating the final 360-degree color image in real time. Specifically, the process of generating a 360-degree color image in real time determines the inpainting direction by determining the foreground direction and background direction in the 360-degree color image, and inpainting is performed according to the determined inpainting direction and the occlusion direction of the missing area. By calculating the kernel and applying the calculated inpainting kernel to the distance map, the missing area can be inpainted using the depth value of the background of the 360-degree color image.

The method of the present invention will be described with reference to FIGS. 3 to 8 as follows.

Binocular fisheye/360-degree stereo: Place two fisheye cameras or 360-degree cameras on a baseline and then use them to estimate depth within the stereo field of view, for example, distance in omni-directional stereo. Similar to traditional epipolar geometry, they apply spherical correction and block-wise cost matching along a large circle. However, the disparity of spherical stereo is proportional to the length of the arc, which is not linearly proportional to the reciprocal of the distance. Epipolar geometry for non-pinhole cameras is not maintained after lens calibration for fisheye or 360-degree cameras. Therefore, correspondence search using expensive spherical sweeping volumes is necessary, which cannot be applied to real-time distance estimates.

Equirectangular projection or latitude-longitude projection has been used to correct fisheye images before calculating stereo matching. This process is memory expensive and causes significant image distortion during projection, and geometric distortion prevents accurate correspondence retrieval in spherical stereo.

Additionally, distance along the baseline axis cannot be properly estimated in this binocular setting. This means that 360-degree panoramas and distance maps cannot be computed directly from this binocular stereo setup due to occlusion between the cameras and, most importantly, because there is no baseline available for alignment. The method of the present invention uses only four cameras, for example, multiple cameras with fisheye lenses, but is capable of capturing 360-degree RGB-D images in real time.

Monocular 360-degree stereo: Existing motion-based structural algorithms were applied to small 360-degree images. However, these methods assume a 360-degree camera moving in a static scene. When these methods are applied to scenes with dynamic objects, performance degrades rapidly. Additionally, the computational cost of this method is high, so it cannot be applied to real-time 360-degree RGB-D images. In addition, monocular stereo images were applied to 360-degree panoramas by learning an omnidirectional image prior. The learned prior can assist in matching correspondences in warped images. However, real-time learning-based methods do not yet exist due to model complexity. Additionally, there is currently no actual dataset of omnidirectional RGB-D images that can be used for deep learning. These methods were trained on hand-crafted 3D models and synthetic rendered images from 3D scanning. Because of the domain gap between real and rendered images, these models can often exhibit suboptimal performance with unseen real data.

Multi-view fisheye stereo: Multiple fisheye cameras can be combined with a spherical light source camera to capture 360-degree RGB-D images. The number of cameras varies from 4 to 20, and as the number of cameras increases, the quality of color and distance images improves significantly, but hardware and computational costs quickly increase. When combining multi-view fisheye stereo images, technical issues still exist, hindering the real-time performance of this setup. First, a complete 360-degree guided image is required to consider reprojection, occlusion, and visibility of distance values in the integrated omnidirectional image space. At this time, a 360-degree guide image cannot be obtained from multi-view input without a 360-degree dense street map. Simple warp and blending methods have been proposed without distance awareness. Because these techniques are designed for short baselines, they often suffer from stitching artifacts when disparity values change in overlapping regions. Second, the geometry of fisheye lens matching can result in multiple actual matches. This can be handled by devising a computationally intensive cost aggregation.

In contrast, the present invention uses a minimal number of fisheye cameras to cover the full 360 degree angle in real time, keeping building costs and computational requirements as low as possible.

High-speed spherical sweeping stereo

Hardware design: The present invention can be used for 360-degree RGB-D video, that is, four cameras with fisheye lenses, as shown in Figure 1. Each fisheye camera can have a field of view of more than 220 degrees. A pair of front and rear fisheye cameras are placed at the top, and another pair of fisheye cameras are at the bottom, but are positioned vertically so that each combination of adjacent stereo pairs has the same baseline.

Spherical Geometry: The invention is based on the classic binocular stereo model. Each pixel of the reference frame I _c0 captured by the reference camera c ₀ may represent the color of the detected ray at an angle (θ, ϕ) in polar coordinates. This corresponds to the point (θ, ϕ, d) in polar coordinates, where d is the distance. This can lead to a 3D position p in the space of c ₀ , and the 3D position p can be expressed as <Equation> below.

Assume that another camera c ₁ is placed at a rotation R _c1 and a position T _c1 with respect to the reference camera c ₀ . The camera captures images I _c0 and I _c1 . The position p in c ₁ space is am. If is the normalization vector of p _c1 , the pixel coordinates in I _c1 can be expressed as <Equation> below.

The pixel coordinate at c ₁ with the camera transformation R _c1 |T _c1 can be expressed as a pixel projection of the angle (θ, ϕ) of the distance d in the reference coordinate system, and can be expressed as <Equation> below.

Assuming a Lambertian surface in the scene, pixel I _c1 (θ _c1 , ϕ _c1 ) of camera c ₁ image is identical to pixel I _c0 (θ, ϕ) of reference camera. Pixels of images from other cameras can be expressed in a reference coordinate system in the same way.

Spherical sweep volume: Similar to multiview stereo with a standard camera model, the present invention builds a sweep volume for several distance candidates d ₀ , ..., d _N-1 . Instead of warping from I _c1 to I _c0 that follow homographies with the planar distance candidates, the present invention can use the mapping and reference frame described above or spherical distance candidates around a given point.

For each candidate, if the distance candidate is correct, we generate a warped version of I _c1 that can match I _c0 . Specifically, as shown in FIG. 3, the present invention finds the corresponding coordinates for all pixel coordinates (θ, ϕ) of I and all distance candidates d _i . after that Assign the value of the spherical sweep volume V as follows. In this volume, the correct distance candidate d _k is , which provides a direct cue for distance estimation. Matching quality can be evaluated through luminance difference or difference after image transformation such as gradient, census transformation, or feature extraction. For robust performance, cost aggregation or deep normalization is required when selecting optimal depth candidates. Multiple views may be used simultaneously.

Adaptive spherical matching

*The present invention can evaluate full depth candidates in all possible combinations of all possible overlapping regions along a baseline in a spherical swept volume, which can be done computationally. To achieve real-time performance, the present invention can utilize a camera selection method that provides the local best camera pair for search correspondence in a spherical sweeping volume with respect to a reference camera.

The present invention can select only the best camera among the three cameras (c ₁ , c ₂ , c ₃ ) for each pixel of the reference view. If multiple cameras have a field of view that can cover the pixels of the reference frame, the camera with the highest distance discrimination ability can be selected. This property can be explained by maximizing the difference between layers of the spherical sweep volume and being able to more clearly identify which candidate matches best.

In order to quantify this for a given pixel position (θ, ϕ) in the reference image I _c0 , the present invention applies distance candidates d ₀ and d _N-1 to the first and last layers of the volume, i.e. 0 and N-1. You can focus. Let p _ck ^<0> be a point in camera c _k space with reference coordinates (θ, ϕ, _di ), then the best camera c _k is the two 3D points p _ck ^<0> and p _ck given from these two distance candidates. This may be the camera that shows the largest angle change between ^<N-1> .

Specifically, if the angle between p _ck ^<0> and p _ck ^<N-1> is high, the sampled position at the selected camera for the sweeping volume changes significantly, which is suitable for distance estimation. The present invention can define a discrimination weight based on these considerations, and the discrimination weight can be expressed as <Equation> below.

here, may mean a normalized vector.

Using this evaluation, the optimal camera c* for each pixel can be selected as shown in Equation 1 below.

[Equation 1]

Figure 4 shows an example of camera selection for a given ray angle in a reference frame and an example diagram of a map in which a camera is selected according to pixel position. Figure 4a shows how the ray of c ₀ is reprojected from c ₁ and c ₂ . An example of evaluation is shown, and FIG. 4b shows an example in which a camera showing optimal distance determination for each pixel of c ₀ is selected.

As shown in Figure 4, the camera best suited for matching for this pixel is the camera that shows the maximum displacement for a given distance. Distance discrimination improves when small distance changes lead to high displacements, and as shown in Figure 4a, for a particular ray in c ₀ , c ₁ is better at matching cameras than c ₂ despite the baselines between the two pairs being similar. You can see that it is.

Efficient spherical cost aggregation

The present invention can calibrate four cameras using a dual spherical model and perform 220-degree distance estimation using two opposing top cameras as two references. For each pixel of each criterion, selective matching can be used to select the best camera. If I _cs is the image from the camera selected at pixel (θ, ϕ) and I _c0 is the reference frame, the matching cost of the ith distance candidate can be expressed as <Equation> below.

here, may mean a spherical sweeping volume from the selected camera to the reference camera. Additionally, the present invention can normalize each slice of the spherical cost volume using a fast filtering method.

High-speed cross-scale bidirectional filtering

When aggregating sparse distances to obtain a dense distance map, there are many available methods to smooth the cost in an edge-aware manner. The interactive grid-based methods that show the most impressive features are still computationally expensive to be applied to 3D cost volumes and often produce blocky artifacts even in post-domain transformation processing. A faster, more hardware-friendly version of the two-way solver has been designed that provides powerful performance for a single depth map, but is more hardware-specific and still computationally inefficient for real-time application to the entire cost volume. Another popular edge-aware filtering is the guided filter used with cost volume pyramids or multi-scale cost aggregation. While optimizing the complexity of O(n), high-speed performance cannot be achieved on GPUs due to computing overhead when calculating integrated images in a parallel environment. The present invention can introduce a high-speed between-scale bidirectional filtering method specifically designed for parallel computing environments to achieve 2-megapixel real-time RGB-D imaging at 29 fps on embedded machines with GPUs.

Edge-preserving downsampling: The first step in the filtering of the present invention is to reduce the input image without blurring to prevent edge bleeding and halos. To this end, the present invention filters adjacent pixels using a bidirectional weight before decimation. In the present invention, if I ₀ is defined as the original image and I _l is defined as the image after downsampling by two l times, the bidirectional weight can be expressed as <Equation 2> below.

[Equation 2]

Here, σ _I refers to the edge preservation parameter, and (x, y) may refer to pixel coordinates. At this time, the downsampling method can be expressed as <Equation 3> below.

[Equation 3]

here, may mean a normalization constant. in the pyramid Note that we can define the number of scale levels L.

Edge-preserving upsampling: Unlike existing edge-preserving upsampling methods that use high-resolution images as a guide, our method can achieve optimal complexity by using bidirectional weighting between downsampling and full-resolution images. At this time, the present invention intentionally does not use Gaussian spatial weighting to focus on efficiency, and can use partial constant output.

In addition to bidirectional weighting, the present invention blends scales using a Gaussian function of the current scale exponent. For each scale It can be defined as, where σ _s may mean a smoothness parameter. Weights at higher resolution scales naturally It can be.

Figure 5 shows an example of a multiscale filtering process, and shows an example of bidirectional filtering between scales.

As shown in Figure 5, the multiscale filtering process first aggregates the sparsity cost by downsampling with edge preservation using bidirectional weights between the guide center pixel and its neighboring pixels. Then upsample using minimum pixel support. The present invention can use guide weights calculated between guide centers and pixels to aggregate to lower scales.

After cost volume filtering, the optimal distance is determined through winner-take-all and sub-pixel accuracy can be achieved through quadratic fitting.

Filter Kernel: The final filter kernel obtained after the downsampling/upsampling process produces a smooth decay driven by σ _s when moving away from the center and does not cross edge boundaries as shown in Figure 6. Although each step of the algorithm performs only the minimum pixel support, bidirectional downsampling/upsampling filtering creates a kernel that includes the entire image. Guidance through bidirectional weighting is a higher order exponential configuration further away from a given pixel. This improves guidance between spatial pixels. In the present invention, σ _s may be set to 25, and σ _I may be set to 10.

Complexity: The number of operations follows the sum of a series of geometric images of 1/4 scale. Therefore, the asymptotic complexity is O(n) with n pixels, making the algorithm optimal. As for the number of levels, the size of the lowest level L must be larger than 1 pixel. Therefore, the present invention reduces the samples up to ln ₄ (n) times. Downsampling and upsampling must be performed sequentially in O(ln(n)) levels, but each downsampling and upsampling step can be fully parallelized.

Distance-aware panoramic stitching

As shown in Figure 7, the distance estimation algorithm in the present invention uses a reference frame for edge preservation and can avoid multiple actual matching. This approach improves accuracy but requires additional steps to merge the fisheye images. The present invention presents an efficient method that first synthesizes a distance map at a desired location, then projects the image along 3D coordinates, and finally merges the images through a blending process to give more weight to the least displaced pixel.

New view synthesis: The first step is to reproject the dense street map common to the two criteria to the selected location. To do this, find the location corresponding to each pixel (θ, ϕ), convert it to the selected location, and find the coordinates (θ _r , ϕ _r ) in the reprojected image. Here, the coordinates (θ _r , ϕ _r ) in the reprojected image can be obtained through <Equation> below.

Here, T ^* means the desired position with respect to the camera, may mean an estimated distance map. Forward warping operations inevitably lead to multiple pixels being mapped to the same target pixel in the original street app. That is, several couples (θ, ϕ) can be projected onto the same coordinates (θ _r , ϕ _r ). This ambiguity requires splatting to obtain the final value.

The present invention can merge possible pixels in an occlusion-aware manner. In particular, minimum distance splatting, that is, z-buffering, can be used, and the reprojected distance can be calculated as shown in Equation 4 below.

[Equation 4]

Directional inpainting: Some pixels in the target may have multiple counterparts in the original distance map, while some pixels may have none at all due to occlusion. Because missing areas may be obscured by foreground objects, they can be inpainted using the background. To do this, we first determine the background and foreground directions, which can be given as the derivative of the projection with respect to the distance. In practice, occlusion holes in a projected map occur because areas with different distances are not reprojected to the same location. Therefore, the inpainting direction can be defined as <Equation> below.

This inpainting direction leads to a direction diffusion kernel that can be used repeatedly. The present invention determines the kernel weight around each pixel according to its similarity to the inpainting direction. The kernel weight around each pixel can be expressed as <Equation> below.

Here, + means the positive part, may mean the index of 8 adjacent pixels. Since the dot product gives high weight to aligned vectors, this method can naturally generate a diffusion kernel using pixel values aligned in the inpainting direction, as shown in Figure 8. That is, the depth from the camera position in Figure 8a is projected onto the desired view in a depth-aware manner as shown in Figure 8b, and since occlusion creates a hole in the projected distance map, the inpainting kernel is calculated according to the occlusion direction as shown in Figure 8c. Finally, by applying the calculated inpainting kernel to the distance map, holes can be removed using the background depth value, as shown in Figure 8d.

Once you move to a viewpoint given a distance, color pixels are projected along the 3D coordinates given in the distance map to provide an RGB image to another location.

Blending: After projecting color images to a common location, two 220-degree images must be merged together to create a complete panorama stored in a standard equirectangular projection. To this end, the present invention provides blending weights corresponding to the amount of possible occlusion. For large pixels, changes in distance can significantly modify the image, resulting in larger occlusion areas, more distance-related distortion, and potential artifacts. Therefore, a blending weight that follows Gaussian can be defined for the length of this vector, which can be expressed as <Equation> below.

By setting b _ck (θ, ϕ) = 0, we can process the pixels that can be captured by a camera and estimate the derivative through finite differences over a range of distances.

As such, the method according to embodiments of the present invention can provide an efficient real-time sphere-sweeping stereo technique that can be implemented directly on multi-view fisheye images without additional spherical correction using equirectangular projection or latitude-longitude projection. You can.

Figure 9 shows the configuration of a real-time omnidirectional stereo matching system according to an embodiment of the present invention, and shows the conceptual configuration of a system that performs the method of Figures 1 to 8.

Referring to FIG. 9, the real-time omnidirectional stereo matching system 900 according to an embodiment of the present invention includes a first pair of fisheye cameras 910, a second pair of fisheye cameras 920, a receiver 930, and a selection unit. It includes (940), a generating unit (950), and a matching unit (960).

The first pair of fisheye cameras 910 includes a first fisheye camera 911 and a second fisheye camera 912 arranged to capture images in opposite directions.

The second pair of fisheye cameras 920 includes a third fisheye camera 921 and a fourth fisheye camera 922 arranged to capture images in opposite directions.

Here, the first pair of fisheye cameras and the second pair of fisheye cameras may be arranged vertically to have different heights. Then, you can create a distance map and project the fisheye image along 3D coordinates.

The receiver 930 receives fisheye images of subjects captured through the first to fourth fisheye cameras.

The selection unit 940 selects one fisheye camera among the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates. do.

Here, the selection unit 940 may select the fisheye camera with the highest distance discrimination power as any one fisheye camera for each pixel of the reference fisheye image.

Here, for each pixel of the reference fisheye image, the selection unit 940 may select the fisheye camera having the largest angle change between the first and last distance candidates among the distance candidates as any one fisheye camera.

The generator 950 generates a distance map for all pixels using the reference fisheye image and the fisheye image of one selected fisheye camera.

Here, the generator 950 may generate a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of any one selected fisheye camera.

Furthermore, the generator 950 can generate a 360-degree color image in real time using the generated distance map and pixel values of the fisheye images.

At this time, the generator 950 can generate the final 360-degree color image in real time by inpainting the missing area in the 360-degree color image using the background of the 360-degree color image. Specifically, the process of generating a 360-degree color image in real time determines the inpainting direction by determining the foreground direction and background direction in the 360-degree color image, and inpainting is performed according to the determined inpainting direction and the occlusion direction of the missing area. By calculating the kernel and applying the calculated inpainting kernel to the distance map, the missing area can be inpainted using the depth value of the background of the 360-degree color image.

The matching unit 960 performs real-time stereo matching on the received fisheye images using the generated distance map.

Although the description is omitted in the system of FIG. 9, each component of the system of FIG. 9 may include all the contents described in FIGS. 1 to 8, which is obvious to those skilled in the art. .

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (19)

A first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to take pictures in opposite directions; and
A second pair of fisheye cameras including a third fisheye camera and a fourth fisheye camera arranged to shoot in opposite directions.
Including,
The first pair of fisheye cameras and the second pair of fisheye cameras are arranged vertically to have different heights, generate a distance map and project the fisheye image along 3D coordinates,
The first fisheye camera and the second fisheye camera photograph opposite directions from the same first height, and the third fisheye camera and the fourth fisheye camera photograph opposite directions from the same second height,
A real-time omnidirectional stereo matching system, wherein the first height and the second height are different from each other.
According to paragraph 1,
The first pair of fisheye cameras and the second pair of fisheye cameras
A real-time omnidirectional stereo matching system characterized in that the two pairs are arranged at different heights to create partial overlap of the first to fourth fisheye cameras.
According to paragraph 1,
a receiving unit that receives fisheye images of a subject captured through the first to fourth fisheye cameras;
a selection unit that selects one fisheye camera among the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates;
a generator that generates a distance map for all pixels using the reference fisheye image and the fisheye image of the selected fisheye camera; and
A matching unit that performs real-time stereo matching on the received fisheye images using the generated distance map.
A real-time omnidirectional stereo matching system further comprising:
According to paragraph 3,
The selection section
A real-time omnidirectional stereo matching system, characterized in that, for each pixel of the reference fisheye image, a fisheye camera with the highest distance discrimination power is selected as one of the fisheye cameras.
According to paragraph 3,
The selection section
A real-time omnidirectional stereo matching system, characterized in that, for each pixel of the reference fisheye image, a fisheye camera having the largest angle change between a first and a last distance candidate among the distance candidates is selected as one of the fisheye cameras.
According to paragraph 3,
The generator
A real-time omnidirectional stereo matching system, characterized in that it generates a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of any one of the selected fisheye cameras.
According to paragraph 3,
The generator
A real-time omnidirectional stereo matching system that generates a 360-degree color image in real time using the generated distance map and pixel values of the fisheye images.
In clause 7,
The generator
A real-time omnidirectional stereo matching system that generates a final 360-degree color image in real time by inpainting areas missing from the 360-degree color image using the background of the 360-degree color image.
According to clause 8,
The generator
Determine the inpainting direction by determining the foreground direction and background direction in the 360-degree color image, calculate an inpainting kernel according to the determined inpainting direction and the occlusion direction of the missing area, and calculate the inpainting kernel. A real-time omnidirectional stereo matching system, characterized in that the missing area is inpainted using a depth value for the background of the 360-degree color image by applying to the distance map.
A first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to photograph opposite directions, and a second pair including a third fisheye camera and a fourth fisheye camera disposed to photograph opposite directions. In the real-time omnidirectional stereo matching method in a camera system including fisheye cameras, wherein the first pair of fisheye cameras and the second pair of fisheye cameras are vertically arranged to have different heights,
Receiving fisheye images of a subject captured through the first to fourth fisheye cameras;
selecting one of the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates;
generating a distance map for all pixels using the reference fisheye image and the fisheye image of the selected fisheye camera; and
Performing real-time stereo matching on the received fisheye images using the generated distance map.
Including,
The first fisheye camera and the second fisheye camera photograph opposite directions from the same first height, the third fisheye camera and the fourth fisheye camera photograph opposite directions from the same second height, and the third fisheye camera and the fourth fisheye camera photograph opposite directions from the same second height. A real-time omnidirectional stereo matching method, characterized in that the first height and the second height are different from each other.
According to clause 10,
The step of selecting any one fisheye camera is
A real-time omnidirectional stereo matching method, characterized in that for each pixel of the reference fisheye image, a fisheye camera with the highest distance discrimination ability is selected as one of the fisheye cameras.
According to clause 10,
The step of selecting any one fisheye camera is
A real-time omnidirectional stereo matching method, characterized in that, for each pixel of the reference fisheye image, a fisheye camera having the largest angle change between a first and a last distance candidate among the distance candidates is selected as one of the fisheye cameras.
According to clause 10,
The step of generating a distance map for all pixels is
A real-time omnidirectional stereo matching method, characterized in that generating a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of any one of the selected fisheye cameras.
According to clause 10,
Generating a 360-degree color image in real time using the generated distance map and pixel values of the fisheye images.
A real-time omnidirectional stereo matching method further comprising:
According to clause 14,
The step of generating the 360-degree color image in real time is
A real-time omnidirectional stereo matching method, characterized in that the final 360-degree color image is generated in real time by inpainting the missing area in the 360-degree color image using the background of the 360-degree color image.
According to clause 15,
The step of generating the 360-degree color image in real time is
Determine the inpainting direction by determining the foreground direction and background direction in the 360-degree color image, calculate an inpainting kernel according to the determined inpainting direction and the occlusion direction of the missing area, and calculate the inpainting kernel. A real-time omnidirectional stereo matching method characterized by inpainting the missing area using a depth value for the background of the 360-degree color image by applying to the distance map.
A first pair of fisheye cameras including a first fisheye camera and a second fisheye camera arranged to photograph opposite directions, and a second pair including a third fisheye camera and a fourth fisheye camera disposed to photograph opposite directions. In a real-time omnidirectional stereo matching method in a camera system including fisheye cameras,
Receiving fisheye images of a subject captured through the first to fourth fisheye cameras;
selecting one of the remaining fisheye cameras for each pixel of a preset reference fisheye image among the received fisheye images using a sweep volume for preset distance candidates;
generating a distance map for all pixels using bilateral cost volume filtering on the reference fisheye image and the fisheye image of the selected fisheye camera; and
Performing real-time stereo matching on the received fisheye images using the generated distance map.
A real-time omnidirectional stereo matching method including.
According to clause 17,
The step of selecting any one fisheye camera is
A real-time omnidirectional stereo matching method, characterized in that for each pixel of the reference fisheye image, a fisheye camera with the highest distance discrimination ability is selected as one of the fisheye cameras.
According to clause 17,
The step of selecting any one fisheye camera is
A real-time omnidirectional stereo matching method, characterized in that, for each pixel of the reference fisheye image, a fisheye camera having the largest angle change between a first and a last distance candidate among the distance candidates is selected as one of the fisheye cameras.