KR101901495B1 - Depth Image Estimation Method based on Multi-View Camera - Google Patents

Abstract

A multi-view camera based depth image estimation method is provided. The depth image estimating method includes: converting multi-view images to a virtual viewpoint; Estimating a depth image using the transformed multi-view images; And generating multi-view depth images using the estimated depth images.
Accordingly, the multi-view image-based depth estimation technique is a highly complex technique, but according to embodiments of the present invention, high-quality depth images using about 20 high resolution images can be acquired in real time.

Description

[0001] The present invention relates to a multi-view camera based depth image estimation method,

The present invention relates to a depth image estimating method, and more particularly, to a multi-view depth image estimating method using an image acquired from three or more multi view cameras.

The depth estimation method using multiple cameras can be roughly classified into the method using active sensor, the method using passive sensor, and the method using both sensors. It is most economical to estimate the accurate depth using only the camera information, which is a passive sensor But it is a technique with a high degree of difficulty.

The conventional multi-image depth estimation method is a method of estimating a depth image by comparing two images and fusing an estimated depth image. If a depth image having similar quality is fused, the quality of the depth image is not greatly improved.

In addition, there is a problem that the quality of the depth image estimated when occlusion, lack of texture, slope, and illumination change, which is a problem of stereo matching, generally can be greatly reduced.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of the present invention to estimate a depth image corresponding to each camera view using three or more multi-view images, A method of converging the matching cost, and a method of consistently estimating the depth image.

According to an aspect of the present invention, there is provided a depth image estimating method comprising: converting multi-view images into virtual view points; Estimating a depth image using the transformed multi-view images; And generating multi-view depth images using the estimated depth images.

In the conversion step, the multi-view images can be transformed as images obtained from cameras having equal reference intervals on a straight line.

The depth image estimating step includes: calculating a matching cost; Integrating costs; selecting a variation value; And a post-processing step.

The calculation step may be to calculate and fuse the matching cost with all images based on the mid-view image.

The calculation step may be using an Absolute Difference-Census transform (AD-CENSUS) similarity measure.

The post-processing step may be to remove noise in the depth image through noise cancellation filtering.

The generating includes: projecting the estimated depth image; And filling the empty space generated in the projected multi-view depth images using the median filter.

Meanwhile, according to another embodiment of the present invention, a depth image system includes a generator for generating multi-view images: a multi-view image is converted into a virtual view, a depth image is estimated using the converted multi-view images, And an image processor for generating multi-depth images using the depth image.

As described above, the multi-view image-based depth estimation technique is a highly complex technique, but according to embodiments of the present invention, high-quality depth images using about 20 high resolution images can be acquired in real time.

1 is a flowchart of a multi-viewpoint depth image estimation according to an exemplary embodiment of the present invention,
2 shows an overview of a virtual view image transformation,
3 illustrates a depth image estimation process,
4 illustrates a depth image estimation process,
FIG. 5 shows a result of multi-view three-
6 is a block diagram of an image system according to another embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is a flowchart of a multi-view depth image estimation according to an exemplary embodiment of the present invention.

The image-based depth estimation is a process of searching for corresponding points on two or more images and estimating the three-dimensional information using the acquired corresponding points. In order to acquire depth information using only relatively inexpensive image sensors, It is a technology that is considered as a priority in industries such as driving cars, robots, and games.

In this regard, in the embodiment of the present invention, a method of acquiring a high-quality depth image from a multi-view image is proposed. The method includes a step of converting an input image into a virtual viewpoint, a step of estimating a depth image using the input image, And generating a depth image at a further point.

1. Virtual view image transformation for multi-view depth estimation

In order to estimate the depth image efficiently, the embodiment of the present invention has a process of moving the input image to a virtual viewpoint. The distance Z of the object can be estimated through the amount of change (d, disparity) in the image and can be expressed by the following equation.

Where F is the focal length of the camera and B is the distance between the cameras. This means that the distance between adjacent cameras (B) is kept the same, meaning that the values of the objects obtained from the adjacent cameras (for example, cameras 1 and 2, cameras 10 and 11) are always the same, It is assumed that the cameras have the same focal length.

This process is shown in FIG. 2. The input image is obtained as shown in the left, but it is possible to project the input image to a virtual viewpoint as shown on the right side and convert it as an image acquired by a camera having an equal reference interval. Figure 2 is an overview of virtual view image transformation.

The transformed image satisfies the following conditional expression.

The image before conversion can be defined by an affine transformation as follows.

This leads to the problem of estimating two parameters a and b. Where x_k represents the (x-axis) position of the specific corresponding point viewed from the kth camera, and d corresponds to the mutation value of the corresponding point. Therefore, the transformation parameters can be summarized as follows.

Can be estimated using least squares or singular value decomposition using two or more corresponding points. After the estimation, it is possible to acquire an image having a constant reference interval by using a backward image transformation.

2. Depth estimation method by multi-view image fusion

As shown in FIG. 3, the image depth estimation method according to an exemplary embodiment of the present invention performs matching cost calculation, cost accumulation, variation value selection, and post-processing. 3 is a diagram illustrating a depth image estimating process.

The first step, the initial matching cost process, uses an AD-CENSUS (Absolute Difference-Census transform) similarity measure to calculate and fuse the matching cost with all images based on the mid-view image. It is known as one of the most effective measures for AD-CENSUS depth estimation and is widely used because it is easy to implement. Assuming that C_k is the matching cost estimated from the image of the middle point and the kth point, the matching cost for the left image and the matching cost for the right side can be obtained as follows.

Here, p is a pixel, and d is a variation value, which is a degree of similarity in which a pixel p at an intermediate point is compared with a pixel in the kth image (p is shifted by d by d). Divide the total cost by two without combining or fusing at once, then take the lower value as follows.

This makes it possible to obtain a robust matching cost for the masking phenomenon. Matching Cost After convergence, the following objective function is minimized to aggregate the cost.

Where C is the fused matched cost, P1 and P2 are penalty constants, N_p is a neighboring pixel of p, and T is a function that gives a value of 1 or 0. In the second term, T is given as 1 if the difference between p and q is 1, and 0 otherwise.

After obtaining the depth image through energy minimization, it has a process of removing the noise of depth image through noise cancellation filtering. For this, a weighted median filter and a joint bilateral filter are used. The noise removal process may be performed using any other type of filter.

Finally, the depth image estimated at multiple viewpoints is projected again to each camera viewpoint to obtain the multi viewpoint depth image. The hole generated after the projection is filled with the median filter.

FIG. 4 shows a depth image estimation process, and FIG. 5 shows a multi-view 3D reconstruction result.

6 is a block diagram of an image system according to another embodiment of the present invention. As shown in FIG. 6, the image system according to the embodiment of the present invention includes a multi-view image generation unit 110, an image processor 120, an image output unit 130, and a storage unit 140.

The multi-view image generation unit 110 generates multi-view images using a plurality of cameras, and the image processor 120 estimates depth images from multi-view images, and further generates intermediate view images using the depth images.

The image output unit 130 outputs the generated images to a display, an external device, a network, or the like. The storage unit 140 stores the generated images and provides a storage space necessary for the image processor 120 to estimate a depth image.

Up to now, a multi-view camera based depth image estimation method has been described in detail with a preferred embodiment.

In the multi-view camera-based depth image estimation method according to an embodiment of the present invention, multi-view depth images are estimated using images acquired from three or more multi-view cameras.

Specifically, in an embodiment of the present invention, a sophisticated depth image is estimated using all images simultaneously, not an approach of fusing independently estimated depth images, and a coherent depth image Thereby generating a depth image.

It goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium having a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention may be embodied in computer-readable code form recorded on a computer-readable recording medium. The computer-readable recording medium is any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. In addition, the computer readable code or program stored in the computer readable recording medium may be transmitted through a network connected between the computers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

110: Multi-view image generating unit
120: image processor
130: Video output unit
140:

Claims (8)

Converting the multi-view images into a virtual viewpoint;
Estimating a depth image using the transformed multi-view images;
And generating multi-viewpoint depth images using the estimated depth images,
In the depth image estimating step,
Calculating a matching cost for an image on the left side and a matching cost for an image on the right side based on an intermediate view image;
Selecting a lower matching cost out of the calculated matching costs;
Integrating the selected matching cost;
And estimating a depth image from the integrated matching cost.
The method according to claim 1,
In the conversion step,
Wherein the multi-view images are transformed as images obtained from cameras having equal reference intervals on a straight line.
The method according to claim 1,
In the depth image estimating step,
And a step of post-processing the estimated depth image.
delete The method of claim 3,
In the calculating step,
And an AD-CENSUS (Absolute Difference-Census transform) similarity measure.
The method of claim 3,
The post-
Wherein noise is removed from the depth image through noise cancellation filtering.
The method according to claim 1,
In the generating step,
Projecting an estimated depth image;
And filling the empty space generated in the projected multi-view depth images using a median filter.
Generating unit for generating multi-view images:
An image processor for converting multi-viewpoint images into virtual viewpoints, estimating depth images using the transformed multi-view images, and generating multi-viewpoint depth images using the estimated depth images,
In the image processor,
The matching cost for the image on the left side and the matching cost for the image on the right side are calculated based on the mid-view image, the selected matching cost is selected by selecting the lower matching cost among the calculated matching costs, And the depth image is estimated from the depth image.

Images