CN111899295B - A monocular scene depth prediction method based on deep learning - Google Patents

Abstract

A monocular scene depth prediction method based on deep learning is suitable for monocular pictures or videos, a calibrated binocular color image is used for training a depth prediction model, a DenseNet convolution module is used for extracting a feature space in a network architecture, and dense blocks and transition layers in the feature space are used for directly connecting each layer in the network with the previous layer so as to realize the repeated utilization of features; the binocular matching loss is improved, the depth prediction problem is regarded as the image reconstruction problem, the color image and the parallax image of the input left viewpoint are sampled to generate a virtual color image and a parallax image, and the consistency of the generated virtual view and the corresponding input right viewpoint image is restrained at an RGB level and a parallax level by utilizing a stereo matching algorithm of a binocular image pair, so that better depth is obtained; the depth smoothing loss is improved, the high-quality dense depth map can be generated, the problem of artifacts caused by shielding in monocular scene depth prediction is effectively solved, and the requirements of converting 2D (two-dimensional) to 3D (three-dimensional) of a plurality of indoor and outdoor real scenes can be met.

Description

A monocular scene depth prediction method based on deep learning

technical field

The invention relates to a monocular scene depth prediction method based on deep learning, belonging to the fields of computer vision and image processing.

Background technique

Monocular depth prediction is a research topic that has attracted much attention in computer vision, and has a wide range of application values in areas such as autonomous driving, VR game production, and film and television production. However, there are still many unsolved problems in this field, such as:

1) The process of using radar laser to collect depth data is costly and greatly affected by weather;

2) The generated depth image produces artifacts and smears due to light shadows or occlusions in the original image;

3) The method based on the depth information restored by the sparse depth map has the problem of discontinuous edge depth;

4) The depth model is not completely differentiable, and the computability of the gradient in optimization is lost, making the training suboptimal;

5) The image generation model does not have the ability to scale images to large output resolutions;

6) The generalization ability of the model is generally restricted by the training data.

Contents of the invention

In order to solve the above problems, the present invention provides a monocular scene depth prediction method based on deep learning, which is suitable for monocular pictures or videos. This method can obtain dense depth maps of scene images, and the accuracy rate can reach 91%. , which can meet the depth prediction requirements of multiple real-life scenes indoors and outdoors.

In order to achieve the above object, the technical solution of the present invention is as follows: a method for predicting the depth of a monocular scene based on deep learning, using calibrated color image pairs as training input, using the DenseNet convolution module to improve the network architecture of the encoder part, in Multiple levels of binocular stereo matching and image smoothing strengthen the loss constraint, use post-processing to improve the occlusion problem, and generally improve the effect of monocular image depth prediction. The method includes the following steps:

Step 1: Preprocessing operation, resize the high-resolution binocular color image pair to 256x512, and perform random flip and contrast transformation on the unified image pair to increase the amount of input data. Input into the encoder of the convolutional network.

Step 2: In the network encoder part, use the DenseNet convolution module to extract visual features, improve the transmission of information and gradients in the network through dense connections, alleviate the problem of gradient disappearance, and strengthen feature propagation;

Step 3: Set the skip scope in the decoder part, directly stitch some feature maps of the encoding process into the decoding process, use 64 7x7 convolution kernel convolutions to achieve upsampling, and use the sigmoid function as the activation function to generate parallax;

Step 4: Enhance the binocular matching loss and depth smoothing loss, optimize the model in iterations, improve the prediction accuracy, and make the depth map smooth while maintaining its edges;

Step 5: Optimized the post-processing part. On the one hand, due to the existence of three-dimensional occlusion, the left view can see more content on the left side, and it is easy to lose the information on the right side of the object. Therefore, the input image is flipped and the corresponding disparity map is generated, which is combined with the disparity map of the original image. edge information, optimize the output parallax, and alleviate the occlusion problem of the image edge; on the other hand, based on the object detection technology, use the original image to correct the output disparity map, highlight the object edge in the scene, and effectively eliminate the smear.

大小的tensor，随后进入四个由denseblock和过渡层组合而成的模块中；As an improvement of the present invention, step 2 is specifically as follows, the training data set of the depth prediction model is a calibrated binocular color image pair, the size is adjusted to 256x512 as the input of the network, and 64 7x7 convolution kernel volumes are processed once Product and maximum pooling, get

The tensor of size, then enters four modules composed of denseblock and transition layer;

The layers contained in the four denseblocks are 2, 6, 12, and 24 respectively, so that the number of network layers is continuously deepened, and the growth rate (growth_rate) of the dense layer in all denseblocks is set to 32, and the default bn_size is 4. In the dense block Bottleneck layer, adding 1x1 convolution before 3x3 convolution, the purpose is to reduce the parameter amount of the network, the transition layer uses average pooling to integrate global information between every two dense blocks, and improves the compactness of the model . The transfer of information and gradients in the network is improved through dense connections, and the problem of gradient disappearance is alleviated while the network is deepened, and feature propagation is strengthened.

As an improvement of the present invention, the step 4 is specifically as follows: In order to optimize the model and obtain a more accurate dense depth map, binocular matching loss and depth smoothing loss are added, and the constraints of the loss function on the network are strengthened:

(4.1) Binocular matching loss: The matching cost calculation is an important measure of the stereo matching algorithm. Using the correlation between the binocular image pair pixels, comparing the similarity between the reconstructed image and the relative view of the sampled image can strengthen the stereo matching. The present invention adds constraints on the parallax level on the basis of the RGB level.

(i，j表示像素点的位置坐标)，根据其预测的视差以及原有的右图，可以通过扭曲操作得到重构后的左图，这里的扭曲操作得出的

是根据左图每个像素点对应的视差值，在右图中寻找对应的像素点再插值得到的，使用L1和单尺度SSIM项组合作为图像重建的光度成本Cap，计算重建视图

与真实输入的视图

的匹配代价，In the case of spatial similarity, there is a strong correlation between the pixels of the RGB image, assuming that the original left image is

(i, j represent the position coordinates of the pixel point), according to the predicted parallax and the original right image, the reconstructed left image can be obtained through the distortion operation, and the distortion operation here is obtained

According to the parallax value corresponding to each pixel in the left image, find the corresponding pixel in the right image and then interpolate. Use the combination of L1 and single-scale SSIM items as the photometric cost Cap of image reconstruction to calculate the reconstructed view

view with real input

the matching cost of

C_lr促进预测的左右视差一致，In terms of disparity, the present invention tries to make the disparity map of the left viewpoint equal to the projected disparity map of the right viewpoint. Specifically, the d r with the right image as the reference image is used as the input image for the reconstruction operation, and the d ^r of the left image as the reference image ^l is used as the input disparity map, and the reconstructed disparity map of d ^l will be obtained after the warping operation

C _lr promotes consistent left and right disparity predictions,

(4.2) Depth smoothing loss: When there is an edge in the image, there must be a larger gradient value. On the contrary, when there is a smoother part in the image, the gray value changes less, and the corresponding gradient is also smaller. Direct The gray scale of the edge of the predicted disparity map (depth map) changes significantly, and the sense of hierarchy is strong. The problem of depth discontinuity often occurs at the gradient of the image. When encountering occlusion problems, the necessary object boundaries must also be maintained;

进行L1惩罚，然而，这种假设会错误地惩罚边缘，并不适用于通常对应于像素强度高变化区域的物体边界。因此，通过引入一个边缘感知项e，使用图像梯度

通过边缘感知项减少错误惩罚带来的影响，Due to the need for a dense disparity map, in order to keep the disparity locally smooth, the disparity gradient can be adjusted

Applying an L1 penalty, however, this assumption incorrectly penalizes edges and does not apply to object boundaries which usually correspond to regions of high variation in pixel intensity. Therefore, by introducing an edge-aware term e, using the image gradient

Reduce the impact of error penalties through edge-aware terms,

On the basis of introducing the edge-aware item e, the method of adaptively finding the field based on crossover is added, the idea is to limit the support field of a certain pixel point p, and reduce the impact of the error penalty edge. For point p, four arms are extended at the top, bottom, left, and right respectively, and the points that meet the conditions are included in the support area U _p of point p, and iterated until the following conditions are not met. (that is, the intensity cannot exceed too much, and the distance cannot be too far)

. In this solution, in the part of binocular matching loss, the present invention strengthens the matching loss between binocular image pairs based on the reconstruction technology, and increases the constraint of parallax level on the basis of RGB level. According to the original right-view RGB image and the predicted disparity image, twist to obtain the corresponding left image after reconstruction, compare the reconstructed left RGB image and disparity image with the original left image and the predicted disparity image, and use it as a binocular matching loss.

In the depth smoothing loss part, the present invention smoothes the depth map while maintaining the depth variation of object edges and occlusions. The smoothing operation is performed on the local part of the disparity map. At the same time, for the object boundary in the area of high pixel intensity change, not only an edge perception item e is introduced to reduce the impact of error penalty, but also an adaptive method based on crossover is introduced to find the field. The support area U _p of a certain pixel point p, so that compared with point p, the intensity of all pixels in U _p cannot exceed too much, and the distance cannot be too far, and iterate until the following conditions are not met.

As an improvement of the present invention, step (5) is specifically as follows, and the generated disparity map is improved. In terms of alleviating the occlusion of the image edge, for the input picture I, not only the disparity map D _l is calculated, but also the disparity map D _l ′ is calculated for the mirror flipped image I′ of the picture I, and then the disparity map is flipped to obtain the disparity map D _l ″ , D _l ″ and D _l are aligned. Combining the left 5% of D _l ″ and the right 5% of D _l , and the average between the two form the final result to reduce the impact of image edge stereo occlusion. At the same time, it increases the smear part of the elimination object edge, based on object recognition The technology recognizes the objects that may exist in the original image scene, aligns them with the output disparity map, enhances the pixels on the edge of the object in the disparity map, and sets the pixels in the smear part to the average value of the pixels in the field that removes the edge, highlighting the image in the scene. object, which improves the quality of the disparity map. In this scheme, due to the existence of stereo occlusion, the left view can see more content on the left side, and it is easy to lose the information on the right part of the object. Therefore, for the input picture I, not only the disparity map is calculated D _l , also calculates the disparity map D _l ′ for the mirror flipped image I′ of the picture I, selects the appropriate edge information of the two disparity maps, and optimizes the output parallax to reduce the influence of the three-dimensional occlusion of the image edge.

Since there may be moving objects in the scene, the input image has blurred parts. In order to highlight the objects in the scene, the present invention recognizes the objects that may exist in the original scene based on the object recognition technology, aligns them with the output disparity map, and converts the disparity map The pixel enhancement of the edge of the object in the middle is set to the pixel of the smear part as the average value of the field pixels except the edge, which eliminates the smear phenomenon to a certain extent and improves the quality of the disparity map.

Since there may be moving objects in the scene, the input image has blurred parts. In order to highlight the objects in the scene, the present invention recognizes the objects that may exist in the original scene based on the object recognition technology, aligns them with the output disparity map, and converts the disparity map The pixel enhancement of the edge of the object in the middle is set to the pixel of the smear part as the average value of the field pixels except the edge, which eliminates the smear phenomenon to a certain extent and improves the quality of the disparity map.

The technical solution is applicable to monocular pictures or videos. The present invention utilizes calibrated binocular color image pairs to train the depth prediction model, uses the DenseNet convolution module to extract the feature space in the network architecture, and uses the dense blocks and transition layers therein. Each layer in the network is directly connected to its previous layer to realize the reuse of features; the binocular matching loss is improved, the depth prediction problem is regarded as an image reconstruction problem, and the input left-viewpoint color map and disparity map are sampled to generate a virtual image. Color image and disparity image, using the stereo matching algorithm of binocular image pairs, constrain the consistency of the generated virtual view and the corresponding input right viewpoint image at the RGB level and disparity level, so as to obtain better depth; improved depth smoothing Loss, L1 penalty is applied to the smoother part of the image or the area with discontinuous depth. For the object boundary or occlusion part of the area with high pixel intensity change, the cross-based adaptive finding field is added on the basis of introducing the edge perception item e The method limits the support area of a certain pixel point p, and reduces the impact of the error penalty edge; in the post-processing optimization, based on the object detection technology, the output disparity map is corrected by using the original image, and the object edge in the scene is highlighted, effectively Eliminate smearing; in order to produce better effects on the right side of the object, flip the input original image to generate a corresponding flipped disparity map, combine the reversed disparity map with the original disparity map to intercept 5% of the left and right, and take the two in the middle The average value constitutes the final result; since the disparity map predicted by the moving object may have blur and smear, in order to highlight the objects in the scene, the present invention recognizes the objects that may exist in the original image scene based on the object recognition technology, and outputs the disparity map Alignment, enhances the pixels on the edge of the object in the disparity map, and sets the pixels in the smear part to the average value of the pixels in the area where the edge is removed, which eliminates the smear phenomenon to a certain extent and improves the quality of the disparity map. The invention can generate a high-quality dense depth map, effectively improve the artifact problem caused by occlusion in the monocular scene depth prediction, and can meet the 2D to 3D conversion requirements of multiple real scenes indoors and outdoors.

Compared with the prior art, the present invention has the following advantages: 1) The present invention improves the convolution module on the basis of using binocular image pairs to train and obtain a scene depth prediction model. Utilizing the feature of dense linking of the feature map in the DenseNet module, each layer accepts the feature input of the previous layer and outputs the feature of this layer to all subsequent layers, which reduces information consumption in the transmission process and improves the depth prediction accuracy; 2) In the binocular matching loss part, the invention uses reconstruction technology to increase the left and right parallax consistency constraints on the basis of binocular RGB image structural similarity constraints, and fully utilizes the advantages of binocular images as training data; 3) the present invention In the disparity smoothing loss part, the disparity smoothing operation is performed locally on the disparity map to obtain a smooth depth map. At the same time, the edge perception item and the method of adaptively finding the field are introduced to preserve the edge information of the image and occlusion, and a clearer depth map is obtained; 4) The present invention optimizes the edge of the output disparity map, and obtains more information on the right edge of the image by inverting the input image mirror, which effectively alleviates the occlusion problem; 5) the present invention can effectively eliminate the smear in the disparity map The problem is that by detecting objects in the original image and aligning them with the disparity map, the pixel values of the object edges are enhanced, and the accuracy of single object disparity prediction is improved.

Description of drawings

Fig. 1 is the overall flowchart of the present invention,

Figure 2 is a schematic diagram of binocular matching loss,

Figure 3 is a schematic diagram of finding a reasonable domain adaptively.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings, and the specific steps are as follows.

Embodiment 1: As shown in Figure 1, a method for predicting the depth of a monocular scene based on deep learning uses a calibrated color image pair as a training input, and uses the DenseNet convolution module to improve the network architecture of the encoder part. Multiple levels of stereo matching and image smoothing strengthen the loss constraints, use post-processing to improve the occlusion problem, and generally improve the effect of monocular image depth prediction. The method includes the following steps:

Step 1: Preprocessing operation, resize the high-resolution binocular color image pair to 256x512, and perform random flip and contrast transformation on the unified image pair to increase the amount of input data. Input into the encoder of the convolutional network.

Step 2: In the network encoder part, use the DenseNet convolution module to extract visual features, improve the transmission of information and gradients in the network through dense connections, alleviate the problem of gradient disappearance, and strengthen feature propagation;

Step 3: Set the skip scope in the decoder part, directly stitch some feature maps of the encoding process into the decoding process, use 64 7x7 convolution kernel convolutions to achieve upsampling, and use the sigmoid function as the activation function to generate parallax;

Step 4: Enhance the binocular matching loss and depth smoothing loss, optimize the model in iterations, improve the prediction accuracy, and smooth the depth map while maintaining its edges.

Step 5: Optimized the post-processing part. On the one hand, due to the existence of three-dimensional occlusion, the left view can see more content on the left side, and it is easy to lose the information on the right side of the object. Therefore, the input image is flipped and the corresponding disparity map is generated, which is combined with the disparity map of the original image. edge information, optimize the output parallax, and alleviate the occlusion problem of the image edge; on the other hand, based on the object detection technology, the original image is used to correct the output disparity map, highlighting the object edge in the scene, and effectively eliminating smear.

大小的tensor，随后进入四个由denseblock和过渡层组合而成的模块中；The details of step 2 are as follows. The training data set of the depth prediction model is a calibrated binocular color image pair, and the size is adjusted to 256x512 as the input of the network. After a convolution of 64 7x7 convolution kernels and maximum pooling, the obtained

The tensor of size, then enters four modules composed of denseblock and transition layer;

The layers contained in the four denseblocks are 2, 6, 12, and 24 respectively, so that the number of network layers is continuously deepened, and the growth rate (growth_rate) of the dense layer in all denseblocks is set to 32, and the default bn_size is 4. In the dense block Bottleneck layer, adding 1x1 convolution before 3x3 convolution, the purpose is to reduce the parameter amount of the network, the transition layer uses average pooling to integrate global information between every two dense blocks, and improves the compactness of the model . The transfer of information and gradients in the network is improved through dense connections, and the problem of gradient disappearance is alleviated while the network is deepened, and feature propagation is strengthened. The step 4 is as follows: In order to optimize the model and obtain a more accurate dense depth map, binocular matching loss and depth smoothing loss are added, and the constraints of the loss function on the network are strengthened:

(4.1) Binocular matching loss: The matching cost calculation is an important measure of the stereo matching algorithm. Using the correlation between the binocular image pair pixels, comparing the similarity between the reconstructed image and the relative view of the sampled image can strengthen the stereo matching. The present invention adds constraints on the parallax level on the basis of the RGB level.

(i，j表示像素点的位置坐标)，根据其预测的视差以及原有的右图，可以通过扭曲操作得到重构后的左图，这里的扭曲操作得出的

是根据左图每个像素点对应的视差值，在右图中寻找对应的像素点再插值得到的，使用L1和单尺度SSIM项组合作为图像重建的光度成本Cap，计算重建视图

与真实输入的视图

的匹配代价，In the case of spatial similarity, there is a strong correlation between the pixels of the RGB image, assuming that the original left image is

(i, j represent the position coordinates of the pixel point), according to the predicted parallax and the original right image, the reconstructed left image can be obtained through the distortion operation, and the distortion operation here is obtained

According to the parallax value corresponding to each pixel in the left image, find the corresponding pixel in the right image and then interpolate. Use the combination of L1 and single-scale SSIM items as the photometric cost Cap of image reconstruction to calculate the reconstructed view

view with real input

the matching cost of

C_lr促进预测的左右视差一致，In terms of disparity, the present invention tries to make the disparity map of the left viewpoint equal to the projected disparity map of the right viewpoint. Specifically, the d r with the right image as the reference image is used as the input image for the reconstruction operation, and the d ^r of the left image as the reference image ^l is used as the input disparity map, and the reconstructed disparity map of d ^l will be obtained after the warping operation

C _lr promotes consistent left and right disparity predictions,

(4.2) Depth smoothing loss: When there is an edge in the image, there must be a larger gradient value. On the contrary, when there is a smoother part in the image, the gray value changes less, and the corresponding gradient is also smaller. Direct The gray scale of the edge of the predicted disparity map (depth map) changes significantly, and the sense of hierarchy is strong. The problem of depth discontinuity often occurs at the gradient of the image. When encountering occlusion problems, the necessary object boundaries must also be maintained;

进行L1惩罚，然而，这种假设会错误地惩罚边缘，并不适用于通常对应于像素强度高变化区域的物体边界。因此，通过引入一个边缘感知项e，使用图像梯度

通过边缘感知项减少错误惩罚带来的影响，Due to the need for a dense disparity map, in order to keep the disparity locally smooth, the disparity gradient can be adjusted

Applying an L1 penalty, however, this assumption incorrectly penalizes edges and does not apply to object boundaries which usually correspond to regions of high variation in pixel intensity. Therefore, by introducing an edge-aware term e, using the image gradient

Reduce the impact of error penalties through edge-aware terms,

On the basis of introducing the edge-aware item e, the method of adaptively finding the field based on crossover is added, the idea is to limit the support field of a certain pixel point p, and reduce the impact of the error penalty edge. For point p, four arms are extended at the top, bottom, left, and right respectively, and the points that meet the conditions are included in the support area U _p of point p, and iterated until the following conditions are not met. (that is, the intensity cannot exceed too much, and the distance cannot be too far)

. In this solution, in the part of binocular matching loss, the present invention strengthens the matching loss between binocular image pairs based on the reconstruction technology, and increases the constraint of parallax level on the basis of RGB level. According to the original right-view RGB image and the predicted disparity image, twist to obtain the corresponding left image after reconstruction, compare the reconstructed left RGB image and disparity image with the original left image and the predicted disparity image, and use it as a binocular matching loss.

In the depth smoothing loss part, the present invention smoothes the depth map while maintaining the depth variation of object edges and occlusions. The smoothing operation is performed on the local part of the disparity map. At the same time, for the object boundary in the area of high pixel intensity change, not only an edge perception item e is introduced to reduce the impact of error penalty, but also an adaptive method based on crossover is introduced to find the field. The support area U _p of a certain pixel point p, so that compared with point p, the intensity of all pixels in U _p cannot exceed too much, and the distance cannot be too far, and iterate until the following conditions are not met.

The specific process of step (5) is as follows, and the generated disparity map is improved. In terms of alleviating the occlusion of the image edge, for the input picture I, not only the disparity map D _l is calculated, but also the disparity map D _l ′ is calculated for the mirror flipped image I′ of the picture I, and then the disparity map is flipped to obtain the disparity map D _l ″ , D _l ″ and D _l are aligned. Combining the left 5% of D _l ″ and the right 5% of D _l , and the average between the two form the final result to reduce the impact of image edge stereo occlusion. At the same time, it increases the smear part of the elimination object edge, based on object recognition The technology recognizes the objects that may exist in the original image scene, aligns them with the output disparity map, enhances the pixels on the edge of the object in the disparity map, and sets the pixels in the smear part to the average value of the pixels in the field that removes the edge, highlighting the image in the scene. object, which improves the quality of the disparity map. In this scheme, due to the existence of stereo occlusion, the left view can see more content on the left side, and it is easy to lose the information on the right part of the object. Therefore, for the input picture I, not only the disparity map is calculated D _l , also calculates the disparity map D _l ′ for the mirror flipped image I′ of the picture I, selects the appropriate edge information of the two disparity maps, and optimizes the output parallax to reduce the influence of the three-dimensional occlusion of the image edge.

Since there may be moving objects in the scene, the input image has blurred parts. In order to highlight the objects in the scene, the present invention recognizes the objects that may exist in the original scene based on the object recognition technology, aligns them with the output disparity map, and converts the disparity map The pixel enhancement of the edge of the object in the middle is set to the pixel of the smear part as the average value of the field pixels except the edge, which eliminates the smear phenomenon to a certain extent and improves the quality of the disparity map.

Application example:

As shown in Figure 1, the present invention mainly includes three parts, namely the improvement of the encoder module of the disparity prediction network, the enhancement of the loss function algorithm and the optimization of the post-processing. The following describes each part in detail:

1. Depth prediction network structure based on DenseNet module

The network is input with a resized 256x512 binocular RGB image pair. In the encoder part of the network, after a convolution of 64 7x7 convolution kernels and a maximum pooling maxpool, the image size becomes 1/4 of the original, The number of channels is 64.

The data then enters four modules composed of denseblock and transition layer. Set the growth rate (growth_rate) of the dense layer in all denseblocks to 32, the default bn_size=4, the Bottleneck layer of the denselayer increases Conv1x1, and outputs 128 channels (bn_size *growth_rate), output 32 channels (growth_rate) at Conv3x3. The transition layer uses average pooling to integrate global information, and sets the number of output channels to half of the number of input channels.

2. Algorithm optimization for binocular matching loss and depth smoothing loss

(1) Binocular matching loss calculation is an important measure of stereo matching algorithm. The present invention is based on reconstruction technology. This paper uses image reconstruction technology at two levels of RGB and parallax, compares the similarity between the reconstructed image and the relative view of the sampled image, and aggregates the calculation cost:

(i，j表示像素点的位置坐标)，根据其预测的视差以及原有的右图，可以通过扭曲操作得到重构后的左图。这里的扭曲操作得出的

是根据左图每个像素点对应的视差值，在右图中寻找对应的像素点再插值得到的。C_ap通过比较输入的原图

和和重建图片

得出，使用了包含3×3块滤波器而不是高斯滤波器的简化SSIM，其中N是像素数：Using a combination of L1 and single-scale SSIM terms as the photometric cost C _ap of image reconstruction promotes reconstructed images that are close in appearance to the training input. Suppose the original left image is

(i, j represent the position coordinates of the pixel point), according to the predicted disparity and the original right image, the reconstructed left image can be obtained through the warping operation. Here the warp operation results in

It is obtained by finding the corresponding pixel in the right image and then interpolating according to the parallax value corresponding to each pixel in the left image. C _ap by comparing the input original image

and reconstruct the picture

It follows that a simplified SSIM containing a 3×3 block filter instead of a Gaussian filter is used, where N is the number of pixels:

注意，这里得到的是重建的视差图，而非重构的左图,因此，左右视差一致性损失可以写作C_lr，促进预测的左右视差一致。In order to generate a more accurate disparity map, the network is trained to predict the disparity of the left and right images, and only the left view is used as the input of the network convolution part, and the output is not only the disparity map d ^l with the left image as the reference image, but also the right image is the disparity map d ^r of the reference image. To ensure consistency, an L1 left-right disparity consistency loss is introduced as part of the model. This cost tries to make the left-view disparity map equal to the projected right-view disparity map, specifically, taking d ^r with the right image as the reference image as the input image for the reconstruction operation, and d ^l with the left image as the reference image as the input The disparity map, after the distortion operation, the reconstructed disparity map of d ^l will be obtained

Note that what is obtained here is the reconstructed disparity map, not the reconstructed left image. Therefore, the left and right disparity consistency loss can be written as C _lr to promote the consistency of the predicted left and right disparity.

即为重建出来的

和其他所有的项一样，该代价也对应了右视点的视差图，可以在所有输出尺度上预测。Here N is the number of pixels,

to be reconstructed

Like all other terms, this cost also corresponds to the right-view disparity map, which can be predicted at all output scales.

(2) The gray level of the edge of the predicted disparity map contour changes significantly, and the sense of hierarchy is strong, and the problem of depth discontinuity often appears at the image gradient. Since this paper requires a dense disparity map, it is not only necessary to maintain local smoothness, but also to preserve the information of object edges and occlusions.

进行L1惩罚。然而，这种假设会错误地惩罚边缘，并不适用于通常对应于像素强度高变化区域的物体边界。因此，通过引入一个边缘感知项e，使用图像梯度

通过边缘感知项减少错误惩罚带来的影响。视差平滑损失

定义如下：In order to keep the disparity locally smooth, this paper uses the disparity gradient

Perform L1 penalty. However, this assumption incorrectly penalizes edges and does not apply to object boundaries that typically correspond to regions of high variation in pixel intensity. Therefore, by introducing an edge-aware term e, using the image gradient

Reduce the impact of error penalties through edge-aware terms. Parallax smoothing loss

It is defined as follows:

和图像梯度

Here are the disparity gradients using the x direction and the y direction respectively

and image gradient

On the basis of introducing the edge-aware item e, the method of adaptively finding the field based on crossover is added, the idea is to limit the support field of a certain pixel point p, and reduce the impact of the error penalty edge. For point p, four arms are extended at the top, bottom, left, and right respectively, and the points that meet the conditions are included in the support area U _p of point p, and iterated until the following conditions are not met. (that is, the intensity cannot exceed too much, and the distance cannot be too far)

3. In the post-processing part, not only the occlusion problem of the image edge is alleviated, but also the smear phenomenon of moving objects is effectively eliminated, and the output disparity map is optimized.

Due to the existence of three-dimensional occlusion, the left view can see more content on the left side, and it is easy to lose the information on the right side of the object. Therefore, for the input picture I, not only the disparity map D _l is calculated, but also the mirror image I′ of the picture I is calculated. Calculate the disparity map D _l ′, select the appropriate edge information of the two disparity maps, and optimize the output disparity to reduce the impact of image edge stereo occlusion.

Since there may be moving objects in the scene, the input image has blurred parts. In order to highlight the objects in the scene, the present invention recognizes the objects that may exist in the original scene based on the object recognition technology, aligns them with the output disparity map, and converts the disparity map The pixel enhancement of the edge of the object in the middle is set to the pixel of the smear part as the average value of the field pixels except the edge, which eliminates the smear phenomenon to a certain extent and improves the quality of the disparity map.

It should be noted that the above-mentioned embodiments are only preferred embodiments of the present invention, and are not used to limit the protection scope of the present invention. Equivalent replacements or replacements made on the basis of the above technical solutions all belong to the protection scope of the present invention.

Claims (4)

1. A method for predicting the depth of a monocular scene based on deep learning, characterized in that the method comprises the following steps: Step 1: Preprocessing operation, resize the high-resolution binocular color image pair to 256x512, and perform random flip and contrast transformation on the unified image pair to increase the amount of input data. Input into the encoder of the convolutional network, Step 2: In the network encoder part, use the DenseNet convolution module to extract visual features, improve the transmission of information and gradients in the network through dense connections, alleviate the problem of gradient disappearance, and strengthen feature propagation; Step 3: Set the skip scope in the decoder part, directly stitch some feature maps of the encoding process into the decoding process, use 64 7x7 convolution kernel convolutions to achieve upsampling, and use the sigmoid function as the activation function to generate parallax; Step 4: Enhance the binocular matching loss and depth smoothing loss, optimize the model in iterations, improve the prediction accuracy, and make the depth map smooth while maintaining its edges; Step 5: Optimize the post-processing part, reverse the input image to generate the corresponding parallax and combine the parallax of the original image to alleviate the edge occlusion problem, based on the object detection technology, align the original image with the parallax image, and strengthen the pixels on the edge of the object, to a certain extent Eliminate smearing phenomenon. 2. The monocular scene depth prediction method based on deep learning according to claim 1, wherein step 2 is specifically as follows, the training data set of the depth prediction model is a calibrated pair of binocular color images, and the size is adjusted It is 256x512 as the input of the network, after a 64 7x7 convolution kernel convolution and maximum pooling, it is obtained

The tensor of size, then enters four modules composed of denseblock and transition layer; The layers contained in the four denseblocks are 2, 6, 12, and 24 respectively, so that the number of network layers is continuously deepened, and the growth rate (growth_rate) of the dense layer in all denseblocks is set to 32, and the default bn_size is 4. In the dense block The Bottleneck layer adds a 1x1 convolution before the 3x3 convolution. 3. the monocular scene depth prediction method based on deep learning according to claim 1, is characterized in that, described step 4 is specifically as follows: (3.1) Binocular matching loss: The matching cost calculation is an important measure of the stereo matching algorithm, using the correlation between binocular image pairs of pixels, In the case of spatial similarity, there is a strong correlation between the pixels of the RGB image, assuming that the original left image is

(i, j represent the position coordinates of the pixel point), according to the predicted parallax and the original right image, the reconstructed left image is obtained through the distortion operation, and the distortion operation here is obtained

According to the parallax value corresponding to each pixel in the left image, find the corresponding pixel in the right image and then interpolate. Use the combination of L1 and single-scale SSIM items as the photometric cost Cap of image reconstruction to calculate the reconstructed view

view with real input

the matching cost of

At the disparity level, make the disparity map of the left viewpoint equal to the projected disparity map of the right viewpoint, take the d ^r of the right image as the reference image as the input image of the reconstruction operation, and then use the d ^l of the left image as the reference image as the input disparity map, After the warping operation, the reconstructed disparity map of d ^l will be obtained

C _lr promotes consistent left and right disparity predictions,

(3.2) Depth smoothing loss: by introducing an edge-aware term e, using the image gradient

Reduce the impact of error penalties through edge-aware terms,

On the basis of introducing the edge-aware item e, add a crossover-based self-adaptive method to find the field. For point p, extend four arms in the upper, lower, left, and right sides respectively, and bring the points that meet the conditions into the support area U _p of point p, and iterate continuously Terminate until the following conditions are not met;

4. The monocular scene depth prediction method based on deep learning according to claim 1, wherein step (5) is specifically as follows, the generated disparity map is improved, and in alleviating the occlusion of the image edge, for the input picture I, not only calculate the disparity map D _l , but also calculate the disparity map D _l ′ for the mirror flip image I′ of the picture I, and then flip the disparity map to obtain the disparity map D _l ″, D _l ″ and D _l are aligned; combined with D The left 5% of _l ″ and the right 5% of D _l , and the average between the two form the final result, so as to reduce the influence of image edge stereoscopic occlusion.

Images