CN111325782A - Unsupervised monocular view depth estimation method based on multi-scale unification - Google Patents

Abstract

The invention belongs to the technical field of image processing, and discloses an unsupervised monocular view depth estimation method based on multi-scale unity, comprising the following steps: S1: performing pyramid multi-scale processing on input stereo image pairs; S2: constructing a network framework for encoding and decoding ; S3: transfer the features extracted in the encoding stage to the inverse convolutional neural network to achieve feature extraction of input images of different scales; S4: uniformly upsample the disparity maps of different scales to the original input size; S5: use the input original image Perform image reconstruction with the corresponding disparity map; S6: Constrain the accuracy of image reconstruction; S7: Use the gradient descent method to train the network model; S8: Fit the corresponding disparity map according to the input image and the pre-trained model. The design of the invention does not need to use real depth data to supervise network training, and easily obtained binocular images are used as training samples, which greatly reduces the acquisition difficulty of network training and solves the problem of depth map holes caused by low-scale disparity map blurring.

Description

An unsupervised monocular view depth estimation method based on multi-scale unification

technical field

The invention relates to the technical field of image processing, in particular to an unsupervised monocular view depth estimation method based on multi-scale unification.

Background technique

With the development of science and technology and the explosive growth of information, people's attention to image scenes is gradually shifting from two-dimensional to three-dimensional, and the three-dimensional information of objects has played a great role in daily life. Among them, three-dimensional information is the most widely used The most important thing is the assisted driving system in the driving scene. Due to the wealth of information contained in images, vision sensors cover almost all relevant information needed for driving, including but not limited to lane geometry, traffic signs, lights, object positions and speeds, etc. Among all forms of visual information, depth information plays a very important role in driver assistance systems. For example, collision avoidance systems issue collision warnings by calculating the depth information between the obstacle and the vehicle. When the distance between the pedestrian and the vehicle is too small, the pedestrian protection system will automatically take measures to slow down the vehicle. Therefore, only by obtaining the depth information between the current vehicle and other traffic participants in the driving scene, the driving assistance system can accurately obtain the connection with the external environment, so that the early warning subsystem can work normally.

At present, there are many sensors on the market that can obtain depth information, such as Sick's LiDAR. Lidar can generate sparse 3D point cloud data, but its disadvantages are high cost and limited use scenarios, so people turn their attention to recovering 3D structural information of scenes from images.

The traditional image-based depth estimation methods are mostly based on geometric constraints and manual features assumed by the shooting environment, and widely used methods such as recovering structures from motion. The environment requirements are not high and it is easy to operate, but the disadvantage of this method is that it is easily affected by feature extraction and matching errors, and only relatively sparse depth data can be obtained.

As Convolutional Neural Networks shined on other vision tasks, many researchers began to explore the application of deep learning methods for depth estimation in monocular images. People use the powerful learning ability of neural networks to design various models to fully exploit the connection between the original image and the depth map, so as to train the scene depth that can be predicted based on the input image, but as mentioned above, the real depth information of the scene It is very rare, which means that we need to break away from the real depth label of the scene and use an unsupervised method to complete the depth estimation task. One of the unsupervised methods is to use the time series information of the monocular video as the supervision signal, but this kind of unsupervised depth estimation method uses the video information collected during the motion process, so the camera itself has motion, and the image sequence The relative poses of the cameras are unknown, which leads to the need to train a pose estimation network in addition to the depth estimation network, which undoubtedly increases the difficulty of the already complex depth estimation task. In addition, due to the scale uncertainty of monocular video, this method can only obtain relative depth results, that is, it can only obtain the relative distance between pixels in the image, but cannot obtain the distance from the object in the image to the camera. In addition, the unsupervised depth estimation method has the situation that the depth map texture is missing or even hollow due to the blurred details of low-scale feature maps, which directly affects the accuracy of depth estimation.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the shortcomings of the prior art, and propose an unsupervised monocular view depth estimation method based on multi-scale unity.

In order to achieve the above object, the present invention adopts the following technical solutions:

An unsupervised monocular view depth estimation method based on multi-scale unity, including the following steps:

Step S1: perform pyramid multi-scale processing on the input stereo image pair, so as to perform feature extraction on multiple scales;

Step S2: constructing a network framework for encoding and decoding to obtain a disparity map that can be used to obtain a depth map;

Step S3: delivering the features extracted in the encoding stage to the inverse convolutional neural network to achieve feature extraction of input images of different scales, and fitting disparity maps of input images of different scales in the decoding stage;

Step S4: uniformly up-sampling the disparity maps of different scales to the original input size;

Step S5: using the original image of the input stereoscopic image and the corresponding disparity map to perform image reconstruction;

Step S6: Constrain the accuracy of image reconstruction through appearance matching loss, left-right disparity conversion loss, and disparity smoothing loss;

Step S7: Using the idea of minimizing loss, the gradient descent method is used to train the network model;

Step S8: In the testing phase, fit a corresponding disparity map according to the input image and the pre-trained model; use the triangulation principle of binocular imaging to calculate the corresponding scene depth map from the disparity map.

Preferably, in the step S1, the input image is down-sampled to four sizes of 1, 1/2, 1/4, and 1/8 of the original image to form a pyramid input structure, and then sent to the encoding model for feature extraction .

Preferably, in the step S2, the ResNet-101 network structure is used as the network model in the coding stage, and the ResNet network structure adopts the residual design, which reduces information loss while the network is deepened.

Preferably, in the step S3, feature extraction is performed on input images of different scales in the encoding stage, and the extracted features are sent to the inverse convolutional neural network in the decoding stage to implement disparity map fitting, specifically:

Step S41: In the encoding stage, the input images of the pyramid structure are respectively subjected to feature extraction through the ResNet-101 network, and in the extraction process, the input images of different sizes are reduced to 1/16, and the obtained size is 1/16 of the original input image, 1/32, 1/64, 1/128 characteristics;

Step S42: Input the four-dimensional features obtained in the encoding stage into the network in the decoding stage, and deconvolve the input features layer by layer in the process to restore them to the original input image 1, 1/2, 1/ 4, 1/8 size pyramid structure, fit the disparity map of the 4 size images according to the input features and the deconvolution network;

Preferably, in the step S4, the disparity maps whose sizes are 1, 1/2, 1/4, and 1/8 of the original input image are uniformly up-sampled to the size of the original input image.

原右图I^r与左视差图d^l重建出左图

Preferably, in the step S5, since the disparity maps of 4 sizes are uniformly upsampled to the original input size, the right ^image is reconstructed using the originally input left image I1 and right disparity image d ^r

The left image is reconstructed from the original right image I ^r and the left disparity image d ^l

Preferably, in the step S6, the original input left and right views and the reconstructed left and right views are used to calculate the loss to constrain the accuracy of image reconstruction;

The gradient descent method is used to minimize the loss function to train the image reconstruction network, which is as follows:

Step S71: The loss function is composed of three parts, which are the appearance matching loss C _a , the smoothing loss C _s and the disparity conversion loss C _t ; for each loss, the calculation methods of the left and right images are the same, and the final loss A function is composed of three items:

Step S72: Calculate the losses on the different disparity maps of the original input size and the input original image respectively, and obtain corresponding 4 losses C _i , i=1, 2, 3, 4, and the total loss function is

Preferably, in the step S7, the idea of minimizing the loss is used, and the gradient descent method is used to train the network model.

Preferably, in the step S8, in the testing stage, the input single image and the pre-trained model are used to fit the disparity map corresponding to the input image, and the corresponding depth image is generated by using the disparity map according to the triangulation principle of binocular imaging, Specifically:

Among them, (i,j) is the pixel-level coordinate of any point in the image, D(i,j) is the depth value of the point, d(i,j) is the disparity value of the point, and b is the known two The distance between the two cameras, f is the known focal length of the camera.

The multi-scale unified unsupervised monocular view depth estimation method of the present invention requires a real depth image corresponding to the input image when the common deep learning method solves the depth estimation problem, but the acquisition of the real depth data is expensive, And only the sparse point cloud depth can be obtained, which cannot fully meet the application requirements; in this case, this paper adopts the image reconstruction loss to supervise the training process of the model, and uses the relatively easy-to-obtain binocular image instead of the real depth for training, so as to solve the problem. achieve unsupervised depth estimation;

The multi-scale unified unsupervised monocular view depth estimation method of the present invention reduces the influence of objects of different sizes on the depth estimation by performing pyramid multi-scale processing on the input stereo image pair in the encoding stage;

In the unsupervised monocular view depth estimation method based on multi-scale unity, in the case of low-scale depth map blur, all disparity maps are uniformly sampled to the original input size, and the image is reconstructed on this size And the calculation of the loss, which improves the problem of holes in the depth map;

The invention has a reasonable design, does not need to use real depth data to supervise network training, and uses easily obtained binocular images as training samples, which greatly reduces the acquisition difficulty of network training, and also solves the depth map hole caused by the blurring of low-scale disparity maps. The problem.

Description of drawings

1 is a flowchart of a method for depth estimation of unsupervised monocular view based on multi-scale unity proposed by the present invention;

2 is a network model structure diagram of a multi-scale unified unsupervised monocular view depth estimation method proposed by the present invention;

3 is a schematic diagram of a network structure bottleneck module of a multi-scale unified unsupervised monocular view depth estimation method proposed by the present invention;

4 is a schematic diagram of scale unification of an unsupervised monocular view depth estimation method based on multi-scale unification proposed by the present invention;

5 is an estimation result diagram of a multi-scale unified unsupervised monocular view depth estimation method proposed by the present invention on the classic driving data set KITTI, (a) is an input image, (b) is a depth estimation result diagram;

6 is a generalization effect diagram of a multi-scale unified unsupervised monocular view depth estimation method proposed by the present invention on real-time captured pictures of road scenes, (a) is the input image, (b) is the depth estimation result diagram .

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments.

Referring to Figure 1-6, an unsupervised monocular view depth estimation method based on multi-scale unity, the unsupervised depth monocular depth estimation network model is carried out on the desktop workstation of this laboratory, the graphics card adopts NVIDIA GeForceGTX 1080Ti, and the training system For Ubuntu14.04, use TensorFlow 1.4.0 as the framework to build the platform; train on the classic driving dataset KITTI 2015 stereo dataset.

As shown in FIG. 1 , an unsupervised monocular view depth estimation method based on multi-scale unity of the present invention specifically includes the following steps:

Step S1: Use the binocular data set in the classic driving KITTI as the training set, set the scale parameter to 4, downsample the image to 1/2, 1/4, and 1/8 of the input image, plus the original image has a total of 4 sizes The input image forms a pyramid structure, which is then sent to the ResNet-101 neural network model for feature extraction;

Step S2: constructing a network framework for encoding and decoding to obtain a disparity map that can be used to obtain a depth map; the specific process is:

The ResNet-101 network structure is used as the network model in the coding stage. The ResNet network structure adopts the residual design, which reduces the loss of information while the network is deepened; the residual structure in the ResNet network is shown in Figure 3(a), first use 1 The convolution of ×1 reduces the feature dimension, and then restores it through the convolution of 1 × 1, the parameter quantity is:

1×1×256×64+3×3×64×64+1×1×64×256=69632

The usual ResNet module is shown in Figure 3(b), and the parameters are:

3×3×256×256×2=1179648

It can be seen that using the residual module with a bottleneck structure can greatly reduce the amount of parameters;

Step S3: The features extracted in the encoding stage are sent to the inverse convolutional neural network to achieve feature extraction of input images of different scales, and the disparity maps of input images of different scales are fitted in the decoding stage, specifically:

Step S31: During the network decoding process, in order to ensure that the size of the feature map in the deconvolutional neural network corresponds to the size of the ResNet-101 residual network feature map, the network uses skip connections to directly encode some feature maps in the ResNet-101 encoding process. connected to a deconvolutional neural network;

Step S32: In the encoding stage, the input image of the pyramid structure is extracted by the ResNet-101 network, and in the extraction process, the input image of different sizes is reduced to 1/16, and the obtained size is 1/16 of the original input image, 1/32, 1/64, 1/128 characteristics;

Step S33: Input the four-dimensional features obtained in the encoding stage into the network in the decoding stage, and deconvolve the input features layer by layer in this process to restore them to the original input image 1, 1/2, 1/ 4. The pyramid structure of 1/8 size, according to the input feature and the deconvolution network, respectively fit the approximate disparity map of the 4 size images;

Step S4: uniformly up-sampling the disparity maps whose sizes are 1, 1/2, 1/4, and 1/8 of the original input image to the size of the original input image;

Step S5: use the input original image and the corresponding disparity map to reconstruct the image, use the disparity map and its corresponding left view to reconstruct the right view, then use the original right image and the left disparity map to reconstruct the left image, and finally reconstruct the image. The output left and right images are compared with the input left and right original images respectively;

Step S6: Reuse appearance matching loss, left and right parallax conversion loss, and parallax smoothing loss to constrain the accuracy of image synthesis; specifically:

Step S61: the loss function consists of three parts, which are the appearance matching loss C _a , the smoothing loss C _s and the disparity conversion loss C _t ;

In the image reconstruction process, the appearance matching loss C _a is first used to judge the accuracy between the reconstructed image and the corresponding input image pixel by pixel. The loss is composed of the structural similarity index and L ₁ loss. Figure as an example:

Among them, S is the structural similarity index, which consists of three parts: brightness measurement, contrast measurement and structural comparison. It is used to measure the similarity between two images. The more similar the two images are, the higher the similarity index value is; L ₁ The loss is to minimize the absolute error loss, which is used to compare the difference between the two images pixel by pixel, and is less sensitive to outliers _than the L2 loss; α is the weight coefficient of the structural similarity in the appearance matching loss, N is the total number of pixels in the image;

Secondly, the smoothing loss α _s can reduce the discontinuity of the disparity map caused by the excessive local gradient, and ensure the smoothness of the disparity map. Take the left image as an example, the specific formula is as follows:

The purpose of the disparity conversion loss C _t is to reduce the conversion error between the right disparity map generated from the left image and the left disparity map generated from the right image, and to ensure the consistency between the two disparity maps. The specific formula is as follows:

For each loss, the left and right graphs are calculated in the same way, and the final loss function is composed of three terms:

where α _a is the weight of the appearance matching loss in the overall loss, α _s is the weight of the smoothing loss in the overall loss, and α _t is the weight of the transformation loss in the overall loss;

Step S62: Calculate the losses on the different disparity maps on the original input size and the input original image respectively, and obtain four corresponding losses C _i , i=1, 2, 3, 4, and the total loss function is

Step S7: Using the idea of minimizing loss, the gradient descent method is used to train the network model, specifically: in the training process of the stereo image pair, we use the open source TensorFlow 1.4.0 platform to build a depth estimation model, and use the The KITTI dataset is used as a training set, and the 29,000 pairs in the dataset are used for model training; during training, the initial learning rate lr is set, and after 40 cycles, the learning rate is changed to the current learning rate every 10 cycles half of , and a total of 70 cycles of training; at the same time, set the batch size to bs, that is, to process bs pictures at a time; use the Adam optimizer to optimize the model, and set β ₁ , β ₂ to control the decay rate of the weight coefficient moving average, and finally It took 34 hours to complete all training on the GTX 1080Ti experimental platform;

Table 1 Loss function and training parameters

Step S8: In the test phase, fit the corresponding disparity map according to the input image and the pre-training model; use the triangulation principle of binocular imaging to calculate the depth map of the corresponding scene from the disparity map; in the KITTI road driving data set used in this experiment. , the baseline distance of the camera is fixed at 0.54m, and the focal length of the camera changes according to the different camera models. Different camera models are reflected in different image sizes in the KITTI dataset. The corresponding relationship is as follows:

Then the conversion formula of depth and disparity is as follows:

Among them, (i, j) is the pixel coordinate of any point in the image, D(i, j) is the depth value of the point, and d(i, j) is the disparity value of the point;

Therefore, according to the input image and the pre-trained network model using the principle of binocular image reconstruction, the disparity map corresponding to the input image is fitted, and according to the known camera focal length and baseline distance, the input image captured by the camera can be calculated Corresponds to the scene depth map.

The standard parts used in the present invention can be purchased from the market, the special-shaped parts can be customized according to the description in the specification and the drawings, and the specific connection methods of each part adopt mature bolts, rivets, welding, etc. in the prior art Conventional means, machinery, parts and equipment all adopt conventional models in the prior art, and circuit connections adopt conventional connection methods in the prior art, which will not be described in detail here.

Claims (9)

1. An unsupervised monocular view depth estimation method based on multi-scale unification is characterized by comprising the following steps:

step S1: carrying out pyramid multi-scale processing on the input stereo image pair so as to extract features of multiple scales;

step S2: constructing a network framework of coding and decoding to obtain a disparity map which can be used for calculating a depth map;

step S3: the features extracted in the encoding stage are transmitted to a reverse convolution neural network to realize the feature extraction of the input images with different scales, and the disparity maps of the input images with different scales are fitted in the decoding stage;

step S4: uniformly up-sampling disparity maps of different scales to an original input size;

step S5: reconstructing an image by using the input original image and a corresponding disparity map;

step S6: the accuracy of image reconstruction is constrained through appearance matching loss, left-right parallax conversion loss and parallax smoothing loss;

step S7: training a network model by using a gradient descent method by using a loss minimization idea;

step S8: in the testing stage, fitting a corresponding disparity map according to an input image and a pre-training model; and calculating a corresponding scene depth map by using a binocular imaging triangulation principle and the disparity map.

2. The method as claimed in claim 1, wherein in step S1, the input image is down-sampled to four sizes of 1, 1/2, 1/4, 1/8 of the original image to form a pyramid input structure, and then sent to the coding model for feature extraction.

3. The unsupervised monocular view depth estimation method based on multi-scale unification as claimed in claim 1, wherein in step S2, a ResNet-101 network structure is adopted as a network model in an encoding stage.

4. The unsupervised monocular view depth estimation method based on multi-scale unification as claimed in claim 1, wherein in step S3, feature extraction is performed on input images of different scales in an encoding stage, and the extracted features are transmitted to a deconvolution neural network in a decoding stage to implement disparity map fitting, specifically:

step S41: respectively performing feature extraction on the input image with the pyramid structure through a ResNet-101 network in an encoding stage, and reducing the input image to 1/16 in the extraction process relative to input images with different sizes to obtain features of original input images 1/16, 1/32, 1/64 and 1/128;

step S42: inputting the features of four sizes obtained in the encoding stage into the network in the decoding stage, deconvoluting the input features layer by layer in the process to restore the input features to the pyramid structures of the original input images 1, 1/2, 1/4 and 1/8 sizes, and respectively fitting the disparity maps of the images of 4 sizes according to the input features and the deconvolution network.

5. The unsupervised monocular view depth estimation method based on multi-scale unification as claimed in claim 1, wherein in the step S4, the disparity maps with the size of 1, 1/2, 1/4, 1/8 of the original input image are unified up-sampled to the size of the original input image.

6. The unsupervised monocular view depth based on multi-scale unification of claim 1The estimation method is characterized in that in the step S5, since the disparity maps of 4 sizes are uniformly up-sampled to the original input size, the originally input left map I is used^lAnd the right parallax image d^rReconstruct a right image

Original right picture I^rAnd left parallax map d^lReconstruct the left image

7. The unsupervised monocular view depth estimation method based on multi-scale unification as claimed in claim 1, wherein in step S6, accuracy of image reconstruction is constrained by calculating loss using the original input left and right views and the reconstructed left and right views;

minimizing a loss function by adopting a gradient descent method, and training an image reconstruction network by adopting the method, specifically:

step S71: the loss function is composed of three parts, namely appearance matching loss and appearance loss C_aSmoothing loss C_sAnd parallax conversion loss C_t(ii) a For each term loss, the left and right graphs are computed in the same way, and the final loss function is composed of three terms:

step S72: respectively calculating losses on different parallax maps and the input original image on the original input size to obtain 4 losses C_iI is 1,2,3,4, the total loss function is

8. The unsupervised monocular view depth estimation method based on multi-scale unification as claimed in claim 1, wherein in step S7, a network model is trained by using a gradient descent method using an idea of minimizing loss.

9. The unsupervised monocular view depth estimation method based on multi-scale unification as claimed in claim 1, wherein in the step S8, in the testing stage, an input single image and a pre-trained model are used to fit a disparity map corresponding to the input image, and according to a principle of triangulation of binocular imaging, the disparity map is used to generate a corresponding depth image, specifically:

where (i, j) is the pixel-level coordinate of any point in the image, D (i, j) is the depth value of the point, D (i, j) is the parallax value of the point, b is the known distance between two cameras, and f is the known focal length of the camera.

Images