CN114519772A - Three-dimensional reconstruction method and system based on sparse point cloud and cost aggregation - Google Patents

Abstract

The invention relates to a three-dimensional reconstruction method and a system based on sparse point cloud and cost aggregation, wherein the method comprises the following steps: acquiring a multi-view image and a plurality of corresponding sparse point clouds, and preprocessing the sparse point clouds to obtain depth maps under a plurality of views; extracting features of the multi-view image, constructing one or more cost bodies, and modulating and regularizing each cost body by using the plurality of sparse point clouds to obtain a plurality of probability bodies; and restoring a depth map from each probability body, and fusing the depth map with the filtered depth maps under multiple viewing angles to obtain a reconstructed point cloud model. According to the method, the constructed cost body is modulated by using sparse prior through a strategy based on sparse point guide, and the accuracy of the cost body in estimating the depth of a weak texture region and a detailed structure is improved by using means such as regularization, so that the reconstruction quality of the three-dimensional point cloud model is improved.

Description

A 3D reconstruction method and system based on sparse point cloud and cost aggregation

technical field

The invention belongs to the technical field of computer vision, and in particular relates to a three-dimensional reconstruction method and system based on sparse point cloud and cost aggregation.

Background technique

Image-based 3D reconstruction, which aims to recover 3D geometry from multiple input images, is an important and challenging problem in the field of computer vision. Compared with active 3D reconstruction based on lidar, image-based 3D reconstruction has the advantages of low cost and strong versatility.

The traditional multi-view 3D reconstruction method performs cross-view similarity search based on hand-designed features, and can achieve good reconstruction results in an ideal Lambertian scene, but in weak texture areas and areas with specular reflections, due to the image Features are difficult to extract, resulting in unsatisfactory reconstruction results. In recent years, deep neural networks have been widely used in the field of computer vision. The deep learning method is based on a large amount of labeled data, and automatically learns the features of the input image through the deep neural network. Compared with traditional methods, the features extracted by deep neural networks contain more semantic information.

In 2020, scholars Xu and Tao from Huazhong University of Science and Technology replaced the variance-based cost index with Average Group-wise Correlation, which reduced GPU memory overhead without reducing the quality of model reconstruction. At the same time, the multi-view depth estimation problem is modeled as an inverse depth regression problem, which makes the model perform better in scenes with a large depth range.

Although the methods based on deep learning have made great progress, the results of sparse reconstruction have not been fully utilized, only the camera pose information is utilized, and the sparse point cloud information is ignored or not fully utilized.

SUMMARY OF THE INVENTION

In order to make full use of sparse point cloud information and improve the accuracy of depth estimation, thereby improving the reconstruction quality of the 3D point cloud model, especially to solve the problem that image features are difficult to extract in weak texture areas and areas with specular reflection, in the first step of the present invention. On the one hand, a 3D reconstruction method based on sparse point cloud and cost aggregation is provided, including: acquiring a multi-view image and its corresponding multiple sparse point clouds, and preprocessing the multiple sparse point clouds to obtain multiple sparse point clouds. The depth map under the perspective; perform feature extraction on the multi-view image and construct one or more cost volumes, and use the multiple sparse point clouds to modulate and regularize each cost volume to obtain multiple probability volumes; The depth map is recovered from each probability volume and fused with the filtered depth maps from multiple perspectives to obtain the reconstructed point cloud model.

In some embodiments of the present invention, the performing feature extraction on the multi-view image and constructing one or more cost volumes includes: using a convolutional neural network to perform feature extraction on each image of the multi-view image, respectively, Obtain multiple feature maps; select one feature map in the multiple feature maps as a reference feature map, and the remaining multiple feature maps as source feature maps, and calculate the feature body of each source feature map on the reference feature map, Obtain feature volumes of multiple views; aggregate the feature volumes of the multiple views into a cost volume.

Further, the aggregation of the feature volumes of the multiple views into a cost volume is implemented by the following method:

表示所有特征体的均值。where C represents the cost volume, M represents the element-by-element variance calculation, vi represents the _ith feature volume, N represents the total number of feature volumes,

Represents the mean of all feature volumes.

In some embodiments of the present invention, using the multiple sparse point clouds to modulate and regularize each cost volume to obtain multiple probability volumes includes: constructing a Gaussian modulation function based on depth maps from multiple viewing angles ; modulate each cost volume according to the Gaussian modulation function; use a 3D segmentation network to regularize each cost volume to obtain a filtered probability volume.

Further, the regularization is implemented by the following methods:

为正则化操作后体素v₀的代价；ω_k为第k个采样位置处的权重，v_k为卷积感受野中固定的偏置，Δv_k为自适应代价聚合过程中学习的偏置。where C(v ₀ ) is the cost of voxel v ₀ in each cost volume,

is the cost of the voxel v ₀ after the regularization operation; ω _k is the weight at the k-th sampling position, v _k is the fixed bias in the convolution receptive field, and Δv _k is the bias learned during the adaptive cost aggregation process.

In the above-mentioned embodiment, the preprocessing of the multiple sparse point clouds to obtain the depth maps from multiple perspectives includes: acquiring 3D points corresponding to all key points in each view, filtering invisible 3D points ; For the filtered multiple three-dimensional points, the depth value of each three-dimensional point in its image coordinate system is obtained through projection change and coordinate transformation according to the camera external parameters of the current view.

A second aspect of the present invention provides a 3D reconstruction system based on sparse point cloud and cost aggregation, including: an acquisition module for acquiring a multi-view image and a plurality of corresponding sparse point clouds, and for the plurality of sparse point clouds. The sparse point cloud is preprocessed to obtain depth maps from multiple perspectives; a building module is used to perform feature extraction on the multi-view image and construct one or more cost volumes, and use the multiple sparse point clouds for each The cost volume is modulated and regularized to obtain multiple probability volumes; the reconstruction module is used to recover the depth map from each probability volume and fuse it with the filtered depth maps from multiple perspectives to obtain the reconstructed point cloud model.

Further, the building module includes: an extraction unit for extracting features from each image of the multi-view image by using a convolutional neural network to obtain a plurality of feature maps; a calculation unit for selecting the plurality of One feature map in the feature map is used as the reference feature map, and the remaining multiple feature maps are used as the source feature map, and the feature volume of each source feature map on the reference feature map is calculated to obtain the feature volume of multiple views; the aggregation unit, for aggregating the feature volumes of the multiple views into a cost volume.

A third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are stored by the one or more programs The processor executes, so that the one or more processors implement the three-dimensional reconstruction method based on sparse point cloud and cost aggregation provided in the first aspect of the present invention.

A fourth aspect of the present invention provides a computer-readable medium on which a computer program is stored, wherein, when the computer program is executed by a processor, the sparse point cloud-based and cost aggregation provided in the first aspect of the present invention is implemented 3D reconstruction method.

The beneficial effects of the present invention are:

1. The present invention makes full use of the result of sparse reconstruction, and integrates the sparse point cloud obtained by sparse reconstruction into the cost body as prior information. Using the sparse depth map obtained by sparse point cloud projection as the geometric prior of the scene, by enhancing the depth hypothesis near the sparse prior and suppressing the depth hypothesis far from the prior depth value, the accuracy of depth estimation is improved, especially in the Small structures and depth discontinuities;

2. An adaptive cost aggregation module is introduced in the cost body regularization stage, so that the network model has the ability of scene perception, adaptively learns the bias in a data-driven manner, and obtains more accurate depth in weak texture areas and object boundaries estimate.

Description of drawings

FIG. 1 is a schematic flowchart of a basic flow of a 3D reconstruction method based on sparse point cloud and cost aggregation in some embodiments of the present invention;

FIG. 2 is a specific flowchart of a method for keeping the range of road network data of the electronic horizon minimized in some embodiments of the present invention;

3 is a schematic structural diagram of a convolutional neural network for image feature extraction in some embodiments of the present invention;

4 is a schematic diagram of sparse point cloud data preprocessing in some embodiments of the present invention;

5 is a schematic diagram of the working principle of a Gaussian modulation function in some embodiments of the present invention;

6 is a schematic diagram of a 3DUNet network structure for adaptive cost aggregation in some embodiments of the present invention;

7 is a schematic structural diagram of a 3D reconstruction system based on sparse point cloud and cost aggregation in some embodiments of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device in some embodiments of the present invention.

Detailed ways

The principles and features of the present invention will be described below with reference to the accompanying drawings. The examples are only used to explain the present invention, but not to limit the scope of the present invention.

Referring to FIG. 1 and FIG. 2, in a first aspect of the present invention, a 3D reconstruction method based on sparse point cloud and cost aggregation is provided, including: S100. Acquire a multi-view image and a plurality of corresponding sparse point clouds, and Preprocess the multiple sparse point clouds to obtain depth maps from multiple perspectives; S200. Perform feature extraction on the multi-view image and construct one or more cost volumes, and use the multiple sparse point clouds to pair Each cost volume is modulated and regularized to obtain multiple probability volumes; S300. Recover the depth map from each probability volume, and fuse it with the filtered depth maps from multiple perspectives to obtain reconstructed points Cloud model.

It is understandable that deep learning methods have made great progress, but have not yet fully utilized the results of sparse reconstruction, only using camera pose information, ignoring or not fully utilizing sparse point cloud information. The above method overcomes this problem and makes full use of sparse point cloud information, thereby improving the accuracy of the 3D reconstruction model. Multi-perspective images refer to multiple images obtained from different shooting angles (perspectives) of the same scene, that is, the multiple images originate from the same scene or image.

In step S200 of some embodiments of the present invention, the performing feature extraction on the multi-view image and constructing one or more cost volumes includes: S201. Using a convolutional neural network to perform feature extraction on each of the multi-view images respectively Perform feature extraction on the image to obtain multiple feature maps; S202. Select one feature map from the multiple feature maps as a reference feature map, and the remaining multiple feature maps as source feature maps, and calculate each source feature map in the reference From the feature volumes on the feature map, the feature volumes of multiple views are obtained; S203. Aggregate the feature volumes of the multiple views into a cost volume.

进行特征提取，得到N幅图像对应的特征图

卷积神经网络中包含8 个卷积层，除了最后一层卷积外，每个卷积后面都有BN层和ReLU激活函数。图像特征提取模块实现3×H×W到

的映射，其中H和W分别为输入图像的高和宽，C为特征图的通道维度。提取的特征图用于后续的代价体构建。Specifically, in step S201, the convolutional neural network shown in FIG. 3 is used as the input image

Perform feature extraction to obtain feature maps corresponding to N images

The convolutional neural network contains 8 convolutional layers. Except for the last layer of convolution, each convolution is followed by a BN layer and a ReLU activation function. The image feature extraction module realizes 3×H×W to

, where H and W are the height and width of the input image, respectively, and C is the channel dimension of the feature map. The extracted feature maps are used for subsequent cost volume construction.

为与参考图像进行匹配的源图像的特征图，

为稀疏重建得到的各个视图对应的相机内参矩阵、旋转矩阵和平移向量。以参考图像特征 F₁为基准，参考图像特征上的像素p，基于深度假设d_j，在源图像特征F_i上的对应像素点p_i,j的计算公式为：Next, in step S202, _F1 is the feature extracted from the reference image that needs depth estimation,

is the feature map of the source image matched with the reference image,

The camera intrinsic parameter matrix, rotation matrix and translation vector corresponding to each view obtained for sparse reconstruction. Taking the reference image feature F ₁ as the benchmark, the pixel p on the reference image feature, based on the depth hypothesis d _j , the calculation formula of the corresponding pixel p _i,j on the source image feature F _i is:

为了灵活地处理任意数量地输入视图，本发明利用基于方差的度量指标，将多视图的特征体

聚合为代价体C。where d _j is the jth depth hypothesis, where j∈{1,2,…,N _d }, N _d is the number of depth hypotheses, and the subscript ref represents the reference image feature. The feature map of each image is obtained by projective transformation to obtain the corresponding feature volume

In order to flexibly handle any number of input views, the present invention utilizes variance-based metrics to combine the feature volumes of multiple views

Aggregate as cost body C.

为所有特征体的均值，M为逐元素的方差计算，v_i表示第i个特征体，N表示特征体的总数。in the formula

is the mean of all feature volumes, M is the element-wise variance calculation, vi represents the _i -th feature volume, and N represents the total number of feature volumes.

In step S200 of some embodiments of the present invention, the modulating and regularizing each cost volume by using the multiple sparse point clouds to obtain multiple probability volumes includes: S204. Depth maps based on multiple viewing angles , construct a Gaussian modulation function; S205. Modulate each cost volume according to the Gaussian modulation function; S206. Use a 3D segmentation network to regularize each cost volume to obtain a filtered probability volume.

Referring to FIG. 5 , schematically, step S204 includes: constructing a Gaussian modulation function with the sparse depth value d′ as the center and the depth assumption d as an independent variable. The Gaussian modulation function is used for feature enhancement of the cost volume. Specifically, the response of the depth hypotheses near d=d' is enhanced, and the depth hypotheses far from d' are suppressed. Gaussian modulation function:

In the formula, d is the depth hypothesis value; d' is the sparse depth value; k is the magnitude of the Gaussian function, and c is the bandwidth. Considering the index of variance, the smaller the cost, the higher the confidence of the depth hypothesis, so the above formula is rewritten as:

where Ω _sparse is the set of pixels with a priori depth.

Referring to FIG. 6 , schematically, step S205 includes: using a 3DU-Net network to perform a regularization operation on the cost volume to obtain a filtered probability volume. The present invention introduces an adaptive cost aggregation module in the cost body regularization step, and the adaptive cost aggregation module can adaptively learn the offset, so that more accurate reconstruction accuracy can be obtained at the depth discontinuity. Optionally, use variants of the U-Net series or 3D image segmentation networks such as Segnet to regularize the cost volume.

到

的映射，沿深度方向对概率体进行soft－max归一化操作。Specifically, in step S206, the probability volume is normalized, and 3D convolution is used to convert the cost volume of the C channel into the probability volume of the single channel, so as to realize

arrive

The mapping of , performs soft-max normalization on the probability volume along the depth direction.

In step S206, the regularization is implemented by the following method:

为正则化操作后体素v₀的代价；ω_k为第k个采样位置处的权重，v_k为卷积感受野中固定的偏置，Δv_k为自适应代价聚合过程中学习的偏置。where C(v ₀ ) is the cost of voxel v ₀ in each cost volume,

is the cost of the voxel v ₀ after the regularization operation; ω _k is the weight at the k-th sampling position, v _k is the fixed bias in the convolution receptive field, and Δv _k is the bias learned during the adaptive cost aggregation process.

It can be understood that voxelization is to convert the geometric representation of an object into a voxel representation closest to the object to generate a volume data set, which not only contains the surface information of the model, but also can describe the internal properties of the model. Voxels are equivalent to pixels in an image, which can be understood as pixels in three-dimensional objects, which enables objects in three-dimensional space to be represented based on regular space voxels.

The representation methods of voxels usually include SDF and TSDF: SDF (signed distance field) is the effective distance field. That is, the surface of the object is simulated by assigning an SDF to each voxel. If the SDF value is greater than 0, it means that the voxel is in front of the current surface. If the SDF value is less than 0, it means that the voxel is behind the surface. The closer the SDF is to 0, the closer it is to the real surface of the scene. However, this representation method will take up a lot of resources, so someone proposed TSDF;

TSDF is proposed to reduce the resource consumption of voxel representation methods. TSDF uses grid cubes to represent the three-dimensional space. Each grid stores the distance to the surface of the object. The positive and negative values of the TSDF value indicate that they are occluded and visible by the surface, while the points on the surface pass through the zero point.

In step S300 of some embodiments of the present invention, the depth map is recovered from each probability volume, and the depth map is fused with the filtered depth maps from multiple viewing angles to obtain a reconstructed point cloud model including: Follow the steps below:

S301. Depth map regression: In order to achieve sub-pixel-level depth estimation accuracy, the present invention uses the weighted average of all depth hypotheses as the final depth output (soft-argmin operation), and the weight of each item is the probability value corresponding to the hypothesis. The pixel-wise depth estimate is calculated by:

where P(d) is the probability value corresponding to the depth hypothesis d;

S302. Calculation of photometric confidence map: The photometric confidence map is used to measure the quality of multi-view photometric consistency matching. The present invention sums the probabilities of the four hypothetical values closest to the depth hypothesized value to obtain the photometric confidence degree. The photometric confidence map is used for the depth map filtering in the above embodiment;

S303. Depth map filtering: Depth map filtering uses photometric consistency and geometric consistency for robust depth map filtering. The present invention uses a probability map to measure the quality of depth estimation, and filters out points with probability values lower than τ ₀ as outliers. Geometric consistency is used to measure depth continuity between multiple views. Project the depth value d ₁ at the pixel point p ₁ of the reference image to the point pi of the neighborhood view, and then _{reproject the depth value d i at the reference image p reproj to the reference image preproj , and its depth value d reproj , if it satisfies} |p _reproj -p ₁ |<τ ₁ and |d _reproj -d ₁ |/d ₁ <τ ₂ , then the depth estimation value d ₁ at the pixel p ₁ is said to be continuous between the two views. In the present invention, in order to ensure the continuity of depth estimation across views, only depth estimations that satisfy the continuity of at least n _τ views will be retained. The filtered pixels are back projected into the 3D space to obtain a dense 3D point cloud model.

In step S100 of the above-mentioned embodiment, the preprocessing of the multiple sparse point clouds to obtain depth maps from multiple viewing angles includes: S101. Obtaining 3D points corresponding to all key points in each view, filtering Invisible three-dimensional points; S102. For the filtered multiple three-dimensional points, the depth value of each three-dimensional point in its image coordinate system is obtained through projection change and coordinate transformation according to the camera external parameters of the current view.

Schematically, step S101 can refer to FIG. 4 : acquire 3D points corresponding to all key points in each view, filter the invisible 3D points, and record the visible 3D points in the view as P _world . Transform the three-dimensional point in the world coordinate system into the camera coordinate system according to the camera external parameters of the current view, and obtain the point P _cam in the camera coordinate system:

P _cam =R·P _world +t,

where R is the rotation matrix and t is the translation vector. According to the projection relationship, the projection point and its depth value of the 3D point in the image coordinate system are obtained

d(u,v,1) ^T =K·P _cam ,

where (u, v, 1) ^T is the homogeneous coordinate of the pixel coordinates of the image, and d is the depth value of the pixel. K is the camera internal parameter.

It should be understood that, before implementing the method of the present invention, parameters in the process of cost volume construction, cost volume modulation, depth map filtering and point cloud fusion need to be set. In practical applications, there are differences in the depth maps regressed by cost volumes with different numbers of depth assumptions; the Gaussian modulation functions corresponding to different amplitudes and bandwidths have different constraints, and different depth map fusion parameters will obtain 3D point clouds with different visual effects. Model. Various parameters of the present invention: Depth assumption plane number N _d = 192, k = 10, c = 2Δd, Δd = d _j+1 -d _j , τ ₀ =0.8, τ ₁ =1, τ ₂ =0.01, n _τ =3.

Example 2

Referring to FIG. 7 , a second aspect of the present invention provides a 3D reconstruction system 1 based on sparse point cloud and cost aggregation, comprising: an acquisition module 11 for acquiring a multi-view image and a plurality of corresponding sparse point clouds, and preprocess the multiple sparse point clouds to obtain depth maps from multiple perspectives; the building module 12 is used to perform feature extraction on the multi-view images and construct one or more cost volumes, and use the multiple perspective images to perform feature extraction. A sparse point cloud modulates and regularizes each cost volume to obtain multiple probability volumes; the reconstruction module 13 is used to recover the depth map from each probability volume and compare it with the filtered images from multiple viewpoints. The depth map is fused to obtain the reconstructed point cloud model.

Further, the building module 12 includes: an extraction unit for extracting features from each image of the multi-view image by using a convolutional neural network to obtain a plurality of feature maps; a calculation unit for selecting the multi-view images. One feature map in each feature map is used as a reference feature map, and the remaining multiple feature maps are used as source feature maps, and the feature volume of each source feature map on the reference feature map is calculated to obtain feature volumes of multiple views; aggregation unit , which is used to aggregate the feature volumes of the multiple views into a cost volume.

Example 3

8, a third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are described One or more processors execute such that the one or more processors implement the method of the first aspect of the invention.

Electronic device 500 may include processing means (eg, central processing unit, graphics processor, etc.) 501 that may be loaded into random access memory (RAM) 503 according to a program stored in read only memory (ROM) 502 or from storage means 508 program to perform various appropriate actions and processes. In the RAM 503, various programs and data necessary for the operation of the electronic device 500 are also stored. The processing device 501 , the ROM 502 , and the RAM 503 are connected to each other through a bus 504 . An input/output (I/O) interface 505 is also connected to bus 504 .

Typically the following devices can be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibrators output device 507 , etc.; storage device 508 including, for example, a hard disk; and communication device 509 . Communication means 509 may allow electronic device 500 to communicate wirelessly or by wire with other devices to exchange data. Although FIG. 8 shows electronic device 500 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in FIG. 8 can represent one device, and can also represent multiple devices as required.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 509 , or from the storage device 508 , or from the ROM 502 . When the computer program is executed by the processing device 501, the above-described functions defined in the methods of the embodiments of the present disclosure are performed. It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Rather, in embodiments of the present disclosure, a computer-readable signal medium may include a data signal in baseband or propagated as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more computer programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device:

Computer program code for carrying out operations of embodiments of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, Python, or a combination thereof, Also included are conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider to via Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. A three-dimensional reconstruction method based on sparse point cloud and cost aggregation is characterized by comprising the following steps:

acquiring a multi-view image and a plurality of corresponding sparse point clouds, and preprocessing the sparse point clouds to obtain depth maps under a plurality of views;

extracting features of the multi-view image, constructing one or more cost bodies, and modulating and regularizing each cost body by using the plurality of sparse point clouds to obtain a plurality of probability bodies;

and restoring a depth map from each probability body, and fusing the depth map with the filtered depth maps under multiple viewing angles to obtain a reconstructed point cloud model.

2. The sparse point cloud and cost aggregation-based three-dimensional reconstruction method of claim 1, wherein the feature extraction of the multi-view image and the construction of one or more cost bodies comprises:

respectively extracting the features of each image of the multi-view images by using a convolutional neural network to obtain a plurality of feature maps;

selecting one of the feature maps as a reference feature map, using the rest feature maps as source feature maps, and calculating feature bodies of each source feature map on the reference feature map to obtain feature bodies of multiple views;

and aggregating the characteristic bodies of the plurality of views into a cost body.

3. The sparse point cloud and cost aggregation-based three-dimensional reconstruction method according to claim 2, wherein the aggregation of the feature volumes of the plurality of views into the cost volume is achieved by:

where C represents a cost body, M represents a variance calculation element by element, v_iRepresenting the ith feature, N representing the total number of features,

mean values for all the features are indicated.

4. The sparse point cloud and cost aggregation-based three-dimensional reconstruction method of claim 1, wherein the modulating and regularizing each cost body by using the plurality of sparse point clouds to obtain a plurality of probability bodies comprises:

constructing a Gaussian modulation function based on the depth maps under a plurality of visual angles;

modulating each cost body according to the Gaussian modulation function;

and regularizing each cost body by using a 3D segmentation network to obtain a filtered probability body.

5. The sparse point cloud and cost aggregation-based three-dimensional reconstruction method of claim 4, wherein the regularization is achieved by:

wherein C (v)₀) For each voxel v in the cost volume₀The cost of (a) of (b),

operating on the post-regularization voxel v₀The cost of (d); omega_kIs the weight at the kth sample position, v_kFor a fixed offset in the convolution field, Δ v_kThe bias learned in the process of adaptive cost aggregation.

6. The sparse point cloud and cost aggregation-based three-dimensional reconstruction method according to any one of claims 1 to 5, wherein the preprocessing the plurality of sparse point clouds to obtain depth maps at a plurality of viewing angles comprises:

acquiring three-dimensional points corresponding to all key points under each view, and filtering invisible three-dimensional points;

and the depth value of each three-dimensional point in the image coordinate system is obtained by projection change and coordinate transformation of the filtered three-dimensional points according to the camera external parameters of the current view.

7. A three-dimensional reconstruction system based on sparse point cloud and cost aggregation comprises:

the system comprises an acquisition module, a depth map generation module and a depth estimation module, wherein the acquisition module is used for acquiring a multi-view image and a plurality of corresponding sparse point clouds and preprocessing the sparse point clouds to obtain depth maps under a plurality of views;

the construction module is used for extracting features of the multi-view image, constructing one or more cost bodies, and modulating and regularizing each cost body by using the plurality of sparse point clouds to obtain a plurality of probability bodies;

and the reconstruction module is used for recovering the depth map from each probability body and fusing the depth map with the filtered depth maps under the multiple viewing angles to obtain a reconstructed point cloud model.

8. The sparse point cloud and cost aggregation based three-dimensional reconstruction system of claim 7, wherein the construction module comprises:

the extraction unit is used for respectively extracting the features of each image of the multi-view images by using a convolutional neural network to obtain a plurality of feature maps;

the calculation unit is used for selecting one of the feature maps as a reference feature map, using the rest feature maps as source feature maps, and calculating feature bodies of each source feature map on the reference feature map to obtain feature bodies of a plurality of views;

and the aggregation unit is used for aggregating the characteristic bodies of the multiple views into a cost body.

9. An electronic device, comprising: one or more processors; a storage device to store one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the sparse point cloud and cost aggregation based three-dimensional reconstruction method of any one of claims 1 to 6.

10. A computer readable medium having a computer program stored thereon, wherein the computer program when executed by a processor implements the sparse point cloud and cost aggregation based three-dimensional reconstruction method of any one of claims 1 to 6.

Images