CN106910242A - The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera - Google Patents

Abstract

The present invention discloses a kind of method and system that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera.Wherein, the method includes that obtaining depth image carries out self adaptation bilateral filtering；Visual odometry estimation is carried out using filtered depth image, view-based access control model content does automatic segmentation to image sequence, closed loop detection is done between section and section, and carry out global optimization；Volume data fusion is weighted according to the camera trace information after optimization, so as to rebuild indoor full scene threedimensional model.The embodiment of the present invention realizes guarantor side, the denoising to depth map by self adaptation bilateral filtering algorithm, view-based access control model content automatic segmentation algorithm can effectively reduce the accumulated error during visual odometry is estimated and improve registration accuracy, weighted body data anastomosing algorithm is additionally used, the geometric detail of body surface can be effectively kept.Thus, the technical problem for how improving reconstruction accuracy under indoor scene is solved such that it is able to obtain indoor scene model that is complete, accurate, becoming more meticulous.

Description

Method and system for 3D reconstruction of indoor complete scene based on depth camera

technical field

The present invention relates to the technical field of computer vision, in particular to a method and system for three-dimensional reconstruction of a complete indoor scene based on a consumer-grade depth camera.

Background technique

High-precision 3D reconstruction of indoor scenes is one of the challenging research topics in computer vision, involving theories and technologies in many fields such as computer vision, computer graphics, pattern recognition, and optimization. There are many ways to achieve 3D reconstruction. The traditional method is to use laser, radar and other ranging sensors or structured light technology to obtain the structural information of the scene or object surface for 3D reconstruction. However, most of these instruments are expensive and not easy to carry, so the application occasions are limited. . With the development of computer vision technology, researchers began to study the use of pure vision methods for 3D reconstruction, and a lot of useful research work emerged.

After the launch of the consumer-grade depth camera Microsoft Kinect, people can directly use the depth data to perform 3D reconstruction of indoor scenes more conveniently. The KinectFusion algorithm proposed by Newcombe et al. uses Kinect to obtain the depth information of each point in the image, and uses the Iterative Closest Point (ICP) algorithm to compare the coordinates of the 3D points in the current frame camera coordinate system with those in the global model. Align the coordinates of the current frame to estimate the pose of the camera in the current frame, and then iteratively perform volume data fusion through the surface implicit function (Truncated Signed Distance Function, TSDF) to obtain a dense 3D model. Although the depth acquired by Kinect is not affected by lighting conditions and texture richness, its depth data range is only 0.5-4m, and the position and size of the grid model are fixed, so it is only suitable for local and static indoor scenes.

The 3D reconstruction of indoor scenes based on consumer-grade depth cameras generally has the following problems: (1) The resolution of depth images obtained by consumer-grade depth cameras is small and the noise is large, making it difficult to maintain the surface details of objects, and the range of depth values is limited and cannot be used directly. 3D reconstruction of complete scenes; (2) The cumulative error generated by camera pose estimation will cause errors and distorted 3D models; (3) Consumer-grade depth cameras are generally hand-held for shooting, and the movement state of the camera is relatively random. There are good and bad, which affect the reconstruction effect.

In order to perform a complete 3D reconstruction of indoor scenes, Whelan et al. proposed the Kintinuous algorithm, which is a further extension of KinectFusion. The algorithm uses ShiftingTSDFVolume to recycle video memory to solve the problem of grid model memory consumption during large scene reconstruction, and uses DBoW to find matching key frames for closed-loop detection, and finally optimizes the pose graph and model to obtain a 3D model of the large scene . Choi et al. proposed the idea of Elastic Fragment. First, the RGBD data stream is segmented every 50 frames, and the visual odometry is estimated for each segment separately, and the geometric descriptor FPFH is extracted from the point cloud data between two segments to find a match. Closed-loop detection, and then introduce line processes constraints to optimize the detection results, remove wrong closed-loops, and finally use the optimized odometer information for volume data fusion. Indoor complete scene reconstruction is achieved through segmentation processing and loop closure detection, but the local geometric details of objects are not considered, and this method of fixed segmentation is not robust for real indoor scene reconstruction. Zeng et al. proposed the concept of 3D Match descriptors. This algorithm first processes the RGBD data stream into fixed segments and reconstructs the local model, and extracts key points from the 3D model of each segment as the 3D convolutional network (ConvNet). Input, use the feature vector learned by the network as the input of another matrix network (Metric network), and output the matching result through similarity comparison. Since the deep network has a very obvious feature learning advantage, using 3D Match for geometric registration can improve the reconstruction accuracy compared to other descriptors. However, this method needs to perform local 3D reconstruction first, use the deep learning network to do geometric registration, and then output the global 3D model, and network training requires a large amount of data, so the efficiency of the entire reconstruction process is low.

In terms of improving the accuracy of 3D reconstruction, Angela et al. proposed the VSBR algorithm. The main idea is to use the Shape from Shading (SFS) technology to optimize the TSDF data layered and then fuse them to solve the problem of excessive TSDF data fusion. Smoothing leads to the loss of details on the surface of objects, so that a more refined three-dimensional structure model can be obtained. However, this method is only effective for single-body reconstruction under ideal light sources, and the accuracy of indoor scenes is not significantly improved due to large changes in light sources.

In view of this, the present invention is proposed.

Contents of the invention

In order to solve the above problems in the prior art, that is, to solve the technical problem of how to improve the accuracy of 3D reconstruction in indoor scenes, a method and system for 3D reconstruction of complete indoor scenes based on consumer-grade depth cameras are provided.

In order to achieve the above purpose, on the one hand, the following technical solutions are provided:

A method for 3D reconstruction of a complete indoor scene based on a consumer-grade depth camera, the method may include:

Get the depth image;

performing adaptive bilateral filtering on the depth image;

Perform block fusion and registration processing based on visual content on the filtered depth image;

According to the processing results, the weighted volume data fusion is carried out to reconstruct the 3D model of the complete indoor scene.

Preferably, the performing adaptive bilateral filtering on the depth image specifically includes:

Adaptive bilateral filtering is performed according to the following formula:

Wherein, the u and the u _k respectively represent any pixel on the depth image and its domain pixels; the Z(u) and the Z(u _k ) respectively represent the pixels corresponding to the u and the u The depth value of _k ; the Indicates the corresponding depth value after filtering; the W indicates that in the field The normalization factor on ; the w _s and the w _c respectively represent the Gaussian kernel function filtered in the spatial domain and the value domain.

Preferably, the Gaussian kernel function filtered in the spatial domain and the value domain is determined according to the following formula:

Wherein, described δ _s and described δ _c are respectively the variance of space domain and range Gaussian kernel function;

Wherein, described δ _s and described δ _c are determined according to the following formula:

Wherein, the f represents the focal length of the depth camera, and the K _s and the K _c represent constants.

Preferably, performing block fusion and registration processing on the filtered depth image based on visual content specifically includes: segmenting the depth image sequence based on visual content, and performing block fusion on each segment, and the Perform closed-loop detection between the above segments, and perform global optimization on the results of closed-loop detection.

Preferably, the step of segmenting the depth image sequence based on the visual content, performing block fusion for each segment, and performing closed-loop detection between the segments, and performing global optimization on the result of the closed-loop detection specifically includes:

Segment the depth image sequence based on the automatic segmentation method of visual content detection, divide similar depth image content into one segment, and perform block fusion for each segment, and determine the transformation relationship between the depth images , and perform closed-loop detection between segments according to the transformation relationship, so as to achieve global optimization.

Preferably, the automatic segmentation method based on visual content detection segments the depth image sequence, divides similar depth image content into one segment, and performs block fusion for each segment to determine the depth image The transformation relationship between them, and perform closed-loop detection between segments according to the transformation relationship to achieve global optimization, specifically including:

Use the Kintinuous framework to estimate the visual odometry and obtain the camera pose information under each frame of the depth image;

According to the camera pose information, back-project the point cloud data corresponding to the depth image of each frame into the initial coordinate system, compare the similarity between the depth image obtained after projection and the depth image of the initial frame, and calculate the similarity When it is lower than the similarity threshold, initialize the camera pose and perform segmentation;

Extract the PFFH geometric descriptor in each segment point cloud data, and perform rough registration between each two segments, and use the GICP algorithm for fine registration to obtain the matching relationship between segments;

Using the pose information of each segment and the matching relationship between the segments, construct a graph and use the G2O framework to optimize the graph to obtain optimized camera trajectory information, thereby realizing the global optimization.

Preferably, according to the camera pose information, the point cloud data corresponding to each frame of depth image is back-projected into the initial coordinate system, and the similarity is compared between the depth image obtained after projection and the depth image of the initial frame , and when the similarity is lower than the similarity threshold, initialize the camera pose and perform segmentation, including:

Step 1: Calculate the similarity between the depth image of each frame and the depth image of the first frame;

Step 2: judging whether the similarity is lower than a similarity threshold;

Step 3: If yes, segment the depth image sequence;

Step 4: Use the next frame depth image as the starting frame depth image of the next segment, and repeat step 1 and step 2 until all frame depth images are processed.

Preferably, said step 1 specifically includes:

According to the projection relationship and the depth value of any frame depth image, and use the following formula to calculate the first space three-dimensional point corresponding to each pixel on the depth image:

p=π ^-1 (u _p ,Z(u _p ))

Wherein, the up is any pixel on the depth image; the Z(up) and the _p represent the depth value corresponding to the up and the three- _dimensional point in the first space, respectively; the _π represent the projection relationship;

According to the following formula, the rotation and translation transformation of the three-dimensional point in the first space is transformed into the world coordinate system to obtain the three-dimensional point in the second space:

q=T _i p

Wherein, the T _i represents the rotation and translation matrix of the i-th frame depth map corresponding to the three-dimensional point in the space to the world coordinate system; the p represents the three-dimensional point in the first space, and the q represents the three-dimensional point in the second space; The i takes a positive integer;

The three-dimensional point in the second space is back-projected to the two-dimensional image plane according to the following formula to obtain the projected depth image:

Wherein, the u _q is the pixel on the projected depth image corresponding to the q; the f _x , the f _y , the c _x and the _cy represent the internal reference of the depth camera; the x _q , y _q and z _q represent the coordinates of the q; the T represents the transposition of the matrix;

The number of effective pixels on the depth image of the initial frame and the projected depth image of any frame are respectively calculated, and the ratio of the two is used as the similarity.

Preferably, performing weighted volume data fusion according to the processing results, so as to reconstruct the 3D model of the complete indoor scene specifically includes: according to the processing results, using the truncated signed distance function grid model to fuse the depth images of each frame, and using voxel The grid is used to represent the 3D space, so as to obtain the 3D model of the complete indoor scene.

Preferably, according to the processing result, the depth images of each frame are fused using a truncated signed distance function grid model, and a voxel grid is used to represent a three-dimensional space, thereby obtaining a three-dimensional model of a complete indoor scene, specifically including:

Based on the noise characteristics and the region of interest model, the weighted fusion of the truncated signed distance function data is performed using the Volumetric method framework;

The Marching cubes algorithm is used to extract the Mesh model, so as to obtain the 3D model of the complete indoor scene.

Preferably, the truncated signed distance function is determined according to the following formula:

f _i (v)＝[K ^-1 z _i (u)[u ^T ,1] ^T ] _z -[v _i ] _z

Among them, f _i (v) represents the truncated sign distance function, that is, the distance from the grid to the surface of the object model, the positive and negative indicate whether the grid is on the side where the surface is occluded or on the visible side, and the zero-crossing point is the distance on the surface point; the K represents the internal parameter matrix of the camera; the u represents a pixel; the z _i (u) represents the depth value corresponding to the pixel u; the v _i represents a voxel.

Preferably, the data weighted fusion is performed according to the following formula:

Wherein, the v represents a voxel; the f _i (v) and the w _i (v) respectively represent a truncated signed distance function and a weight function corresponding to the voxel v; the n is a positive integer; The F(v) represents the truncated signed distance function value corresponding to the voxel v after fusion; the W(v) represents the weight of the truncated signed distance function value corresponding to the voxel v after fusion;

Wherein, the weight function can be determined according to the following formula:

Wherein, the d _i represents the radius of the region of interest; the δ _s is the noise variance in the depth data; the w is a constant.

In order to achieve the above purpose, on the other hand, a system for 3D reconstruction of a complete indoor scene based on a consumer-grade depth camera is also provided, the system includes:

Obtaining module, used for obtaining depth image;

A filtering module, configured to perform adaptive bilateral filtering on the depth image;

The block fusion and registration module is used to perform block fusion and registration processing based on visual content on the filtered depth image;

The volume data fusion module is used to perform weighted volume data fusion according to the processing results, so as to reconstruct the 3D model of the complete indoor scene.

Preferably, the filtering module is specifically used for:

Adaptive bilateral filtering is performed according to the following formula:

Wherein, the u and the u _k respectively represent any pixel on the depth image and its domain pixels; the Z(u) and the Z(u _k ) respectively represent the pixels corresponding to the u and the u The depth value of _k ; the Indicates the corresponding depth value after filtering; the W indicates that in the field The normalization factor on ; the w _s and the w _c respectively represent the Gaussian kernel function filtered in the spatial domain and the value domain.

Preferably, the block fusion and registration module can be specifically configured to: segment the depth image sequence based on visual content, perform block fusion for each segment, and perform loop closure detection between the segments, and The results of loop closure detection are globally optimized.

Preferably, the block fusion and registration module can also be specifically used for:

Segmenting the depth image sequence based on the automatic segmentation method of visual content detection, dividing similar depth image content into one segment, performing block fusion for each segment, and determining the transformation relationship between the depth images, And perform closed-loop detection between segments according to the transformation relationship, so as to realize global optimization.

Preferably, the block fusion and registration module specifically includes:

The camera pose information acquisition unit is used to use the Kintinuous framework to perform visual odometry estimation and obtain the camera pose information under each frame of depth image;

The segmentation unit is used to back-project the point cloud data corresponding to the depth image of each frame into the initial coordinate system according to the camera pose information, and use the depth image obtained after projection to perform similarity with the depth image of the initial frame Compare, and when the similarity is lower than the similarity threshold, initialize the camera pose and perform segmentation;

The registration unit is used to extract the PFFH geometric descriptor in each segment point cloud data, and perform rough registration between each two segments, and use the GICP algorithm for fine registration to obtain the matching between segments relation;

An optimization unit is configured to use the pose information of each segment and the matching relationship between the segments to construct a graph and optimize the graph using the G2O framework to obtain optimized camera trajectory information, thereby realizing the global optimization .

Preferably, the segmentation unit specifically includes:

A calculation unit, configured to calculate the similarity between the depth image of each frame and the depth image of the first frame;

a judging unit, configured to judge whether the similarity is lower than a similarity threshold;

a segmentation subunit, configured to segment the sequence of depth images when the similarity is lower than a similarity threshold;

The processing unit is configured to use the next frame of depth image as the starting frame of depth image of the next segment, and repeatedly execute the calculation unit and the judging unit until all the frame depth images are processed.

Preferably, the volume data fusion module is specifically configured to: according to the processing result, use the truncated signed distance function grid model to fuse the depth images of each frame, and use the voxel grid to represent the three-dimensional space, so as to obtain the complete indoor scene 3D model.

Preferably, the volume data fusion module specifically includes:

A weighted fusion unit is used to perform the weighted fusion of the truncated signed distance function data using the Volumetric method framework based on the noise characteristics and the region of interest;

The extraction unit is configured to extract the Mesh model by using the Marching cubes algorithm, so as to obtain the 3D model of the complete indoor scene.

Embodiments of the present invention provide a method and system for three-dimensional reconstruction of a complete indoor scene based on a consumer-grade depth camera. Among them, the method includes acquiring a depth image; performing adaptive bilateral filtering on the depth image; performing block fusion and registration processing based on visual content on the filtered depth image; performing weighted volume data fusion according to the processing results, thereby reconstructing indoor 3D model of the complete scene. The embodiment of the present invention can effectively reduce the cumulative error in visual odometry estimation and improve the registration accuracy by performing block fusion and registration based on visual content on the depth image, and also adopts a weighted volume data fusion algorithm, which can effectively The geometric details of the surface of the object are maintained, thereby solving the technical problem of how to improve the accuracy of 3D reconstruction in indoor scenes, so that a complete, accurate and refined indoor scene model can be obtained.

Description of drawings

FIG. 1 is a schematic flowchart of a method for three-dimensional reconstruction of a complete indoor scene based on a consumer-grade depth camera according to an embodiment of the present invention;

Fig. 2a is a color image corresponding to a depth image according to an embodiment of the present invention;

Fig. 2b is a schematic diagram of a point cloud obtained from a depth image according to an embodiment of the present invention;

2c is a schematic diagram of a point cloud obtained by performing bilateral filtering on a depth image according to an embodiment of the present invention;

Fig. 2d is a schematic diagram of a point cloud obtained by performing adaptive bilateral filtering on a depth image according to an embodiment of the present invention

FIG. 3 is a schematic flow diagram of segmentation fusion and registration based on visual content according to an embodiment of the present invention;

4 is a schematic diagram of a weighted volume data fusion process according to an embodiment of the present invention;

Figure 5a is a schematic diagram of the 3D reconstruction result using the unweighted volume data fusion algorithm;

Fig. 5b is a schematic diagram of local details of the three-dimensional model in Fig. 5a;

FIG. 5c is a schematic diagram of a three-dimensional reconstruction result obtained by a weighted volume data fusion algorithm proposed according to an embodiment of the present invention;

Fig. 5d is a schematic diagram of local details of the three-dimensional model in Fig. 5c;

6 is a schematic diagram of the effect of three-dimensional reconstruction using the method proposed by the embodiment of the present invention on the 3D Scene Data dataset according to the embodiment of the present invention;

7 is a schematic diagram of the effect of three-dimensional reconstruction using the method proposed in the embodiment of the present invention on the Augmented ICL-NUIM Dataset dataset according to an embodiment of the present invention;

8 is a schematic diagram showing the effect of three-dimensional reconstruction using indoor scene data collected by Microsoft Kinect for Windows according to an embodiment of the present invention;

9 is a schematic structural diagram of a system for performing three-dimensional reconstruction of a complete indoor scene based on a consumer-grade depth camera according to an embodiment of the present invention.

detailed description

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention, and are not intended to limit the protection scope of the present invention.

An embodiment of the present invention provides a method for three-dimensional reconstruction of a complete indoor scene based on a consumer-grade depth camera. As shown in Figure 1, the method includes:

S100: Acquire a depth image.

Specifically, this step may include: using a consumer-grade depth camera based on the principle of structured light to acquire a depth image.

Among them, consumer-grade depth cameras (Microsoft Kinect for Windows and Xtion, referred to as depth cameras) based on the principle of structured light acquire depth data of depth images by emitting structured light and receiving reflection information.

In practical applications, real indoor scene data can be collected using a handheld consumer-grade depth camera Microsoft Kinect for Windows.

Depth data can be calculated according to the following formula:

Among them, f represents the focal length of the consumer-grade depth camera; B represents the baseline; D represents the disparity.

S110: Perform adaptive bilateral filtering on the depth image.

In this step, adaptive bilateral filtering is performed on the obtained depth image by using the noise characteristics of the consumer-grade depth camera based on the principle of structured light.

Among them, the adaptive bilateral filtering algorithm refers to performing filtering on both the spatial domain and the value domain of the depth image.

In practical applications, the parameters of the adaptive bilateral filtering algorithm can be set according to the noise characteristics of the depth camera and its internal parameters, which can effectively remove noise and preserve edge information.

For the partial derivative of the depth Z with respect to the parallax D, there is the following relationship:

The noise of the depth data is mainly generated in the quantization process. From the above formula, it can be seen that the variance of the depth noise is proportional to the square of the depth value, that is to say, the greater the depth value, the greater the noise. In order to effectively remove the noise in the depth image, the embodiment of the present invention defines a filtering algorithm based on this noise characteristic.

Specifically, the above adaptive bilateral filtering can be performed according to the following formula:

Among them, u and u _k respectively represent any pixel on the depth image and its domain pixels; Z(u) and Z(u _k ) represent the depth values corresponding to u and u _k respectively; Indicates the corresponding depth value after filtering; W indicates that in the field The normalization factor on ; w _s and w _c represent the Gaussian kernel function filtered in the spatial domain and the value domain, respectively.

In the above embodiment, w _s and w _c can be determined according to the following formula:

Among them, δ _s and δ _c are the variance of Gaussian kernel function in space domain and value domain respectively.

δ _s and δ _c are related to the depth value, and their values are not fixed.

Specifically, in the above-mentioned embodiment, δ _s and δ _c can be determined according to the following formula:

Among them, f represents the focal length of the depth camera, K _s and K _c represent constants, and their specific values are related to the parameters of the depth camera.

Figures 2a-d exemplarily show the effect comparison diagrams of different filtering algorithms. Wherein, Fig. 2a shows a color image corresponding to a depth image. Figure 2b shows the point cloud obtained from the depth image. Figure 2c shows the point cloud resulting from bilateral filtering of the depth image. Figure 2d shows the point cloud obtained by adaptive bilateral filtering on the depth image.

In the embodiment of the present invention, by adopting an adaptive bilateral filtering method, edge preservation and denoising of the depth map can be realized.

S120: Perform block fusion and registration processing based on visual content on the depth image.

In this step, the depth image sequence is segmented based on the visual content, and each segment is fused in blocks, and the closed-loop detection is performed between the segments, and the result of the closed-loop detection is globally optimized. Wherein, the depth image sequence is a depth image data stream.

Preferably, this step may include: determining the transformation relationship between the depth images, segmenting the depth image sequence based on the method of automatic segmentation of visual content, dividing similar depth image contents into one segment, and dividing each segment Segments are merged into blocks, the transformation relationship between depth images is determined, and loop closure detection is performed between segments according to the transformation relationship, and global optimization is realized.

Further, this step may include:

S121: Use the Kintinuous framework to perform visual odometry estimation to obtain camera pose information under each frame of depth image.

S122: According to the camera pose information, back-project the point cloud data corresponding to each frame of depth image into the initial coordinate system, compare the similarity between the depth image obtained after projection and the depth image of the initial frame, and when the similarity is lower than When the similarity threshold is reached, the camera pose is initialized for segmentation.

S123: Extract the PFFH geometric descriptor in each segment point cloud data, perform rough registration between every two segments, and use the GICP algorithm to perform fine registration to obtain the matching relationship between segments.

In this step, a closed-loop detection is performed between segments.

S124: Use the pose information of each segment and the matching relationship between segments to construct a graph and use the G2O framework to optimize the graph to obtain optimized camera trajectory information, thereby achieving global optimization.

In this step, the (Simultaneous Localization and Calibration, SLAC) mode is applied to improve non-rigid distortion during optimization, and line processes constraints are introduced to delete wrong closed-loop matching.

The above step S122 may also specifically include:

S1221: Calculate the similarity between each frame of depth image and the first frame of depth image.

S1222: Determine whether the similarity is lower than the similarity threshold, if yes, execute step S1223; otherwise, execute step S1224.

S1223: Segment the depth image sequence.

In this step, the depth image sequence is segmented based on the visual content. This can not only effectively solve the problem of cumulative error generated by visual odometry estimation, but also fuse similar content together to improve registration accuracy.

S1224: Do not segment the depth image sequence.

S1225: Use the next frame depth image as the starting frame depth image of the next segment, and repeat step S1221 and step S1222 until all frame depth images are processed.

In the above embodiment, the step of calculating the similarity between each frame of depth image and the first frame of depth image may specifically include:

S12211: According to the projection relationship and the depth value of any frame of depth image, and use the following formula to calculate the first space three-dimensional point corresponding to each pixel on the depth image:

p=π ^-1 (u _p ,Z(u _p ))

Among them, u _p is any pixel on the depth image; Z(u _p ) and p represent the depth value corresponding to up _p and the 3D point in the first space, respectively; π represents the projection relationship, that is, the point cloud data corresponding to each frame of depth image Backprojection to the 2D-3D projection transformation relationship in the initial coordinate system.

S12212: Transform the rotation and translation of the 3D point in the first space into the world coordinate system according to the following formula to obtain the 3D point in the second space:

q=T _i p

Among them, T _i represents the rotation and translation matrix from the three-dimensional point in the i-th frame depth map to the world coordinate system, which can be estimated by visual odometry; i takes a positive integer; p represents the first three-dimensional point in space, and q represents the second A three-dimensional point in space, the coordinates of p and q are:

p = (x _p , y _p , z _p ), q = (x _q , y _q , z _q ).

S12213: Back-project the 3D point in the second space onto the 2D image plane according to the following formula to obtain the projected depth image:

Among them, u _q is the pixel on the projected depth image corresponding to q; f _x , f _y , c _x and _cy represent the internal parameters of the depth camera; x _q , y _q , z _q represent the coordinates of q; T represents the matrix Transpose.

S12214: Calculate the number of effective pixels on the initial frame depth image and the projected depth image of any frame respectively, and use the ratio of the two as the similarity.

For example, the similarity is calculated according to the following formula:

Among them, n ⁰ and ⁿⁱ represent the number of effective pixels on the depth image of the initial frame and the projected depth image of any frame, respectively; ρ represents the similarity.

Fig. 3 exemplarily shows a flow chart of segmentation fusion and registration based on visual content.

The embodiment of the present invention adopts an automatic segmentation algorithm based on visual content, which can effectively reduce the cumulative error in visual odometry estimation and improve registration accuracy.

S130: Perform weighted volume data fusion according to the processing result, so as to reconstruct the 3D model of the complete indoor scene.

Specifically, this step may include: according to the block fusion and registration processing results based on visual content, using the truncated signed distance function (TSDF) grid model to fuse the depth images of each frame, and using the voxel grid to represent the three-dimensional space , so as to obtain the 3D model of the complete indoor scene.

This step can further include:

S131: Based on the noise characteristics and the region of interest, use the Volumetric method framework to perform weighted fusion of truncated signed distance function data.

S132: Using the Marching cubes algorithm to extract the Mesh model.

In practical applications, the TSDF grid model can be used to fuse the depth images of each frame according to the estimation results of the visual odometry. A voxel grid with a resolution of m can be used to represent the three-dimensional space, that is, each three-dimensional space is divided into m blocks , each grid v stores two values: the truncated signed distance function f _i (v) and its weight w _i (v).

Among them, the truncated signed distance function can be determined according to the following formula:

f _i (v)＝[K ^-1 z _i (u)[u ^T ,1] ^T ] _z -[v _i ] _z

Among them, f _i (v) represents the truncated sign distance function, that is, the distance from the grid to the surface of the object model, the positive and negative indicate whether the grid is on the side where the surface is occluded or on the visible side, and the zero-crossing point is the distance on the surface point; K represents the internal parameter matrix of the camera; u represents a pixel; z _i (u) represents the depth value corresponding to pixel u; v _i represents a voxel. Wherein, the camera may be a depth camera or a depth camera.

Among them, data weighted fusion can be performed according to the following formula:

Among them, f _i (v) and w _i (v) represent the truncated signed distance function (TSDF) and its weight function corresponding to voxel v respectively; n takes a positive integer; F(v) represents the fused voxel corresponding to The truncated signed distance function value of ; W(v) represents the weight of the truncated signed distance function value corresponding to voxel v after fusion.

In the above embodiments, the weight function may be determined according to the noise characteristics of the depth data and the region of interest, and its value is not fixed. In order to maintain the geometric details of the object surface, the weight of the area with low noise and the area of interest is set to be large, and the weight of the area with large noise or the area of no interest is set to be small.

Specifically, the weight function can be determined according to the following formula:

Among them, d _i represents the radius of the region of interest, the smaller the radius, the more interested it is, and the greater the weight; δ _s is the noise variance in the depth data, and its value is consistent with the variance of the spatial domain kernel function of the adaptive bilateral filtering algorithm; w is a constant, preferably, it can take a value of 1 or 0.

Fig. 4 exemplarily shows a schematic diagram of a weighted volume data fusion process.

In the embodiment of the present invention, the weighted volume data fusion algorithm can effectively maintain the geometric details of the object surface, and can obtain a complete, accurate, and refined indoor scene model, which has good robustness and scalability.

Fig. 5a exemplarily shows the 3D reconstruction result using the unweighted volume data fusion algorithm; Fig. 5b exemplarily shows the local details of the 3D model in Fig. 5a; Fig. 5c exemplarily shows The 3D reconstruction results obtained by the proposed weighted volume data fusion algorithm; Figure 5d exemplarily shows the local details of the 3D model in Figure 5c.

Fig. 6 exemplarily shows the schematic diagram of the effect of three-dimensional reconstruction using the method proposed by the embodiment of the present invention on the 3D Scene Data dataset; Fig. 7 exemplarily shows the use of the present invention on the Augmented ICL-NUIM Dataset dataset A schematic diagram of the effect of three-dimensional reconstruction by the method proposed in the embodiment; FIG. 8 exemplarily shows a schematic diagram of the effect of three-dimensional reconstruction using indoor scene data collected by Microsoft Kinect for Windows.

It should be pointed out that although the embodiments of the present invention are described in the above order, those skilled in the art can understand that the present invention can also be implemented in a sequence different from that described here, and these simple changes should also be included in this document. within the scope of protection of the invention.

Based on the same technical concept as the method embodiment, the embodiment of the present invention also provides a system for 3D reconstruction of a complete indoor scene based on a consumer-grade depth camera. As shown in FIG. 9 , the system 90 includes: an acquisition module 92 and a filtering module 94 , block fusion and registration module 96 and volume data fusion module 98. Wherein, the obtaining module 92 is used for obtaining a depth image. The filtering module 94 is used for performing adaptive bilateral filtering on the depth image. The block fusion and registration module 96 is used to perform block fusion and registration processing based on visual content on the filtered depth image. The volume data fusion module 98 is used to perform weighted volume data fusion according to the processing results, so as to reconstruct the 3D model of the complete indoor scene.

The embodiments of the present invention can effectively reduce the cumulative error in visual odometry estimation by adopting the above technical solution, and improve the registration accuracy, can effectively maintain the geometric details of the object surface, and can obtain a complete, accurate and refined indoor scene model .

In some embodiments, the filtering module is specifically configured to: perform adaptive bilateral filtering according to the following formula:

Among them, u and u _k respectively represent any pixel on the depth image and its domain pixels; Z(u) and Z(u _k ) represent the depth values corresponding to u and u _k respectively; Indicates the corresponding depth value after filtering; W indicates that in the field The normalization factor on ; w _s and w _c represent the Gaussian kernel function filtered in the spatial domain and the value domain, respectively.

In some embodiments, the block fusion and registration module can be specifically used to: segment the depth image sequence based on visual content, perform block fusion for each segment, and perform closed-loop detection between segments, and perform closed-loop detection The detection results are globally optimized.

In some other embodiments, the block fusion and registration module can also be specifically used to: determine the transformation relationship between depth images, segment the depth image sequence based on the automatic segmentation method of visual content detection, and divide similar depth images The content is divided into a segment, and each segment is fused in blocks to determine the transformation relationship between depth images, and perform closed-loop detection between segments according to the transformation relationship, and achieve global optimization.

In some preferred embodiments, the block fusion and registration module may specifically include: a camera pose information acquisition unit, a segmentation unit, a registration unit and an optimization unit. Among them, the camera pose information acquisition unit is used to use the Kintinuous framework to perform visual odometry estimation and obtain the camera pose information under each frame of depth image. The segmentation unit is used to back-project the point cloud data corresponding to the depth image of each frame into the initial coordinate system according to the camera pose information, and compare the similarity between the depth image obtained after projection and the depth image of the initial frame, and when similar When the degree is lower than the similarity threshold, the camera pose is initialized for segmentation. The registration unit is used to extract the PFFH geometric descriptor in each segment point cloud data, and perform rough registration between each two segments, and use the GICP algorithm for fine registration to obtain the matching relationship between segments . The optimization unit is used to use the pose information of each segment and the matching relationship between segments to construct a graph and use the G2O framework for graph optimization to obtain optimized camera trajectory information, thereby achieving global optimization.

Wherein, the segmentation unit may specifically include: a computing unit, a judging unit, a segmentation subunit, and a processing unit. Wherein, the calculation unit is used to calculate the similarity between each frame of depth image and the first frame of depth image. The judging unit is used for judging whether the similarity is lower than a similarity threshold. The segmentation subunit is used to segment the depth image sequence when the similarity is lower than the similarity threshold. The processing unit is used to use the next frame of depth image as the starting frame of depth image of the next segment, and repeatedly execute the calculation unit and the judging unit until all frames of depth images are processed.

In some embodiments, the volume data fusion module can be specifically used to fuse the depth images of each frame by using the truncated signed distance function grid model according to the processing results, and use the voxel grid to represent the three-dimensional space, so as to obtain the three-dimensional space of the complete indoor scene. Model.

In some embodiments, the volume data fusion module may specifically include a weighted fusion unit and an extraction unit. Among them, the weighted fusion unit is used to perform weighted fusion of the truncated sign distance function data based on the noise characteristics and the region of interest, using the Volumetric method framework. The extraction unit is used to extract the Mesh model by using the Marching cubes algorithm, so as to obtain the 3D model of the complete indoor scene.

The present invention will be described in detail below with a preferred embodiment.

The system for 3D reconstruction of indoor complete scenes based on consumer-grade depth cameras includes an acquisition module, a filtering module, a block fusion and registration module, and a volume data fusion module. in:

The collection module is used for collecting depth images of indoor scenes by using a depth camera.

The filtering module is used to perform adaptive bilateral filtering on the obtained depth image.

The collection module is an equivalent replacement of the above acquisition module. In practical applications, real indoor scene data can be collected using a handheld consumer-grade depth camera Microsoft Kinect for Windows. Then, adaptive bilateral filtering is performed on the collected depth image, and the parameters in the adaptive bilateral filtering method are automatically set according to the noise characteristics of the depth camera and its internal parameters, so the embodiment of the present invention can effectively remove noise and retain edge information .

The block fusion and registration module is used to automatically segment the data stream based on the visual content, perform block fusion for each segment, perform closed-loop detection between segments, and perform global optimization on the results of closed-loop detection.

The block fusion and registration module performs automatic block fusion and registration based on visual content.

In a more preferred embodiment, the block fusion and registration module specifically includes: a pose information acquisition module, a segmentation module, a rough registration module, a fine registration module and an optimization module. Among them, the pose information acquisition module is used to use the Kintinuous framework to perform visual odometry estimation and obtain the camera pose information under each frame of depth image. The segmentation module is used to back-project the point cloud data corresponding to each frame of depth image to the initial coordinate system according to the camera pose information, and compare the similarity between the projected depth image and the depth image of the initial frame. If the similarity is lower than The similarity threshold initializes the camera pose and performs new segmentation. The coarse registration module is used to extract the PFFH geometric descriptor in each segment point cloud data, and perform coarse registration between each two segments; the fine registration module is used to use the GICP algorithm for fine registration to obtain The matching relationship between segments. The optimization module is used to use the pose information of each segment and the matching relationship between segments to construct a graph and use the G2O framework for graph optimization.

Preferably, the above optimization module is further used to apply SLAC (Simultaneous Localization and Calibration) mode to optimize non-rigid distortion, and use line processes constraints to delete wrong closed-loop matching.

The above block fusion and registration module processes RGBD data streams in segments based on visual content, which can not only effectively solve the problem of cumulative error generated by visual odometry estimation, but also fuse similar content together, thereby improving registration precision.

The volume data fusion module is used to perform weighted volume data fusion according to the optimized camera trajectory information to obtain a 3D model of the scene.

The volume data fusion module defines the weight function of the truncated sign distance function according to the noise characteristics of the depth camera and the region of interest, so as to maintain the geometric details of the object surface.

Experiments on the system of 3D reconstruction of indoor complete scenes based on consumer-grade depth cameras show that: the high-precision 3D reconstruction method based on consumer-grade depth cameras can obtain a complete, accurate and refined indoor scene model, and the system has good robustness and scalability.

The above-mentioned system embodiment for 3D reconstruction of a complete indoor scene based on a consumer-grade depth camera can be used to implement a method embodiment for 3D reconstruction of a complete indoor scene based on a consumer-grade depth camera. Its technical principles, technical problems to be solved, and technical effects produced For similarity, reference may be made to each other; for convenience and brevity of description, descriptions of the same parts between the various embodiments are omitted.

It should be noted that when the system and method for 3D reconstruction of a complete indoor scene based on a consumer-grade depth camera provided by the above-mentioned embodiments perform 3D reconstruction of a complete indoor scene, only the division of the above-mentioned functional modules, units or steps is used for illustration. For example, the acquisition module in the foregoing can also be used as an acquisition module. In practical applications, the above-mentioned functions can be allocated by different functional modules, units or steps according to needs, that is, the modules, units or steps in the embodiments of the present invention Then decompose or combine, for example, the acquisition module or the collection and filtering module can be combined into a data preprocessing module.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims (20)

1. A method for performing indoor complete scene three-dimensional reconstruction based on a consumer-grade depth camera, the method comprising:

acquiring a depth image;

performing adaptive bilateral filtering on the depth image;

carrying out block fusion and registration processing based on visual contents on the filtered depth image;

and according to the processing result, performing weighted volume data fusion so as to reconstruct an indoor complete scene three-dimensional model.

2. The method according to claim 1, wherein the adaptive bilateral filtering of the depth image specifically comprises:

adaptive bilateral filtering is performed according to:

wherein said u and said u_kRespectively representing any pixel and a domain pixel thereof on the depth image; the Z (u) and the Z (u)_k) Respectively represent corresponding to the u and the u_kDepth value of (d); the above-mentionedRepresenting the corresponding depth value after filtering; said W represents in the fieldA normalization factor of (a); said w_sAnd said w_cRepresenting the gaussian kernel filtered in the spatial and value domains, respectively.

3. The method of claim 2, wherein the gaussian kernel function for spatial and value domain filtering is determined according to the following equation:

$\left\{\begin{array}{c}{w}_s=\exp (-\frac{{(u-u_k)}^2}{2{\mathrm{\δ}}_s^2})\\ w_c=\exp (-\frac{{\left(Z\right(u)-Z(u_k\left)\right)}^2}{2{\mathrm{\δ}}_c^2})\end{array}\right.$

wherein, the_sAnd said_cThe variance of the spatial domain and the value domain gaussian kernel function respectively;

wherein the sum is determined according to:

$\left\{\begin{array}{c}{\mathrm{\δ}}_s=\frac{K_{s}Z\left(u\right)}{f}\\ {\mathrm{\δ}}_c=K_{c}Z{\left(u\right)}^2\end{array}\right.$

wherein f represents a focal length of the depth camera, K_sAnd said K_cRepresenting a constant.

4. The method according to claim 1, wherein the process of visual content-based block fusion and registration of the filtered depth image comprises in particular: and segmenting the depth image sequence based on visual content, performing block fusion on each segment, performing closed-loop detection between the segments, and performing global optimization on the result of the closed-loop detection.

5. The method according to claim 4, wherein the segmenting the depth image sequence based on the visual content, and performing block fusion on each segment, and performing closed-loop detection between the segments, and performing global optimization on the result of the closed-loop detection specifically comprises:

segmenting a depth image sequence based on an automatic segmentation method for visual content detection, dividing similar depth image contents into segments, performing block fusion on each segment, determining a transformation relation between the depth images, and performing closed-loop detection between the segments according to the transformation relation so as to realize global optimization.

6. The method according to claim 5, wherein the automatic segmentation method based on visual content detection is configured to segment a depth image sequence, segment similar depth image contents into one segment, perform block fusion on each segment, determine a transformation relationship between the depth images, and perform closed-loop detection between segments according to the transformation relationship, so as to implement global optimization, and specifically includes:

estimating a visual odometer by adopting a Kintinuous frame to obtain camera pose information under each frame of depth image;

according to the camera pose information, back projecting the point cloud data corresponding to each frame of depth image to an initial coordinate system, comparing the similarity of the depth image obtained after projection with the depth image of the initial frame, and initializing a camera pose and segmenting when the similarity is lower than a similarity threshold value;

extracting a PFFH geometric descriptor in each segmented point cloud data, performing coarse registration between each two segments, and performing fine registration by adopting a GICP algorithm to obtain a matching relation between the segments;

and constructing a graph by using the pose information of each segment and the matching relation between the segments, and performing graph optimization by using a G2O frame to obtain optimized camera track information, thereby realizing the global optimization.

7. The method according to claim 6, wherein the back-projecting the point cloud data corresponding to each frame of depth image to an initial coordinate system according to the camera pose information, comparing the similarity between the depth image obtained after projection and the depth image of the initial frame, and initializing a camera pose for segmentation when the similarity is lower than a similarity threshold, specifically comprises:

step 1: calculating the similarity between each frame of depth image and a first frame of depth image;

step 2: judging whether the similarity is lower than a similarity threshold value;

and step 3: if yes, segmenting the depth image sequence;

and 4, step 4: and taking the depth image of the next frame as the depth image of the starting frame of the next segment, and repeatedly executing the step 1 and the step 2 until all the depth images of the frames are processed.

8. The method according to claim 7, wherein the step 1 specifically comprises:

according to the projection relation and the depth value of any frame of depth image, calculating a first space three-dimensional point corresponding to each pixel on the depth image by using the following formula:

p＝π^-1(u_p,Z(u_p))

wherein u is_pIs any pixel on the depth image; z (u) as defined above_p) And said p represents said u, respectively_pCorresponding depth values and the first spatial three-dimensional points; the pi represents the projection relation;

and rotationally translating the first space three-dimensional point to a world coordinate system according to the following formula to obtain a second space three-dimensional point:

q＝T_ip

wherein, T is_iRepresenting a rotation translation matrix from the spatial three-dimensional point corresponding to the depth map of the ith frame to a world coordinate system; the p represents the first three-dimensional point in space, and the q represents the second three-dimensional point in space; the i is a positive integer;

and back projecting the second space three-dimensional point to a two-dimensional image plane according to the following formula to obtain a projected depth image:

$u_q={(\frac{f_x{x}_q}{z_q}-c_x,\frac{f_y{y}_q}{z_q}-c_y)}^T$

wherein u is_qIs the pixel on the projected depth image corresponding to said q; f is_xThe above-mentioned f_yC to c_xAnd c is as described_yRepresenting an internal reference of the depth camera; said x_q、y_q、z_qCoordinates representing said q; the T represents a transpose of a matrix;

and respectively calculating the number of effective pixels on the depth image of the initial frame and the depth image projected by any frame, and taking the ratio of the two as the similarity.

9. The method according to claim 1, wherein the performing weighted volume data fusion according to the processing result to reconstruct the indoor complete scene three-dimensional model specifically comprises: and according to the processing result, fusing the depth image of each frame by using a truncated symbolic distance function grid model, and representing a three-dimensional space by using a voxel grid so as to obtain an indoor complete scene three-dimensional model.

10. The method according to claim 9, wherein, according to the processing result, fusing the depth image of each frame by using a truncated symbolic distance function mesh model, and representing a three-dimensional space by using a voxel mesh, thereby obtaining a three-dimensional model of an indoor complete scene, specifically comprising:

based on the noise characteristic and the interest region, performing weighted fusion on the truncated symbol distance function data by using a Volumetric method frame;

and (4) extracting a Mesh model by adopting a Marching cubes algorithm, thereby obtaining the indoor complete scene three-dimensional model.

11. The method according to claim 9 or 10, wherein the truncated symbol distance function is determined according to the following equation:

f_i(v)＝[K^-1z_i(u)[u^T,1]^T]_z-[v_i]_z

wherein f is_i(v) Representing a truncated sign distance function, namely the distance from the grid to the surface of the object model, wherein the positive and negative represent that the grid is positioned on the shielded side or the visible side of the surface, and the zero-crossing point is a point on the surface; the K represents an intrinsic parameter matrix of the camera; the u represents a pixel; z is_i(u) representing a depth value corresponding to the pixel u; v is_iThe voxels are represented.

12. The method of claim 10, wherein the data weighted fusion is performed according to the following equation:

$\left\{\begin{array}{c}F\left(v\right)=\frac{{\mathrm{\Σ}}_{i=1}^n{f}_i\left(v\right)w_i\left(v\right)}{W\left(v\right)}\\ W\left(v\right)={\mathrm{\Σ}}_{i=1}^n{w}_i\left(v\right)\end{array}\right.$

wherein v represents a voxel; f is_i(v) And said w_i(v) Respectively representing a truncated symbol distance function and a weight function thereof corresponding to the voxel v; the n is a positive integer; f (v) represents a truncated sign distance function value corresponding to the voxel v after fusion; w (v) represents the weight of the truncated sign distance function value corresponding to the voxel v after fusion;

wherein the weight function may be determined according to the following equation:

$w_i\left(v\right)=\left\{\begin{array}{cc}\frac{e^{\frac{-d_i^2}{2{\mathrm{\&delta;}}_s^2}}}{z_i^4},& 0<z_i<2.8\\ w,& z_i\&GreaterEqual;2.8\end{array}\right.$

wherein d is_iA radius representing the region of interest; the above-mentioned_sIs the noise variance in the depth data; w is a constant.

13. A system for three-dimensional reconstruction of a complete scene indoors based on a consumer-grade depth camera, the system comprising:

the acquisition module is used for acquiring a depth image;

the filtering module is used for carrying out self-adaptive bilateral filtering on the depth image;

the block fusion and registration module is used for carrying out block fusion and registration processing based on visual content on the filtered depth image;

and the volume data fusion module is used for carrying out weighted volume data fusion according to the processing result so as to reconstruct an indoor complete scene three-dimensional model.

14. The system of claim 13, wherein the filtering module is specifically configured to:

adaptive bilateral filtering is performed according to:

wherein said u and said u_kRespectively representing any pixel and a domain pixel thereof on the depth image; the Z (u) and the Z (u)_k) Respectively represent corresponding to the u and the u_kDepth value of (d); the above-mentionedRepresenting the corresponding depth value after filtering; said W represents in the fieldA normalization factor of (a); said w_sAnd said w_cRepresenting the gaussian kernel filtered in the spatial and value domains, respectively.

15. The system of claim 13, wherein the patch fusion and registration module is specifically configured to: and segmenting the depth image sequence based on visual content, performing block fusion on each segment, performing closed-loop detection between the segments, and performing global optimization on the result of the closed-loop detection.

16. The system of claim 15, wherein the patch fusion and registration module is further specific to:

segmenting a depth image sequence based on an automatic segmentation method for visual content detection, dividing similar depth image contents into segments, performing block fusion on each segment, determining a transformation relation between the depth images, and performing closed-loop detection between the segments according to the transformation relation so as to realize global optimization.

17. The system according to claim 16, wherein the block fusion and registration module specifically comprises:

the camera pose information acquisition unit is used for estimating a visual odometer by adopting a Kintinuous frame to obtain camera pose information under each frame of depth image;

the segmentation unit is used for back projecting the point cloud data corresponding to each frame of depth image to an initial coordinate system according to the camera pose information, comparing the similarity of the depth image obtained after projection with the depth image of the initial frame, and initializing a camera pose for segmentation when the similarity is lower than a similarity threshold value;

the registration unit is used for extracting the PFFH geometric descriptor in each segmented point cloud data, performing coarse registration between each two segments, and performing fine registration by adopting a GICP algorithm to obtain the matching relationship between the segments;

and the optimization unit is used for constructing a graph by using the pose information of each segment and the matching relation between the segments, and performing graph optimization by adopting a G2O frame to obtain optimized camera track information so as to realize the global optimization.

18. The system according to claim 17, wherein the segmentation unit comprises in particular:

the calculating unit is used for calculating the similarity between each frame of depth image and the first frame of depth image;

the judging unit is used for judging whether the similarity is lower than a similarity threshold value;

a segmentation subunit, configured to segment the depth image sequence when the similarity is lower than a similarity threshold;

and the processing unit is used for taking the next frame depth image as the starting frame depth image of the next segmentation, and repeatedly executing the calculating unit and the judging unit until all the frame depth images are processed.

19. The system of claim 13, wherein the volumetric data fusion module is specifically configured to: and according to the processing result, fusing the depth image of each frame by using a truncated symbolic distance function grid model, and representing a three-dimensional space by using a voxel grid so as to obtain an indoor complete scene three-dimensional model.

20. The system according to claim 19, wherein the volume data fusion module specifically comprises:

the weighted fusion unit is used for performing weighted fusion on the truncated symbol distance function data by using a Volumetric method frame based on the noise characteristics and the interest region;

and the extraction unit is used for extracting the Mesh model by adopting a Marching cubes algorithm so as to obtain the indoor complete scene three-dimensional model.