CN110070598B - Mobile terminal for 3D scanning reconstruction and 3D scanning reconstruction method thereof - Google Patents
Mobile terminal for 3D scanning reconstruction and 3D scanning reconstruction method thereof Download PDFInfo
- Publication number
- CN110070598B CN110070598B CN201810058604.XA CN201810058604A CN110070598B CN 110070598 B CN110070598 B CN 110070598B CN 201810058604 A CN201810058604 A CN 201810058604A CN 110070598 B CN110070598 B CN 110070598B
- Authority
- CN
- China
- Prior art keywords
- point cloud
- point
- image
- registration
- rgb
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000004927 fusion Effects 0.000 claims abstract description 17
- 238000009877 rendering Methods 0.000 claims abstract description 16
- 239000011159 matrix material Substances 0.000 claims description 83
- 230000006870 function Effects 0.000 claims description 43
- 238000009616 inductively coupled plasma Methods 0.000 claims description 27
- 230000009466 transformation Effects 0.000 claims description 25
- 239000013598 vector Substances 0.000 claims description 21
- 238000013519 translation Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 12
- 238000005259 measurement Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000005516 engineering process Methods 0.000 abstract description 10
- 238000004364 calculation method Methods 0.000 abstract description 9
- 230000008447 perception Effects 0.000 description 6
- 238000013461 design Methods 0.000 description 5
- 238000006073 displacement reaction Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000036039 immunity Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
Abstract
The mobile terminal comprises a structured light laser speckle projection module, an RGB camera, a TR camera, an image acquisition module, an image matching module, a coordinate alignment module, a point cloud registration module, a point cloud fusion module and a volume rendering module, wherein the modules are mutually matched to aim at miniaturization by utilizing the current hardware technology, meanwhile, a part of three-dimensional reconstruction algorithm is realized by utilizing hardware, the complexity of a three-dimensional reconstruction software algorithm is reduced, and the calculation of the algorithm is accelerated by a GPU to realize the real-time scanning reconstruction. The scanning reconstruction method provided by the disclosure improves the practicability of scanning reconstruction; finally, designing the coordinate alignment of the RGB image and the point cloud, so that the reconstruction result has RGB and depth information at the same time; and finally, 3d scanning reconstruction of fusion RGB and depth information in real time in multiple modes on an intelligent mobile terminal platform is realized.
Description
Technical Field
The disclosure relates to the field of mobile communication, image processing technology and integrated circuit technology, in particular to a mobile terminal for 3D scanning reconstruction and a method for 3D scanning reconstruction.
Background
Three-dimensional reconstruction has become a research hotspot and difficulty in the fields of computer vision, computer graphics and the like since the development of the three-dimensional reconstruction, and plays an increasingly important role in the fields of Computer Aided Geometry Design (CAGD), computer animation, medical imaging, digital media, cultural relic repair, human-computer interaction, military application, three-dimensional drawing and the like. Therefore, the research on the three-dimensional reconstruction technology has extremely important significance.
The traditional three-dimensional reconstruction equipment is expensive and not high in portability, three-dimensional data cannot be conveniently acquired in some occasions, and the complexity of the traditional three-dimensional reconstruction algorithm is too high for hardware, so that high-precision real-time three-dimensional reconstruction on a smart phone is difficult to realize. However, due to rapid development of related technologies of smart phones, both hardware and software have qualitative leaps, so that the application of the three-dimensional reconstruction technology to the smart phones becomes possible.
In the prior art, a mobile phone-based scanning device or system adopts a too complex depth perception algorithm, requires complex calibration before scanning, is not the mainstream technology of depth perception at present, does not provide a three-dimensional reconstruction method suitable for a smart phone on the basis of a three-dimensional reconstruction algorithm, and is difficult to realize high-precision three-dimensional reconstruction on a mobile phone platform.
Disclosure of Invention
In order to solve the problems, the present disclosure provides a mobile phone for 3D scanning reconstruction and a method for 3D scanning reconstruction using the mobile phone, which aims to utilize the miniaturization of the current hardware technology, simultaneously reduce the complexity of the three-dimensional reconstruction software algorithm by implementing a part of the three-dimensional reconstruction algorithm using hardware, and enable the scanning reconstruction to implement real-time implementation by accelerating the calculation of the faster algorithm using the GPU; two scanning reconstruction methods are provided, so that the practicability of scanning reconstruction is improved; finally, designing the coordinate alignment of the RGB image and the point cloud, so that the reconstruction result has RGB and depth information at the same time; and finally, 3d scanning reconstruction of fusion RGB and depth information in multiple modes on the smart phone platform is realized in real time. The technical scheme of the disclosure is as follows.
In one aspect, the present disclosure provides a mobile terminal for 3D scanning reconstruction, the mobile terminal including a structured light laser speckle projection module, an RGB camera, a TR camera, an image matching module, a coordinate alignment module, a point cloud registration module, a point cloud fusion module, a volume rendering module; wherein:
the structured light laser speckle projection module configured to: projecting a speckle pattern;
the RGB camera is used for collecting RGB images, and the IR camera is used for collecting laser speckle images;
the image matching module configured to: performing motion matching on the laser speckle image and the reference speckle image, calculating a corresponding depth value according to the offset distance of each pixel point in the laser speckle image, and converting the laser speckle image into a point cloud image; the reference speckle image is solidified in a memory;
the coordinate alignment module configured to: converting the point cloud image and the corresponding RGB image into the same coordinate system;
the point cloud registration module configured to: registering the point cloud images based on the characteristic points of the RGB images to obtain a point cloud registration matrix, so as to realize point cloud registration;
the point cloud fusion module configured to: carrying out point cloud fusion on the point cloud model after point cloud registration to form a point cloud model; establishing a coordinate corresponding relation before and after registration as a coordinate transformation lookup table, and outputting the coordinate transformation lookup table to a volume rendering module;
the volume rendering module configured to: and generating surface points for the whole point cloud model, and performing texture mapping by combining a coordinate transformation lookup table so as to reconstruct a 3D scanning.
In another aspect, the present disclosure provides a method for performing 3D scan reconstruction using a mobile terminal, the method including:
s100, projecting a speckle pattern;
s200, collecting a laser speckle image and an RGB image;
s300, performing motion matching on the laser speckle image and the reference speckle image, calculating a corresponding depth value according to the offset distance of each pixel point in the laser speckle image, converting the laser speckle image into a point cloud image, and converting the point cloud image and a corresponding RGB image into the same coordinate system;
s400, registering the point cloud images based on the characteristic points of the RGB images to obtain a point cloud registration matrix to realize point cloud registration;
s500, point cloud fusion is carried out on the point cloud model after point cloud registration to form a point cloud model; establishing a coordinate corresponding relation before and after registration as a coordinate transformation lookup table;
s600, generating surface points for the point cloud model, and mapping textures by combining a coordinate transformation lookup table.
Compared with the prior art, the 3D scanning reconstruction method and device of the intelligent mobile terminal based on the laser speckle image depth perception technology, which are provided by the disclosure, can greatly improve the mobility, the practicability and the universality of the current three-dimensional reconstruction technology.
Drawings
Fig. 1 is a block diagram of a 3D scanning reconstruction apparatus of an intelligent mobile terminal according to an embodiment of the present invention;
fig. 2 (a), 2 (b), 2 (c) are schematic diagrams of a smartphone according to an embodiment of the present invention;
in the figure: 1-an IR camera, 2-a laser speckle projector, 3-an RGB camera, 4-a display screen, 5-a depth perception control module, 6-a mobile phone processor and 7-a memory;
FIG. 3 is a flowchart of 3D scanning reconstruction of an intelligent mobile terminal according to an embodiment of the present invention;
FIG. 4 is a flow chart of a point cloud registration module based on multiple graphs according to an embodiment of the present invention;
fig. 5 is a flow chart of a video stream-based point cloud registration module according to an embodiment of the present invention.
Detailed Description
In one embodiment, a mobile terminal for 3D scan reconstruction as shown in fig. 1 is provided, the mobile terminal comprising a structured light laser speckle projection module, an RGB camera, an IR camera, an image matching module, a coordinate alignment module, a point cloud registration module, a point cloud fusion module, a volume rendering module; wherein:
the structured light laser speckle projection module configured to: projecting speckles;
the RGB camera is used for collecting RGB images, and the IR camera is used for collecting laser speckle images;
the image matching module configured to: performing motion matching on the laser speckle image and the reference speckle image, calculating a corresponding depth value according to the offset distance of each pixel point in the laser speckle image, and converting the laser speckle image into a point cloud image; the reference speckle image is solidified in a memory;
the coordinate alignment module configured to: converting the point cloud image and the corresponding RGB image into the same coordinate system;
the point cloud registration module configured to: registering the point cloud images based on the characteristic points of the RGB images to obtain a point cloud registration matrix so as to realize point cloud registration;
the point cloud fusion module configured to: carrying out point cloud fusion on the point cloud model after point cloud registration to form a point cloud model; establishing a coordinate corresponding relation before and after registration as a coordinate transformation lookup table, and outputting the coordinate transformation lookup table to a volume rendering module;
the volume rendering module configured to: the method is used for generating surface points of the whole point cloud model and carrying out texture mapping by combining a coordinate transformation lookup table.
In the embodiment, the coordinate alignment of the RGB image and the point cloud is designed, so that the reconstruction result has RGB and depth information at the same time; and finally, 3d scanning reconstruction of fusion RGB and depth information in real time in multiple modes on an intelligent mobile terminal platform is realized.
In one embodiment, because the traditional ICP registration algorithm requires a small difference between every two frames of point clouds and a large coincidence surface, it is difficult to register two discrete point clouds; under certain conditions, it is difficult to perform a scanning reconstruction of the target object in the form of a continuous video stream. Therefore, it is preferable that in the point cloud registration module, a point cloud registration method based on multiple graphs and a point cloud registration method based on video stream are proposed for scan reconstruction in the form of continuous video stream and scan reconstruction in the form of continuous video stream, respectively, and based on the methods, they are allocated to be configured as: a point cloud registration unit based on multiple images and a point cloud registration unit based on a video stream. Wherein:
the multiple image-based point cloud registration unit configured to: splicing images according to the characteristic points in the RGB images, taking a rotation translation matrix spliced by the RGB images as an initial rotation translation matrix for registration, and performing coarse registration; then, accurately registering the point cloud model after coarse registration by using an ICP (inductively coupled plasma) algorithm, and finally obtaining a point cloud registration matrix of two point clouds so as to realize point cloud registration;
the video stream-based point cloud registration unit configured to: acquiring matching point pairs in front and back frame point clouds, and establishing an error measurement function according to the matching point pairs; based on an error metric function, acquiring a point cloud registration matrix by using a least square method so as to realize point cloud registration; taking the current point cloud registration matrix as an initial matrix for iterative estimation of the point cloud registration matrix of the next frame, and taking the unit matrix as the initial matrix for iterative estimation of the point cloud registration matrix of the first two frames of the video stream;
in this embodiment, the ICP algorithm is preferably implemented on a hardware circuit basis.
In this embodiment, it is preferable that the matching point pair includes the following steps:
finding out all matching point pairs of front and back frame point clouds according to a projection method;
and eliminating the wrong matched point pairs by calculating Euclidean distances between the matched point pairs and normal vector included angle constraint conditions.
In this embodiment, it is preferable that, in the mobile terminal, wherein: the error metric function is the sum of the distances between each pair of matched points, expressed as follows:
wherein,
e is an error metric function;
W k determining the weight factor of the matching point pair for the kth pair according to whether the matching point pair is an ORB characteristic point pair in the corresponding RGB image;
t is a rotation and translation matrix of two frames of point cloud images which are related to matching point pairs in an error measurement function;
p k and q is k Two coordinates, p, for the k-th pair of matching point pairs k For a point in the current frame image, q k Points in the previous frame image;
n k is q k The normal vector of (a);
and N is the total number of the matched point pairs.
In one embodiment, the computations involved in the volume rendering module are accelerated using GPU parallel computations, the volume rendering module comprising a surface point generation unit, a texture mapping unit; wherein:
the surface point generating unit configured to: allocating a thread to each pixel in an output image, wherein each thread moves along the direction vertical to the image plane, acquiring a symbol distance function value in an encountered voxel in the moving process, and storing the mean value of the current symbol distance function values; when the symbol distance function value of the current voxel is a zero point, namely the symbol distance function value of the current voxel is opposite to the symbol of the mean value of the symbol distance function stored on the thread, the point is an object surface point, and the three-dimensional coordinate of the current point is stored; until all threads search surface points, the generation of the object surface is completed;
the voxel refers to a cube surrounded by eight adjacent data points in the point cloud model;
the texture mapping unit configured to: allocating a thread to each surface point obtained by the surface point generating unit; and the thread acquires the RGB value of a corresponding pixel point of the corresponding pixel on the RGB image by inquiring a coordinate transformation lookup table generated in the point cloud registration process.
The mobile terminal comprises a mobile phone, a tablet, a Pad and the like.
In one embodiment, a handset as shown in fig. 2 (a), 2 (b), 2 (c) is provided, which comprises an IR camera 1, a laser speckle projector 2, an rgb camera 3, a display screen 4, a depth perception control module 5, a handset processor 6, and a memory 7. The depth perception control module 5 can generate a trigger signal to control the structured light laser speckle projection module to project a solidified speckle pattern, has the function of an image acquisition module, and can control the TR camera and the RGB camera to acquire RGB images and laser speckle images of a scene and an object. The mobile phone processor 6 runs a point cloud fusion module and a volume rendering module. The memory 7 stores the reference speckle images in a fixed manner.
In one embodiment, the present disclosure provides a method for reconstructing a 3D scan using a mobile terminal, where a flowchart is shown in fig. 3, and the method includes the following steps:
s100, projecting a laser speckle image;
s200, collecting a laser speckle image and an RGB image;
s300, performing motion matching on the laser speckle image and the reference speckle image, calculating a corresponding depth value according to the offset distance of each pixel point in the laser speckle image, converting the laser speckle image into a point cloud image, and converting the point cloud image and a corresponding RGB image into the same coordinate system;
s400, registering the point cloud images based on the characteristic points of the RGB images to obtain a point cloud registration matrix to realize point cloud registration;
s500, point cloud fusion is carried out on the point cloud model after point cloud registration to form a point cloud model; establishing a coordinate corresponding relation before and after registration as a coordinate transformation lookup table;
s600, generating surface points for the point cloud model, and mapping textures by combining a coordinate transformation lookup table.
In the method, wherein: the registration of the point cloud image in the step S400 includes two ways, namely point cloud registration based on multiple images and point cloud registration based on video stream;
the point cloud registration based on multiple images comprises the following steps:
s401, splicing images according to the characteristic points in the RGB images, and performing coarse registration by taking a rotational translation matrix spliced by the RGB images as an initial rotational translation matrix for registration;
s402, accurately registering the point cloud model after coarse registration by using an ICP (inductively coupled plasma) algorithm, and finally obtaining a point cloud registration matrix of two point clouds so as to realize point cloud registration;
the video stream based point cloud registration comprises the steps of:
s411, acquiring matching point pairs in the front and rear frame point clouds, and establishing an error measurement function according to the matching point pairs;
s412, estimating a point cloud registration matrix through an ICP (inductively coupled plasma) algorithm, so as to realize point cloud registration; acquiring a point cloud registration matrix by using a least square method based on an error metric function, thereby realizing the registration of the point cloud; taking the current point cloud registration matrix as an initial matrix for iterative estimation of the point cloud registration matrix of the next frame, and taking the unit matrix as the initial matrix for iterative estimation of the point cloud registration matrix of the first two frames of the video stream;
the ICP algorithm is implemented based on hardware circuits.
In the method, wherein: the matching point pair obtaining comprises the following steps:
finding out all matching point pairs of front and back frame point clouds according to a projection method;
and eliminating the error matching point pairs by calculating Euclidean distances between the matching point pairs and normal vector included angle constraint conditions.
In the method, wherein: the computation involved in step S600 is accelerated using GPU parallel computation, wherein:
the surface point generation comprises the following steps:
s601, allocating a thread to each pixel in an output image, wherein each thread moves along a direction vertical to an image plane;
s602, obtaining a sign distance function value in an encountered voxel in a moving process;
s603, when the symbol distance function value of the current voxel is a zero point, namely the symbol distance function value of the current voxel is opposite to the symbol of the mean value of the symbol distance function stored on the thread, the point is an object surface point, and the three-dimensional coordinate of the current point is stored; until all threads search surface points, the generation of the object surface is completed;
the texture mapping comprises the following steps:
s611, distributing a thread to each pixel forming the point cloud model of the point cloud model processed by the surface point generating unit;
and S612, the thread acquires the RGB values of corresponding pixel points of the corresponding pixels on the RGB image by inquiring a coordinate transformation lookup table generated in the point cloud registration process.
In the method, wherein: the error metric function is the sum of the distances between each pair of matched points, and is expressed as follows:
wherein,
e is an error metric function;
W k determining the weight factor of the matching point pair for the kth pair according to whether the matching point pair is an ORB characteristic point pair in the corresponding RGB image;
t is a rotation and translation matrix of two frames of point cloud images, and the two frames of point cloud images are related to matching point pairs in an error measurement function;
p k and q is k Two coordinates, p, for the k-th pair of matching point pairs k For a point in the current frame image, q k Points in the previous frame image;
n k is q k The normal vector of (a);
n is the total number of matching point pairs.
In one embodiment, a specific depth value calculation method is provided, including the steps of:
s301, selecting an image block from the reference speckle pattern, and simultaneously reading search window data with the size of (M × N) and the position corresponding to the input image block as the center from the standard speckle reference pattern, wherein M and N are integers, and M > = M and N > N.
S302, the image block searches the optimal matching block of the motion block in the search window according to a certain search strategy and a similarity measurement criterion, and the displacement (delta x, delta y) between the image block and the reference matching block is obtained, namely the motion vector of the image block. The design realizes the matching calculation of the source matching block and the matching block in the search window by using the sum of absolute value differences of the matching blocks, and the matching block with the minimum absolute value difference sum is the optimal matching block.
S303, after the motion displacement estimation module calculates the displacement offset according to the obtained optimal matching block, the depth information d' of the central point of the input speckle image block is calculated according to the following depth calculation formula by combining the known distance parameter d of the reference speckle image, the focal length f of the image sensor, the baseline distance S between the laser projector and the receiving image sensor and the pixel point distance parameter mu of the image sensor:
the optimal offset delta m is equal to the coordinate value of the center point x of the reference speckle window-the coordinate value of the center point x 'of the optimal matching block, or the coordinate value of the center point y of the reference speckle window-the coordinate value of the center point y' of the optimal matching block, and has positive and negative values.
In the foregoing embodiment, the point cloud image and the corresponding RGB image thereof are converted into the same coordinate system, so as to solve the problem that the coordinates of the point cloud image data acquired by the IR camera and the coordinates of the RGB image data acquired by the RGB camera are not aligned. Suppose a row-column value coordinate point (U) of a point in the physical world in the RGB camera captured image RGB ，V RGB ) And a line-row value coordinate point (X) in the TR camera acquisition image d ，Y d ) T Different, the IR camera and the RGB camera are calibrated to obtain a rotation matrix R between the TR camera and the RGB camera D-RGB And a translation vector t D-RGB And obtaining a space coordinate transformation formula of a point under an IR camera coordinate system and a point under an RGB camera coordinate system as follows:
wherein:
Z d as a point (X) in a point cloud image d ，Y d ) T The depth value of (d);
Z RGB as a point (X) in an RGB image RGB ，Y RGB ) The depth value of (2).
If R is to be D-RGB Write as:
will t D-RGB Writing as (t) x ，t y ，t z )
Simultaneously introduce R D-RGB 、t D-RGB Extended to 4x4 matrix R D-RGB 、t＇ D-RGB Will be (X) RGB ，Y RGB ，Z RGB ) T 、(X d ，Y d ，Z d ) T Extended to 4-dimensional vectors, the spatial transform formula can be simplified as:
wherein:
further, it is possible to obtain:
wherein (X) RGB ，Y RGB ，Z RGB ) T Is a three-dimensional coordinate under the coordinate system of the RGB camera, (X) d ，Y d ，Z d ) T Is a three-dimensional coordinate in the IR camera coordinate system, (R' D-RGB |t’ D-RGB ) Is a rotation-translation matrix from the IR camera coordinate system to the RGB camera coordinate system.
And then according to the internal reference of the RGB camera, the corresponding coordinates of the depth image pixels in the RGB image can be obtained:
in the formula: (U) RGB ，V RGB ) Is the calculated RGB image pixel coordinate, f x 、f y The focal lengths of the RGB camera in the x and y directions respectively, (O) x ，O y ) Is the center point coordinates of the color image.
Further, by rearranging the corresponding pixels of the RGB image according to the pixels of the depth image, the depth image and the color image under the IR camera can be obtained.
In one embodiment, a flow chart of a point cloud registration module based on multiple images is shown in fig. 4, and the input of the module is a point cloud image after matching two images and its corresponding RGB image. The point cloud registration based on multiple images is proposed to solve the problem that under some extreme conditions, scanning reconstruction in a continuous video stream mode is difficult to perform on a target object, and the traditional ICP registration algorithm requires that the difference between every two frames of point clouds is small and a large coincidence surface exists, so that registration of two discrete point clouds is difficult to achieve. The patent provides a hardware implementation method for point cloud registration based on multiple images, which utilizes point cloud RGB data obtained by a coordinate alignment module to obtain an initial conversion matrix, utilizes the initial conversion matrix to register two point clouds to the same coordinate system, so that the two point clouds have high contact ratio and extremely small displacement, and then realizes the point cloud registration of the multiple images through a traditional ICP algorithm, and the detailed steps are as follows:
receiving two continuous point clouds A and B from an image matching module and RGB images corresponding to the point clouds A and B, and respectively performing feature detection on the RGB images corresponding to the point clouds A and B, wherein sift feature point detection is performed on the images in the patent, and the pipeline design idea of a hardware circuit is utilized to accelerate the operation to achieve real-time performance, so that feature points of the two images are obtained;
splicing the images based on the characteristic points of the two RGB images;
taking a transfer matrix spliced by RGB images as an initial transfer matrix of an ICP (inductively coupled plasma) algorithm for point cloud registration to finish the coarse registration of two point clouds;
and performing ICP (inductively coupled plasma) registration on the point cloud after the rough registration, accelerating the point cloud through a hardware pipeline to obtain an accurately registered point cloud model, and performing detailed description on a point cloud registration module based on video stream by using an ICP algorithm based on the pipeline design idea acceleration of a hardware circuit.
In one embodiment, a flow chart of a video stream-based point cloud registration module is shown in fig. 5, the input of the module is a point cloud frame map in a video stream format and a corresponding RGB map, and a conventional point cloud registration algorithm has high computational complexity and is difficult to realize real-time point cloud registration, so that an ICP algorithm is realized through hardware, the pipeline design of a hardware circuit is accelerated, and meanwhile, the iteration times of the ICP algorithm are reduced by detecting ORB feature points of an RGB image and adding the weight of the ORB feature points in a registration flow, so that the real-time performance of scanning reconstruction is realized. The method comprises the following concrete steps:
acquiring point cloud images of two continuous frames, and processing the depth image D i Each pixel point p (u, v) on (p) is according to the IR camera internal reference matrix K IR Calculating corresponding three-dimensional vertex coordinate V of depth map under camera coordinate system through inverse transmission transformation i The calculation formula is as follows:
and then, cross-multiplying two vectors of adjacent vertexes of each vertex to obtain a normal vector corresponding to the vertex, wherein the calculation formula is as follows:
N i (u，v)＝(V i (u+1，v)-V i (u，v))×(V i (u，V+1)-V i (u，v))。
the ICP algorithm has high requirements on initial transformation parameters, and if the initial matrix is not properly selected, iteration is not converged or a registration result falls into local optimum. However, since the method is based on 3D scanning reconstruction of the video stream, it can be considered that the positions between two adjacent frames of point clouds are relatively close, and the initial transformation matrix is set as the point cloud registration matrix of the previous frame of point cloud, especially for the first two frames of point clouds of the video stream
Initializing iteration times and setting the maximum value of the iteration times;
and estimating a point cloud registration matrix, and firstly finding all matching point pairs of front and back frame point clouds according to a projection method, wherein the projection method has good noise immunity and can obtain the best matching result, so that the method has higher matching speed and can meet the requirements of high precision and real-time property. The basic idea of obtaining the matching point pair by the projection method is to collect the point cloud V of the kth frame k Converting to a point cloud set V of a k-1 frame k-1 In the camera coordinate system, is marked as V k For V k ' any point of
Finding out corresponding image coordinates by perspective projection, and finding out a space point q with the same image coordinates on the point cloud space set of the (k-1) th frame from the image coordinates k-1 This point is the corresponding point sought.
After the corresponding point pairs are obtained, weights need to be distributed to the point pairs, in order to improve the iteration efficiency, the method provides that ORB characteristic point pairs obtained from the RGB images are utilized, and higher weight values are given to the characteristic point pairs, so that the purpose of accelerating the iteration convergence speed is achieved.
And after the matching point pairs are obtained, the staggered matching point pairs are required to be eliminated, partial error matching point pairs are eliminated by calculating the Euclidean distance between the matching point pairs and the normal vector included angle constraint condition, when the Euclidean distance and the normal vector included angle of the matching point pairs simultaneously meet the set threshold, the matching point pairs are correct matching point pairs, and otherwise, the matching point pairs are error matching point pairs. The calculation formula of the Euclidean distance and the normal vector included angle is as follows:
s＝||V-V k，g ||＜s thr ，
wherein, V k，g And n k Global three-dimensional coordinates and normal vectors, V and n, respectively, representing spatial points p k-1 Representing the global three-dimensional coordinates and normal vectors, s, respectively, of the spatial point q thr And theta thr Respectively a distance threshold and an angle threshold.
And finally, approximately solving the point cloud registration matrix by using a linear least square method, and simultaneously calculating a distance error function of the point cloud registration matrix obtained by iteration. Assuming that the ith frame point cloud and the (i-1) th frame point cloud have N pairs of matching points, the coordinates of the k pair of matching points can be expressed as p k ＝(p kx ，p ky ，p kz ，1) T And q is k ＝(q kx ，q ky ，q kz ，1) T And q is k With normal vector n k ＝(n kx ，n ky ，n kz ，1) T Then, the distance error function of two frames of point clouds is defined as:
wherein, W k The value of the weight factor can be set. For example, when matching a pair of points (p) k ，q k ) For corresponding color image ORB feature point pairs, W k =10, otherwise W k ＝1。
For the calculation of the point cloud registration matrix, firstly, the initial rotation and translation matrix of the ith frame is set as:
T＝t(t x ，t y ，t z )·R(α，β，γ)，
wherein t = (t) x ，t y ，t z ) For a translation matrix, R (alpha, beta, gamma) is a rotation matrix, and the specific expression is as follows:
wherein:
R z (γ)，R y (β)，R x (α) is a rotation matrix of γ, β, α angles around the z-axis, y-axis and x-axis, respectively, sin θ ≈ 0, cos θ ≈ 1 when the rotation angle is small, having
The error function can be expressed as:
wherein:
wherein:
a k1 ＝n kz p ky -n ky p kz
a k2 ＝n kx p kz -n kz p kx
a k3 ＝n ky p kx -n kx p ky
at the moment, the error function minimization problem is converted into a standard linear minimum problem, and singular value decomposition is adopted for solving, so that the solution of a linear system can be obtained, and a point cloud registration matrix is obtained.
Performing singular value decomposition on the matrix A to obtain:
A＝U∑V T
wherein:
a is an M × N matrix, the resulting U is an M × M square matrix (called left singular vector), Σ is an M × N matrix (except that the diagonal elements are all 0, and the diagonal elements are singular values), V T Is an N x N matrix (called the right singular vector).
Let the generalized inverse matrix of matrix A be A + The method comprises the following steps:
A + ＝V∑ + U T ，
wherein:
∑ + the pseudo-inverse sigma is obtained by inverting each nonzero element on the main diagonal line of the pseudo-inverse sigma and then transposing.
The solution that can be obtained for this linear system is:
x＝A + b，
and finally, judging whether the iteration times reach a set maximum value, if so, finishing the iteration, and otherwise, continuously carrying out iterative computation on the estimated point cloud registration matrix.
Two frames of point clouds can be registered to a coordinate system by utilizing the finally obtained point cloud registration matrix, so that the aim of point cloud registration is fulfilled; and meanwhile, establishing a coordinate corresponding relation before and after registration as a coordinate transformation lookup table, and transmitting the coordinate transformation lookup table to a volume rendering module.
Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-described embodiments and application fields, and the above-described embodiments are illustrative, instructive, and not restrictive. Those skilled in the art, having the benefit of this disclosure, may effect numerous modifications thereto without departing from the scope of the invention as defined by the appended claims.
Claims (10)
1. A mobile terminal for 3D scan reconstruction, characterized by:
the mobile terminal comprises a structured light laser speckle projection module, an RGB camera, an IR camera, an image matching module, a coordinate alignment module, a point cloud registration module, a point cloud fusion module and a volume rendering module; wherein:
the structured light laser speckle projection module configured to: projecting a speckle pattern;
the RGB camera is used for collecting RGB images, and the IR camera is used for collecting laser speckle images;
the image matching module configured to: performing motion matching on the laser speckle image and the reference speckle image, calculating a corresponding depth value according to the offset distance of each pixel point in the laser speckle image, and converting the laser speckle image into a point cloud image; the reference speckle image is solidified in a memory;
the coordinate alignment module configured to: converting the point cloud image and the corresponding RGB image into the same coordinate system;
the point cloud registration module configured to: registering the point cloud images by combining the characteristic points of the RGB images to obtain a point cloud registration matrix so as to realize the registration of the point cloud;
the point cloud fusion module configured to: after point cloud registration, point cloud fusion is carried out to form a point cloud model; establishing a coordinate corresponding relation before and after registration as a coordinate transformation lookup table, and outputting the coordinate transformation lookup table to a volume rendering module;
the volume rendering module configured to: and generating surface points for the whole point cloud model, and performing texture mapping by combining a coordinate transformation lookup table so as to reconstruct a 3D scanning.
2. The mobile terminal of claim 1, wherein: preferably, the point cloud registration module comprises a point cloud registration unit based on a plurality of images and a point cloud registration unit based on a video stream;
the multiple image-based point cloud registration unit configured to: splicing images according to the characteristic points in the RGB images, taking a rotation translation matrix spliced by the RGB images as an initial rotation translation matrix for registration, and performing coarse registration; then, accurately registering the point cloud model after coarse registration by using an ICP (inductively coupled plasma) algorithm, and finally obtaining a point cloud registration matrix of two point clouds so as to realize point cloud registration;
the video stream-based point cloud registration unit configured to: acquiring matching point pairs in front and back frame point clouds, and establishing an error measurement function according to the matching point pairs; based on an error metric function, acquiring a point cloud registration matrix by using a least square method so as to realize point cloud registration; taking the current point cloud registration matrix as an initial matrix for iterative estimation of the point cloud registration matrix of the next frame, and taking the unit matrix as the initial matrix for iterative estimation of the point cloud registration matrix of the first two frames of the video stream;
the ICP algorithm is implemented based on hardware circuits.
3. The mobile terminal according to claim 2, wherein the matching point pair comprises the following steps:
finding out all matching point pairs of front and back frame point clouds according to a projection method;
and eliminating the wrong matched point pairs by calculating Euclidean distances between the matched point pairs and normal vector included angle constraint conditions.
4. The mobile terminal according to claim 1, wherein the computations involved in the volume rendering module are accelerated using GPU parallel computations, the volume rendering module comprising a surface point generation unit, a texture mapping unit;
the surface point generating unit configured to: distributing a thread for each pixel in an output image, moving each thread along the direction vertical to the image plane, acquiring a symbol distance function value in encountered voxels in the moving process, and storing the mean value of the current symbol distance function values; when the symbol distance function value of the current voxel is a zero point, namely the symbol distance function value of the current voxel is opposite to the symbol of the mean value of the symbol distance function stored on the thread, the point is an object surface point, and the three-dimensional coordinate of the current point is stored; until all threads search surface points, the generation of the object surface is completed;
the texture mapping unit configured to: allocating a thread to each surface point obtained by the surface point generating unit; and the thread acquires the RGB value of a corresponding pixel point of the corresponding pixel on the RGB image by inquiring a coordinate transformation lookup table generated in the point cloud registration process.
5. The mobile terminal of claim 2, wherein the error metric function is a sum of distances between each pair of matched points, and is expressed as follows:
wherein,
e is an error metric function;
w k determining the weight factor of the matching point pair for the kth pair according to whether the matching point pair is an ORB characteristic point pair in the corresponding RGB image;
t is a rotation and translation matrix of two frames of point cloud images, and the two frames of point cloud images are related to matching point pairs in an error measurement function;
p k and q is k Two coordinates, p, for the k-th pair of matching point pairs k For a point in the current frame image, q k Points in the previous frame image;
n k is q k The normal vector of (a);
and N is the total number of the matched point pairs.
6. A method for reconstructing 3D scanning using a mobile terminal, the method comprising:
s100, projecting a speckle pattern;
s200, collecting a laser speckle image and an RGB image;
s300, performing motion matching on the laser speckle image and the reference speckle image, calculating a corresponding depth value according to the offset distance of each pixel point in the laser speckle image, converting the laser speckle image into a point cloud image, and converting the point cloud image and a corresponding RGB image into the same coordinate system;
s400, registering the point cloud images based on the characteristic points of the RGB images to obtain a point cloud registration matrix to realize point cloud registration;
s500, after point cloud registration, point cloud fusion is carried out to form a point cloud model; establishing a coordinate corresponding relation before and after registration as a coordinate transformation lookup table;
s600, generating surface points for the point cloud model, and performing texture mapping by combining a coordinate transformation lookup table to perform 3D scanning reconstruction.
7. The method according to claim 6, wherein the registering of the point cloud images in step S400 includes point cloud registration based on multiple images and point cloud registration based on video stream;
the point cloud registration based on multiple images comprises the following steps:
s401, splicing images according to the characteristic points in the RGB images, taking a rotation translation matrix spliced by the RGB images as an initial rotation translation matrix for registration, and carrying out coarse registration;
s402, accurately registering the point cloud model after coarse registration by using an ICP (inductively coupled plasma) algorithm, and finally obtaining a point cloud registration matrix of two point clouds so as to realize point cloud registration;
the video stream based point cloud registration comprises the steps of:
s411, acquiring matching point pairs in the front and back frame point clouds, and establishing an error measurement function according to the matching point pairs;
s412, estimating a point cloud registration matrix through an ICP (inductively coupled plasma) algorithm, so as to realize point cloud registration; based on an error metric function, acquiring a point cloud registration matrix by using a least square method, and setting the current point cloud registration matrix as the point cloud registration matrix of the next frame of point cloud so as to realize the point cloud registration; taking the current point cloud registration matrix as an initial matrix for iterative estimation of the point cloud registration matrix of the next frame, and taking the unit matrix as the initial matrix for iterative estimation of the point cloud registration matrix of the first two frames of the video stream;
the ICP algorithm is implemented on the basis of a hardware circuit.
8. The method of claim 7, wherein the matching point pairs, when obtained, comprise the steps of:
finding out all matching point pairs of front and back frame point clouds according to a projection method;
and eliminating the wrong matched point pairs by calculating Euclidean distances between the matched point pairs and normal vector included angle constraint conditions.
9. The method according to claim 6, characterized in that the computation involved in step S600 is accelerated using GPU parallel computations, wherein:
the surface point generation comprises the following steps:
s601, allocating a thread to each pixel in an output image, wherein each thread moves along a direction vertical to an image plane;
s602, obtaining a sign distance function value in an encountered voxel in a moving process;
s603, when the symbol distance function value of the current voxel is a zero point, namely the symbol distance function value of the current voxel is opposite to the symbol of the mean value of the symbol distance function stored on the thread, the point is an object surface point, and the three-dimensional coordinate of the current point is stored; until all threads search surface points, the generation of the object surface is completed;
the texture mapping comprises the following steps:
s611, distributing a thread to each pixel forming the point cloud model of the point cloud model processed by the surface point generating unit;
and S612, the thread acquires the RGB value of a corresponding pixel point of the corresponding pixel on the RGB image by inquiring a coordinate transformation lookup table generated in the point cloud registration process.
10. The method of claim 7, wherein the error metric function is a sum of distances between each pair of matched pairs, expressed as follows:
wherein,
e is an error metric function;
w k determining the weight factor of the matching point pair for the kth pair according to whether the matching point pair is an ORB characteristic point pair in the corresponding RGB image;
t is a rotation and translation matrix of two frames of point cloud images, and the two frames of point cloud images are related to matching point pairs in an error measurement function;
p k and q is k Two coordinates, p, for the k-th pair of matching point pairs k For a point in the current frame image, q k Points in the previous frame image;
n k is q k The normal vector of (a);
n is the total number of matching point pairs.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810058604.XA CN110070598B (en) | 2018-01-22 | 2018-01-22 | Mobile terminal for 3D scanning reconstruction and 3D scanning reconstruction method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810058604.XA CN110070598B (en) | 2018-01-22 | 2018-01-22 | Mobile terminal for 3D scanning reconstruction and 3D scanning reconstruction method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110070598A CN110070598A (en) | 2019-07-30 |
| CN110070598B true CN110070598B (en) | 2022-11-25 |
Family
ID=67364642
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810058604.XA Active CN110070598B (en) | 2018-01-22 | 2018-01-22 | Mobile terminal for 3D scanning reconstruction and 3D scanning reconstruction method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110070598B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110782488B (en) * | 2019-10-11 | 2022-08-12 | 华中科技大学 | Continuous brain picture image three-dimensional registration method based on shape constraint |
| CN110895823B (en) | 2020-01-10 | 2020-06-05 | 腾讯科技（深圳）有限公司 | Texture obtaining method, device, equipment and medium for three-dimensional model |
| CN111353985B (en) * | 2020-03-02 | 2022-05-03 | 电子科技大学 | Airport self-service consignment luggage detection method based on depth camera |
| CN111583388B (en) * | 2020-04-28 | 2023-12-08 | 光沦科技(深圳)有限公司 | Scanning method and equipment of three-dimensional scanning system |
| CN111553985B (en) * | 2020-04-30 | 2023-06-13 | 四川大学 | O-graph pairing European three-dimensional reconstruction method and device |
| CN112489190B (en) * | 2020-11-18 | 2024-05-31 | 新拓三维技术(深圳)有限公司 | Full-automatic indoor scanning method, system and computer readable storage medium |
| CN112783196A (en) * | 2020-12-17 | 2021-05-11 | 国网山西省电力公司运城供电公司 | Distribution network line unmanned aerial vehicle autonomous flight path planning method and system |
| CN113052884A (en) * | 2021-03-17 | 2021-06-29 | Oppo广东移动通信有限公司 | Information processing method, information processing apparatus, storage medium, and electronic device |
| CN113539444B (en) * | 2021-08-30 | 2024-04-19 | 上海联影医疗科技股份有限公司 | Medical image reconstruction method, device, electronic equipment and storage medium |
| CN116883471B (en) * | 2023-08-04 | 2024-03-15 | 天津大学 | Line structured light contact-point-free cloud registration method for chest and abdomen percutaneous puncture |
| CN116935013B (en) * | 2023-09-14 | 2023-11-28 | 武汉工程大学 | Circuit board point cloud large-scale splicing method and system based on three-dimensional reconstruction |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105931234A (en) * | 2016-04-19 | 2016-09-07 | 东北林业大学 | Ground three-dimensional laser scanning point cloud and image fusion and registration method |
| WO2016188068A1 (en) * | 2015-05-27 | 2016-12-01 | 珠海真幻科技有限公司 | Method and system for stereoscopic vision three-dimensional measurement taking computing laser speckles as texture |
| CN106802138A (en) * | 2017-02-24 | 2017-06-06 | 杭州先临三维科技股份有限公司 | A kind of 3 D scanning system and its scan method |
-
2018
- 2018-01-22 CN CN201810058604.XA patent/CN110070598B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016188068A1 (en) * | 2015-05-27 | 2016-12-01 | 珠海真幻科技有限公司 | Method and system for stereoscopic vision three-dimensional measurement taking computing laser speckles as texture |
| CN105931234A (en) * | 2016-04-19 | 2016-09-07 | 东北林业大学 | Ground three-dimensional laser scanning point cloud and image fusion and registration method |
| CN106802138A (en) * | 2017-02-24 | 2017-06-06 | 杭州先临三维科技股份有限公司 | A kind of 3 D scanning system and its scan method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110070598A (en) | 2019-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110070598B (en) | Mobile terminal for 3D scanning reconstruction and 3D scanning reconstruction method thereof | |
| CN109919911B (en) | Mobile three-dimensional reconstruction method based on multi-view photometric stereo | |
| CN103106688B (en) | Based on the indoor method for reconstructing three-dimensional scene of double-deck method for registering | |
| US11568516B2 (en) | Depth-based image stitching for handling parallax | |
| CN111145238A (en) | Three-dimensional reconstruction method and device of monocular endoscope image and terminal equipment | |
| Park et al. | A multiview 3D modeling system based on stereo vision techniques | |
| CN107292965A (en) | A kind of mutual occlusion processing method based on depth image data stream | |
| CN107240129A (en) | Object and indoor small scene based on RGB D camera datas recover and modeling method | |
| CN108288292A (en) | A kind of three-dimensional rebuilding method, device and equipment | |
| CN115205489A (en) | Three-dimensional reconstruction method, system and device in large scene | |
| EP3135033B1 (en) | Structured stereo | |
| CN111325828B (en) | Three-dimensional face acquisition method and device based on three-dimensional camera | |
| Eichhardt et al. | Affine correspondences between central cameras for rapid relative pose estimation | |
| CN115035235A (en) | Three-dimensional reconstruction method and device | |
| GB2567245A (en) | Methods and apparatuses for depth rectification processing | |
| Yuan et al. | 3D reconstruction of background and objects moving on ground plane viewed from a moving camera | |
| CN113450416B (en) | TCSC method applied to three-dimensional calibration of three-dimensional camera | |
| Perdigoto et al. | Calibration of mirror position and extrinsic parameters in axial non-central catadioptric systems | |
| Resch et al. | On-site semi-automatic calibration and registration of a projector-camera system using arbitrary objects with known geometry | |
| CN113379815A (en) | Three-dimensional reconstruction method and device based on RGB camera and laser sensor and server | |
| CN116579962A (en) | Panoramic sensing method, device, equipment and medium based on fisheye camera | |
| CN117974786A (en) | Multi-vision-based dynamic environment reconstruction and measurement method and system | |
| CN107240149A (en) | Object dimensional model building method based on image procossing | |
| Hartley et al. | Camera models | |
| KR20160049639A (en) | Stereoscopic image registration method based on a partial linear method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |