CN114140510A - Incremental three-dimensional reconstruction method and device and computer equipment - Google Patents
Incremental three-dimensional reconstruction method and device and computer equipment Download PDFInfo
- Publication number
- CN114140510A CN114140510A CN202111470822.2A CN202111470822A CN114140510A CN 114140510 A CN114140510 A CN 114140510A CN 202111470822 A CN202111470822 A CN 202111470822A CN 114140510 A CN114140510 A CN 114140510A
- Authority
- CN
- China
- Prior art keywords
- reconstruction
- rendering
- incremental
- image
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000009877 rendering Methods 0.000 claims abstract description 79
- 230000008569 process Effects 0.000 claims abstract description 36
- 238000005070 sampling Methods 0.000 claims abstract description 31
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 13
- 238000003062 neural network model Methods 0.000 claims description 32
- 238000004364 calculation method Methods 0.000 claims description 15
- 230000001052 transient effect Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 5
- 230000003068 static effect Effects 0.000 claims description 4
- 238000012549 training Methods 0.000 claims description 4
- 238000007476 Maximum Likelihood Methods 0.000 claims description 3
- 230000010339 dilation Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 3
- 238000005286 illumination Methods 0.000 claims description 3
- 230000000877 morphologic effect Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 2
- 230000001629 suppression Effects 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 5
- 210000005036 nerve Anatomy 0.000 abstract description 4
- 238000005457 optimization Methods 0.000 abstract description 4
- 230000005855 radiation Effects 0.000 abstract description 4
- 230000004069 differentiation Effects 0.000 abstract description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20036—Morphological image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computer Graphics (AREA)
- Image Generation (AREA)
- Processing Or Creating Images (AREA)
Abstract
The embodiment of the invention discloses an incremental three-dimensional reconstruction method, an incremental three-dimensional reconstruction device and computer equipment, and aims to solve the problems of complex operation, insufficient precision and the like in the prior art. The incremental three-dimensional reconstruction method mainly comprises the following steps: acquiring a picture sequence of a target scene; calculating the projection ray of each pixel point of each image; discrete sampling and rendering of projection ray rays; implicit reconstruction of the model; and (4) solving the parameters of the camera reversely, and performing incremental reconstruction based on the reconstruction result of the previous image. The method uses the nerve radiation field technology to implicitly reconstruct a three-dimensional model, also uses the volume rendering principle and a micro-rendering algorithm to realize the rendering process from a 3D voxel model to a 2D image, so that the three-dimensional model can use differentiation to carry out nonlinear optimization, thereby realizing end-to-end three-dimensional reconstruction, and having simple and convenient operation and high precision.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a three-dimensional reconstruction method based on images.
Background
With the development of computer vision, technologies such as three-dimensional reconstruction, AR, autopilot, SLAM and the like gradually enter our daily life, wherein image-based three-dimensional reconstruction is always one of the most important challenges in the field of computer vision. The three-dimensional reconstruction method based on the image can restore the geometric shape and the texture information of a scene according to the image, and has wide application value in real life, such as defect detection, digital vestige, electronic maps, navigation and the like. Different application directions have different requirements on three-dimensional reconstruction.
The current three-dimensional reconstruction method mainly comprises three-dimensional reconstruction based on a depth camera, three-dimensional reconstruction based on structured light and multi-view three-dimensional reconstruction, wherein the three-dimensional reconstruction based on the depth camera and the three-dimensional reconstruction based on the structured light are limited by hardware, and the multi-view three-dimensional reconstruction requires abundant shooting feature points and is also limited. Generally speaking, most of the operations are complicated and the precision is not ideal.
Disclosure of Invention
Therefore, the embodiment of the invention provides an incremental three-dimensional reconstruction method, an incremental three-dimensional reconstruction device and computer equipment, and aims to solve the problems of complex operation, insufficient precision and the like in the prior art.
In order to achieve the above object, the embodiments of the present invention provide the following technical solutions:
in a first aspect, an incremental three-dimensional reconstruction method includes the following steps:
s1: acquiring a picture sequence of a target scene; the picture sequence comprises a plurality of images which are obtained correspondingly by shooting a target scene from multiple angles; performing the following steps S2-S4 for any image in the sequence of pictures;
s2: calculating a back projection ray from each pixel point of the image to the center position of the camera;
s3: setting a depth range of the back projection ray, discretely sampling a series of 3D points on the back projection ray in the depth range, solving through a neural network model represented by a multilayer perceptron, and performing micro-rendering on pixel points by using a volume rendering principle to obtain a rendering result of the pixel points, wherein gradient information is obtained in the micro-rendering process;
s4: randomly sampling pixel points on the image, carrying out implicit reconstruction on the model, and obtaining an initial reconstruction result after iterative calculation and rendering according to the steps S2 and S3; updating parameters of the neural network model and attitude information of the camera in the implicit reconstruction process, and utilizing gradient information obtained in the micro-rendering process;
s5: sequentially taking other images in the image sequence, and performing incremental reconstruction based on a reconstruction result of a previous image; traversing the picture sequence to finally obtain an incremental reconstruction result;
wherein each incremental reconstruction comprises: randomly initializing the camera pose corresponding to the current image, and repeatedly executing the steps S2, S3 and S4, wherein only the pose of the camera is updated in the repeated execution process, and the parameters of the neural network model are kept unchanged; then, steps S2, S3, and S4 are repeatedly performed again, and only the parameters of the neural network model are updated during the repeated execution.
Optionally, the pose information of the camera comprises a camera position and a viewing direction.
Optionally, the inputs of the neural network model are camera position, observation direction and observed 3D coordinates, and the output is color c and volume density σ of the 3D point.
Optionally, in step S4, the parameters of the neural network model and the pose information of the camera are updated, specifically: and calculating the gradient of the neural network model and the gradient of the camera parameter by using the gradient information obtained in the micro-rendering process, and updating the parameter of the neural network model and the attitude information of the camera by using an Adam algorithm.
Optionally, between step S3 and step S4, the difference between the rendered color and the actual picture color is also calculated as the error of reconstruction.
Optionally, in step S4, the randomly sampling pixel points on the image, specifically extracting pixel points from the region of interest of each image; the determination method of the interest area comprises the following steps: and (3) carrying out convolution on the gradient image (gradient information) by using the blob and the corner kernel, then carrying out non-maximum inhibition to obtain an image interest point, and then generating an interest region by using a morphological dilation algorithm.
Optionally, the incremental reconstruction described in step S5 further performs the following processing in consideration of dynamic object or illumination influence:
firstly, using the rendering equation capable of micro-rendering in the step S3 as the static part of the model, then adding the transient part, and rendering the color and density of the transient, wherein the density allows the change in the training image;
secondly, an uncertainty field is allowed to be transmitted by the transient part, so that the model can adjust reconstruction loss and ignore unreliable pixels and 3D positions;
and modeling the color of each pixel into isotropic normal distribution, performing maximum likelihood estimation, and rendering the variance of the distribution as a transient color by using volume rendering.
In a second aspect, an incremental three-dimensional reconstruction apparatus comprises:
the image sequence acquisition module is used for acquiring an image sequence of a target scene; the picture sequence comprises a plurality of images which are obtained correspondingly by shooting a target scene from multiple angles;
the projection ray calculation module is used for calculating a back projection ray from each pixel point of the image to the center position of the camera aiming at the currently selected image;
the discrete sampling and rendering module is used for setting a depth range of the back projection ray, discretely sampling a series of 3D points on the back projection ray in the depth range, solving the points through a neural network model represented by a multilayer perceptron, and then carrying out micro-rendering on the pixel points by using a volume rendering principle to obtain a rendering result of the pixel points, wherein gradient information is obtained in the micro-rendering process;
the implicit reconstruction module is used for randomly sampling pixel points on the image, performing implicit reconstruction on the model, and obtaining an initial reconstruction result by iteratively operating the projection ray calculation module and the discrete sampling and rendering module; updating parameters of the neural network model and attitude information of the camera in the implicit reconstruction process, and utilizing gradient information obtained in the micro-rendering process;
the incremental reconstruction module is used for sequentially taking other images in the picture sequence and carrying out incremental reconstruction based on the reconstruction result of the previous image; traversing the picture sequence to finally obtain an incremental reconstruction result;
wherein each incremental reconstruction comprises: randomly initializing a camera pose corresponding to a current image, and repeatedly operating a projection ray calculation module, a discrete sampling and rendering module and an implicit reconstruction module, wherein only the pose of the camera is updated in the repeated operation process, and parameters of the neural network model are kept unchanged; and then, repeatedly operating the projection ray calculation module, the discrete sampling and rendering module and the implicit reconstruction module again, and only updating parameters of the neural network model in the repeated operation process.
In a third aspect, a computer device comprising a memory and a processor, the memory storing a computer program, is characterized in that the processor implements the steps of the above method when executing the computer program.
In a fourth aspect, a computer-readable storage medium, on which a computer program is stored, is characterized in that the computer program realizes the steps of the above-mentioned method when being executed by a processor.
The invention has at least the following beneficial effects:
the method uses a nerve radiation field technology to implicitly reconstruct a three-dimensional model, input data are picture sequences, and parameters to be estimated are the posture and the three-dimensional model of a camera; the rendering process from the 3D voxel model to the 2D image is realized by using a micro-rendering algorithm by using a volume rendering principle, so that the three-dimensional model can be subjected to nonlinear optimization by using differentiation, and an end-to-end three-dimensional reconstruction method is realized; based on the method, a plurality of images which are correspondingly obtained by shooting the target scene in multiple angles are traversed, and an incremental reconstruction result is finally obtained, so that the method is simple and convenient to operate and high in precision.
Drawings
In order to more clearly illustrate the prior art and the present invention, the drawings which are needed to be used in the description of the prior art and the embodiments of the present invention will be briefly described. It should be apparent that the drawings in the following description are merely exemplary, and that other drawings may be derived from the provided drawings by those of ordinary skill in the art without inventive effort.
The structures, proportions, sizes, and other dimensions shown in the specification are for illustrative purposes only and are not intended to limit the scope of the present invention, which is defined by the claims, and it is to be understood that all such modifications, changes in proportions, or alterations in size which do not affect the efficacy or objectives of the invention are not to be seen as within the scope of the present invention.
Fig. 1 is a schematic flow chart of an incremental three-dimensional reconstruction method according to an embodiment of the present invention;
fig. 2 is a schematic view of a virtual module architecture of an incremental three-dimensional reconstruction apparatus according to an embodiment of the present invention;
FIG. 3 is a sample of the actual processing results of one embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In one embodiment, as shown in fig. 1, there is provided an incremental three-dimensional reconstruction method, including the steps of:
s1: acquiring a picture sequence of a target scene; the picture sequence comprises a plurality of images which are obtained correspondingly by shooting a target scene from multiple angles; performing the following steps S2-S4 for any image in the sequence of pictures;
s2: calculating a back projection ray from each pixel point of the image to the center position of the camera;
s3: setting a depth range of the back projection ray, discretely sampling a series of 3D points on the back projection ray in the depth range, solving through a neural network model represented by a multilayer perceptron, and performing micro-rendering on pixel points by using a volume rendering principle to obtain a rendering result of the pixel points, wherein gradient information is obtained in the micro-rendering process;
s4: randomly sampling pixel points on the image, carrying out implicit reconstruction on the model, and obtaining an initial reconstruction result after iterative calculation and rendering according to the steps S2 and S3; updating parameters of the neural network model and attitude information of the camera in the implicit reconstruction process, and utilizing gradient information obtained in the micro-rendering process;
s5: sequentially taking other images in the image sequence, and performing incremental reconstruction based on a reconstruction result of a previous image; traversing the picture sequence to finally obtain an incremental reconstruction result;
wherein each incremental reconstruction comprises: randomly initializing the camera pose corresponding to the current image, and repeatedly executing the steps S2, S3 and S4, wherein only the pose of the camera is updated in the repeated execution process, and the parameters of the neural network model are kept unchanged; then, steps S2, S3, and S4 are repeatedly performed again, and only the parameters of the neural network model are updated during the repeated execution.
The key steps of the method at least comprise:
(1) calculating a back projection ray of each pixel point of each view;
(2) discrete sampling and rendering of projection ray rays;
(3) implicit reconstruction of the model;
(4) inverse solution of camera parameters (parameters of model M are updated using Adam algorithm).
The method is a three-dimensional reconstruction algorithm which is simple and convenient to operate and high in precision, and a three-dimensional model is implicitly reconstructed by using a nerve radiation field technology; the input data is a picture sequence, and the pose and the three-dimensional model of the camera are parameters to be estimated. The method also utilizes a volume rendering principle and a micro-rendering algorithm to realize the rendering process from the 3D voxel model to the 2D image, so that the three-dimensional model can utilize the differential to carry out nonlinear optimization, thereby realizing the end-to-end three-dimensional reconstruction method, finally obtaining the incremental reconstruction result, and having simple and convenient operation and high precision.
Illustratively, the specific description is as follows:
1. firstly, shooting a scene or an object at multiple angles, and recording camera attitude information;
2. camera reference is represented by K
The camera external parameters are represented by a conductive SE (3),
3. model M is characterized using a multi-layer perceptron. The parameters of the perceptron imply a three-dimensional model. The inputs to the neural network are the camera position, the direction of observation, and the 3D coordinates being observed, and the output is the color c and the volume density σ of the 3D point.
4. And calculating the back projection ray of one pixel point according to the internal and external parameters of the image, wherein the central position of the camera is the starting point of the ray. The depth range of the ray is set according to the size of the model, and then sampling is carried out on the ray in the depth range, so that a series of 3D points are obtained. And inputting the camera position, the projection ray and the 3D point coordinate into the model M to obtain the color and the volume density of each 3D point. And solving the rendering result of the pixel point by using a volume rendering formula.
5. Calculating a rendering loss: and calculating the difference between the rendering color and the actual picture color as the reconstruction error.
6. And calculating the gradient of the model M and the gradient of the camera parameter by using the gradient information obtained in the micro-rendering process. Parameters of the model M, parameters of the camera are updated using Adam algorithm.
7. And randomly sampling pixel points. During reconstruction, pixel points need to be randomly sampled on an image. And (4) convolving the gradient image by using the blob and the kernel, and then executing non-maximum suppression to obtain the image interest points. The region of interest is then generated using a morphological dilation algorithm. And when random sampling is carried out, extracting the interest region of each graph. This can significantly improve training speed and accuracy.
8. And repeating the steps 4-7 to obtain an initial reconstruction result.
9. And subsequently, shooting a new scene picture I, and performing incremental reconstruction. Due to dynamic objects or illumination effects, the image may not conform to the model, and therefore the following is also done:
1) first, the rendering equation in step 4 is taken as the static part of our model, then we add the transient part, rendering the color and density of the transient, where the density allows for changes in the training image. This allows the reconstruction of images containing occlusions without introducing a dynamic blur component to the static scene representation.
2) Second, not assuming that all observed pixel colors are equally reliable, allowing our transient part to emit uncertainty fields (color and density) can let the model adjust reconstruction loss, ignoring unreliable pixels and 3D locations that are likely to contain transient occlusions.
3) The color of each pixel is modeled as an isotropic normal distribution and a maximum likelihood estimation is performed, and then the variance of the distribution is rendered as a transient color using volume rendering.
10. The camera pose of the picture is initialized and the steps 4-7 are repeated, but only the pose of the camera is updated, keeping the parameters of the model M unchanged.
And 4-7 steps are executed again, parameters of the model M are updated by using the Adam algorithm, and an incremental reconstruction result is obtained.
In one embodiment, there is provided an incremental three-dimensional reconstruction apparatus, as shown in fig. 2, comprising:
the image sequence acquisition module is used for acquiring an image sequence of a target scene; the picture sequence comprises a plurality of images which are obtained correspondingly by shooting a target scene from multiple angles;
the projection ray calculation module is used for calculating a back projection ray from each pixel point of the image to the center position of the camera aiming at the currently selected image;
the discrete sampling and rendering module is used for setting a depth range of the back projection ray, discretely sampling a series of 3D points on the back projection ray in the depth range, solving the points through a neural network model represented by a multilayer perceptron, and then carrying out micro-rendering on the pixel points by using a volume rendering principle to obtain a rendering result of the pixel points, wherein gradient information is obtained in the micro-rendering process;
the implicit reconstruction module is used for randomly sampling pixel points on the image, performing implicit reconstruction on the model, and obtaining an initial reconstruction result by iteratively operating the projection ray calculation module and the discrete sampling and rendering module; updating parameters of the neural network model and attitude information of the camera in the implicit reconstruction process, and utilizing gradient information obtained in the micro-rendering process;
the incremental reconstruction module is used for sequentially taking other images in the picture sequence and carrying out incremental reconstruction based on the reconstruction result of the previous image; traversing the picture sequence to finally obtain an incremental reconstruction result;
wherein each incremental reconstruction comprises: randomly initializing a camera pose corresponding to a current image, and repeatedly operating a projection ray calculation module, a discrete sampling and rendering module and an implicit reconstruction module, wherein only the pose of the camera is updated in the repeated operation process, and parameters of the neural network model are kept unchanged; and then, repeatedly operating the projection ray calculation module, the discrete sampling and rendering module and the implicit reconstruction module again, and only updating parameters of the neural network model in the repeated operation process.
For the specific limitations of the above apparatus, reference may be made to the above limitations of the incremental three-dimensional reconstruction method, which are not described herein again. The various modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
The device uses a nerve radiation field technology to implicitly reconstruct a three-dimensional model, input data is a picture sequence, and parameters to be estimated are the posture and the three-dimensional model of a camera; the rendering process from the 3D voxel model to the 2D image is realized by using a micro-rendering algorithm by using a volume rendering principle, so that the three-dimensional model can be subjected to nonlinear optimization by using differentiation, and end-to-end three-dimensional reconstruction is realized; based on the method, a plurality of images which are correspondingly obtained by shooting the target scene in multiple angles are traversed, and an incremental reconstruction result is finally obtained, so that the method is simple and convenient to operate and high in precision.
Fig. 3 shows the effect of applying the above-described exemplary scheme incremental reconstruction.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory storing a computer program, and all or part of the procedures in the method of the above embodiment are involved.
In one embodiment, a computer-readable storage medium having a computer program stored thereon is provided, which relates to all or part of the processes of the method of the above embodiments.
The above specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
All the technical features of the above embodiments can be arbitrarily combined (as long as there is no contradiction between the combinations of the technical features), and for brevity of description, all the possible combinations of the technical features in the above embodiments are not described; these examples, which are not explicitly described, should be considered to be within the scope of the present description.
The present invention has been described in considerable detail by the general description and the specific examples given above. It should be noted that it is obvious that several variations and modifications can be made to these specific embodiments without departing from the inventive concept, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. An incremental three-dimensional reconstruction method, comprising the steps of:
s1: acquiring a picture sequence of a target scene; the picture sequence comprises a plurality of images which are obtained correspondingly by shooting a target scene from multiple angles; performing the following steps S2-S4 for any image in the sequence of pictures;
s2: calculating a back projection ray from each pixel point of the image to the center position of the camera;
s3: setting a depth range of the back projection ray, discretely sampling a series of 3D points on the back projection ray in the depth range, solving through a neural network model represented by a multilayer perceptron, and performing micro-rendering on pixel points by using a volume rendering principle to obtain a rendering result of the pixel points, wherein gradient information is obtained in the micro-rendering process;
s4: randomly sampling pixel points on the image, carrying out implicit reconstruction on the model, and obtaining an initial reconstruction result after iterative calculation and rendering according to the steps S2 and S3; updating parameters of the neural network model and attitude information of the camera in the implicit reconstruction process, and utilizing gradient information obtained in the micro-rendering process;
s5: sequentially taking other images in the image sequence, and performing incremental reconstruction based on a reconstruction result of a previous image; traversing the picture sequence to finally obtain an incremental reconstruction result;
wherein each incremental reconstruction comprises: randomly initializing the camera pose corresponding to the current image, and repeatedly executing the steps S2, S3 and S4, wherein only the pose of the camera is updated in the repeated execution process, and the parameters of the neural network model are kept unchanged; then, steps S2, S3, and S4 are repeatedly performed again, and only the parameters of the neural network model are updated during the repeated execution.
2. The incremental three-dimensional reconstruction method of claim 1, wherein the pose information of the camera comprises a camera position and a viewing direction.
3. The incremental three-dimensional reconstruction method according to claim 2, wherein the inputs of the neural network model are camera position, observation direction and observed 3D coordinates, and the output is color c and volume density σ of the 3D point.
4. The incremental three-dimensional reconstruction method according to claim 1, wherein in step S4, the parameters of the neural network model and the pose information of the camera are updated, specifically: and calculating the gradient of the neural network model and the gradient of the camera parameter by using the gradient information obtained in the micro-rendering process, and updating the parameter of the neural network model and the attitude information of the camera by using an Adam algorithm.
5. The incremental three-dimensional reconstruction method according to claim 1, wherein between step S3 and step S4, a difference between a rendered color and an actual picture color is further calculated as an error of reconstruction.
6. The incremental three-dimensional reconstruction method according to claim 1, wherein in step S4, the randomly sampling pixel points on the image, specifically extracting from the region of interest of each image; the determination method of the interest area comprises the following steps: and (3) convolving the gradient image by using the blob and the corner kernel, then executing non-maximum suppression to obtain an image interest point, and then generating an interest region by using a morphological dilation algorithm.
7. The incremental three-dimensional reconstruction method according to claim 1, wherein the incremental reconstruction in step S5 further comprises the following steps in consideration of dynamic object or illumination effect:
firstly, using the rendering equation capable of micro-rendering in the step S3 as the static part of the model, then adding the transient part, and rendering the color and density of the transient, wherein the density allows the change in the training image;
secondly, an uncertainty field is allowed to be transmitted by the transient part, so that the model can adjust reconstruction loss and ignore unreliable pixels and 3D positions;
and modeling the color of each pixel into isotropic normal distribution, performing maximum likelihood estimation, and rendering the variance of the distribution as a transient color by using volume rendering.
8. An incremental three-dimensional reconstruction apparatus, comprising:
the image sequence acquisition module is used for acquiring an image sequence of a target scene; the picture sequence comprises a plurality of images which are obtained correspondingly by shooting a target scene from multiple angles;
the projection ray calculation module is used for calculating a back projection ray from each pixel point of the image to the center position of the camera aiming at the currently selected image;
the discrete sampling and rendering module is used for setting a depth range of the back projection ray, discretely sampling a series of 3D points on the back projection ray in the depth range, solving the points through a neural network model represented by a multilayer perceptron, and then carrying out micro-rendering on the pixel points by using a volume rendering principle to obtain a rendering result of the pixel points, wherein gradient information is obtained in the micro-rendering process;
the implicit reconstruction module is used for randomly sampling pixel points on the image, performing implicit reconstruction on the model, and obtaining an initial reconstruction result by iteratively operating the projection ray calculation module and the discrete sampling and rendering module; updating parameters of the neural network model and attitude information of the camera in the implicit reconstruction process, and utilizing gradient information obtained in the micro-rendering process;
the incremental reconstruction module is used for sequentially taking other images in the picture sequence and carrying out incremental reconstruction based on the reconstruction result of the previous image; traversing the picture sequence to finally obtain an incremental reconstruction result;
wherein each incremental reconstruction comprises: randomly initializing a camera pose corresponding to a current image, and repeatedly operating a projection ray calculation module, a discrete sampling and rendering module and an implicit reconstruction module, wherein only the pose of the camera is updated in the repeated operation process, and parameters of the neural network model are kept unchanged; and then, repeatedly operating the projection ray calculation module, the discrete sampling and rendering module and the implicit reconstruction module again, and only updating parameters of the neural network model in the repeated operation process.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111470822.2A CN114140510B (en) | 2021-12-03 | 2021-12-03 | Incremental three-dimensional reconstruction method and device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111470822.2A CN114140510B (en) | 2021-12-03 | 2021-12-03 | Incremental three-dimensional reconstruction method and device and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114140510A true CN114140510A (en) | 2022-03-04 |
| CN114140510B CN114140510B (en) | 2024-09-13 |
Family
ID=80387799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111470822.2A Active CN114140510B (en) | 2021-12-03 | 2021-12-03 | Incremental three-dimensional reconstruction method and device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114140510B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115049783A (en) * | 2022-05-20 | 2022-09-13 | 支付宝(杭州)信息技术有限公司 | Model determination method, scene reconstruction model, medium, device and product |
| CN115147558A (en) * | 2022-08-31 | 2022-10-04 | 北京百度网讯科技有限公司 | Training method of three-dimensional reconstruction model, three-dimensional reconstruction method and device |
| CN115359195A (en) * | 2022-07-18 | 2022-11-18 | 北京建筑大学 | Orthoimage generation method and device, storage medium and electronic equipment |
| CN116958453A (en) * | 2023-09-20 | 2023-10-27 | 成都索贝数码科技股份有限公司 | Three-dimensional model reconstruction method, device and medium based on nerve radiation field |
| CN117372602A (en) * | 2023-12-05 | 2024-01-09 | 成都索贝数码科技股份有限公司 | Heterogeneous three-dimensional multi-object fusion rendering method, equipment and system |
| CN117974904A (en) * | 2024-02-27 | 2024-05-03 | 北京数原数字化城市研究中心 | Three-dimensional reconstruction model generation method, three-dimensional reconstruction device and related equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170019653A1 (en) * | 2014-04-08 | 2017-01-19 | Sun Yat-Sen University | Non-feature extraction-based dense sfm three-dimensional reconstruction method |
| US20180205941A1 (en) * | 2017-01-17 | 2018-07-19 | Facebook, Inc. | Three-dimensional scene reconstruction from set of two dimensional images for consumption in virtual reality |
| CN110490917A (en) * | 2019-08-12 | 2019-11-22 | 北京影谱科技股份有限公司 | Three-dimensional rebuilding method and device |
| CN113160296A (en) * | 2021-03-31 | 2021-07-23 | 清华大学 | Micro-rendering-based three-dimensional reconstruction method and device for vibration liquid drops |
| CN113538682A (en) * | 2021-07-19 | 2021-10-22 | 北京的卢深视科技有限公司 | Model training method, head reconstruction method, electronic device, and storage medium |
-
2021
- 2021-12-03 CN CN202111470822.2A patent/CN114140510B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170019653A1 (en) * | 2014-04-08 | 2017-01-19 | Sun Yat-Sen University | Non-feature extraction-based dense sfm three-dimensional reconstruction method |
| US20180205941A1 (en) * | 2017-01-17 | 2018-07-19 | Facebook, Inc. | Three-dimensional scene reconstruction from set of two dimensional images for consumption in virtual reality |
| CN110490917A (en) * | 2019-08-12 | 2019-11-22 | 北京影谱科技股份有限公司 | Three-dimensional rebuilding method and device |
| CN113160296A (en) * | 2021-03-31 | 2021-07-23 | 清华大学 | Micro-rendering-based three-dimensional reconstruction method and device for vibration liquid drops |
| CN113538682A (en) * | 2021-07-19 | 2021-10-22 | 北京的卢深视科技有限公司 | Model training method, head reconstruction method, electronic device, and storage medium |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115049783A (en) * | 2022-05-20 | 2022-09-13 | 支付宝(杭州)信息技术有限公司 | Model determination method, scene reconstruction model, medium, device and product |
| CN115049783B (en) * | 2022-05-20 | 2024-04-02 | 支付宝(杭州)信息技术有限公司 | Model determining method, scene reconstruction model, medium, equipment and product |
| CN115359195A (en) * | 2022-07-18 | 2022-11-18 | 北京建筑大学 | Orthoimage generation method and device, storage medium and electronic equipment |
| CN115147558A (en) * | 2022-08-31 | 2022-10-04 | 北京百度网讯科技有限公司 | Training method of three-dimensional reconstruction model, three-dimensional reconstruction method and device |
| CN115147558B (en) * | 2022-08-31 | 2022-12-02 | 北京百度网讯科技有限公司 | Training method of three-dimensional reconstruction model, three-dimensional reconstruction method and device |
| CN116958453A (en) * | 2023-09-20 | 2023-10-27 | 成都索贝数码科技股份有限公司 | Three-dimensional model reconstruction method, device and medium based on nerve radiation field |
| CN116958453B (en) * | 2023-09-20 | 2023-12-08 | 成都索贝数码科技股份有限公司 | Three-dimensional model reconstruction method, device and medium based on nerve radiation field |
| CN117372602A (en) * | 2023-12-05 | 2024-01-09 | 成都索贝数码科技股份有限公司 | Heterogeneous three-dimensional multi-object fusion rendering method, equipment and system |
| CN117372602B (en) * | 2023-12-05 | 2024-02-23 | 成都索贝数码科技股份有限公司 | Heterogeneous three-dimensional multi-object fusion rendering method, equipment and system |
| CN117974904A (en) * | 2024-02-27 | 2024-05-03 | 北京数原数字化城市研究中心 | Three-dimensional reconstruction model generation method, three-dimensional reconstruction device and related equipment |
| CN117974904B (en) * | 2024-02-27 | 2024-08-23 | 北京数原数字化城市研究中心 | Three-dimensional reconstruction model generation method, three-dimensional reconstruction device and related equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114140510B (en) | 2024-09-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114140510B (en) | Incremental three-dimensional reconstruction method and device and computer equipment | |
| Le Pendu et al. | Light field inpainting propagation via low rank matrix completion | |
| Min et al. | Depth video enhancement based on weighted mode filtering | |
| Reinbacher et al. | Real-time panoramic tracking for event cameras | |
| US9253415B2 (en) | Simulating tracking shots from image sequences | |
| TWI738196B (en) | Method and electronic device for image depth estimation and storage medium thereof | |
| da Silveira et al. | Dense 3d scene reconstruction from multiple spherical images for 3-dof+ vr applications | |
| CN111080776B (en) | Human body action three-dimensional data acquisition and reproduction processing method and system | |
| CN107862733B (en) | Large-scale scene real-time three-dimensional reconstruction method and system based on sight updating algorithm | |
| JP2024507727A (en) | Rendering a new image of a scene using a geometric shape recognition neural network conditioned on latent variables | |
| US20200035011A1 (en) | Rendering an object | |
| Franke et al. | Vet: Visual error tomography for point cloud completion and high-quality neural rendering | |
| CN115564639A (en) | Background blurring method and device, computer equipment and storage medium | |
| CN111882613B (en) | Visual odometer method, device, storage medium and equipment based on edge semantics | |
| Hyeon et al. | Automatic spatial template generation for realistic 3d modeling of large-scale indoor spaces | |
| CN115937002B (en) | Method, apparatus, electronic device and storage medium for estimating video rotation | |
| Yang et al. | Deep convolutional grid warping network for joint depth map upsampling | |
| Zhang et al. | A registration method for augmented reality system based on visual slam | |
| CN117372607A (en) | Three-dimensional model generation method and device and electronic equipment | |
| CN113034675B (en) | Scene model construction method, intelligent terminal and computer readable storage medium | |
| US11790606B2 (en) | Determining camera rotations based on known translations | |
| CN116681839A (en) | Live three-dimensional target reconstruction and singulation method based on improved NeRF | |
| CN115409949A (en) | Model training method, visual angle image generation method, device, equipment and medium | |
| CN117252914A (en) | Training method and device of depth estimation network, electronic equipment and storage medium | |
| CN117496074B (en) | Efficient three-dimensional scene reconstruction method suitable for rapid movement of camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |