CN116543117B - A high-precision three-dimensional modeling method for large scenes from drone images - Google Patents

Abstract

The invention discloses a high-precision large-scene three-dimensional modeling method for unmanned aerial vehicle images, which comprises three steps in total, wherein the method avoids the traditional oblique photogrammetry three-dimensional modeling method, an unmanned aerial vehicle aerial image set is divided according to a circular surrounding track, the pose of an unmanned aerial vehicle camera is recovered by using an SFM algorithm after feature extraction, matching and geometric verification of the divided image set, then sub NeRF is trained, and finally sub NeRF around a target visual angle is combined, so that the implicit construction of a large-scene three-dimensional model is completed; through experimental tests, the method achieves good effect and can reconstruct the ground object with smooth surface and small cross section.

Description

A high-precision three-dimensional modeling method for large scenes from drone images

Technical field

The invention relates to the technical field of three-dimensional modeling, and in particular to a high-precision three-dimensional modeling method of large scenes for drone images.

Background technique

Real-life 3D can truly and orderly reflect large-scale spatio-temporal information of human production, life and ecological space. It is an important new integrated system that promotes the development of smart cities and smart digital economy. Three-dimensional scene construction extends traditional 2D data to 3D data and uses it as a core data structure to realize a real-scene environment, replacing the traditional pure geometric visualization architecture of points, lines, and surfaces. Real-life 3D enables computers to comprehensively and three-dimensionally present and perceive the current status and spatial distribution of various natural resource elements. In addition, it can also accurately reflect the spatial distribution of terrain, surface texture details, and morphological characteristics of ground objects in a high-definition, visual way. information. Therefore, the construction of real-life three-dimensional models is an emerging technology that supports remote sensing mapping theory and application issues. It has important scientific research value and practical significance, and provides technical support for the development of digital twins and metaverses. In addition, real-life 3D modeling has also been widely used in the fields of urban planning, CIM, urban transportation, geological surveying and mapping, driverless driving and virtual geographical environment.

As the means of obtaining geographic information data become increasingly abundant, modeling methods for constructing three-dimensional scenes through different data sources are also emerging. Commonly used three-dimensional modeling methods include the following, such as manual modeling through software such as Sketchup and 3dMax, and manual construction of BIM through software such as Revit. Although the model obtained by this method is precise enough, it is time-consuming, labor-intensive, inefficient, and difficult to meet the needs of large-scale users. Requirements for scope scenario modeling. Or obtain the building white film by stretching the two-dimensional vector planar building in CAD software to the height of the building. Although this method does not require manual modeling, it is difficult to obtain accurate data for the height of the building, and the model lacks texture and shape. Then there is the laser point cloud modeling, which mostly uses airborne lidar to construct the target object point cloud and then generate a triangular surface grid. This method has strong resistance to light and wind speed interference and high accuracy, but it has high cost and data noise problems. Data inconsistency remains a challenge. There are also street scene image modeling captured by moving vehicles, aerial photogrammetry modeling, etc., but none of them can reconstruct a relatively complete three-dimensional scene. As for the 3D modeling and reconstruction effect of images obtained through online crowdsourcing, the effect heavily depends on the coverage of the scene by online images.

Therefore, those skilled in the art are committed to developing a high-precision large-scene three-dimensional modeling method for drone images to solve the above-mentioned shortcomings of the existing technology.

Contents of the invention

In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is the defective problem that traditional photogrammetric three-dimensional modeling cannot well reconstruct smooth surfaces and small cross-section features.

In order to achieve the above purpose, the present invention provides a high-precision large-scene three-dimensional modeling method of drone images. The method includes the following steps:

Step 1. Acquisition and processing of drone images;

Step 2. Construction of a single NeRF;

Step 3. Merge the previously constructed NeRF to obtain a three-dimensional scene from any viewing angle;

Further, for step 1, the acquisition and processing of drone images can be divided into drone aerial images, dividing image sets, batch importing images, extracting image features, matching image features, geometric verification, and extracting camera positions in order. posture;

The criteria for dividing the image set are that it needs to cover a certain scene, and there is a high degree of overlap between the subset and adjacent subsets; the subset is divided by the trajectory of the drone around the route, and each UAV images taken in a circular route are a subset;

The extraction of influence features uses the SIFT algorithm (Scale Invariant Feature Transform) to extract the image features of drone aerial images, and uses the SIFTGPU graphics card to accelerate to achieve real-time calculation speed; the SIFT algorithm operates on different scale spaces Find feature points, calculate the direction of the feature points, and generate descriptors;

The matching influence feature uses the Brute-force algorithm for feature point matching. The Brute-force algorithm traverses each pair of feature points, calculates the distance between each pair of feature points, and determines whether each pair of feature points is a matching pair according to the threshold; for For any two images in the UAV aerial image collection, the feature points and descriptors extracted by SiftGPU are used to find matching pairs through the Brute-force algorithm;

The geometric verification uses the RANSAC algorithm to randomly select matching pairs, calculates the fitting matrix, and determines whether the matching pairs are reasonable by calculating the fitting error; the geometric verification can effectively improve the matching accuracy and avoid matching errors;

The incremental SFM algorithm is used to calculate the camera pose when extracting the camera pose; the incremental SFM algorithm can gradually perform three-dimensional reconstruction and effectively process large-scale image sequences; the incremental SFM algorithm can be divided into It is initialization and incremental reconstruction; the initialization includes triangulation and essential matrix decomposition;

Further, for step 2, the single NeRF is constructed by pre-constructing a fully connected neural network (MLP) and setting multi-resolution hash coding and spherical harmonic function coding rules; the construction of the single NeRF enables A ray is emitted for each pixel of the image captured by the camera, and coarse sampling is performed on the ray; the sampling point is After the coordinates are encoded, the fully connected neural network is input together with the appearance embedding vector to perform a round of fine sampling; the probability density function of the sampling points in one round of fine sampling is used to guide the second round of fine sampling, and the /> of the sampling points are After coordinate encoding, the fully connected neural network is input together with the appearance embedding vector to output the color and volume density of each sampling point; the color of the sampling point in the second round of fine sampling is accumulated and integrated through volume rendering to obtain the pixels corresponding to each ray. Color, and compare it with the real value to calculate LOSS, and continue to iterate the process until LOSS is reduced to a lower value;

Further, for step 3, the process of merging previously constructed NeRFs is to select sub-NeRFs and render target perspective images; the rule for selecting sub-NeRFs is to use a given target perspective as the center of the circle and draw a circle with a preset radius, If the origin of the sub-NeRF is projected within the circle, then the sub-NeRF is selected; the rendering target perspective image uses the IDW inverse distance weighting algorithm to interpolate between the images of the rendering target perspective;

Further, for step 3, connecting multiple images from the target perspective to form a trajectory can achieve the effect of roaming in a three-dimensional space;

Using the above solution, the invention discloses a high-precision large-scene three-dimensional modeling method for drone images, which has the following advantages:

(1) A high-precision large-scene three-dimensional modeling method of UAV images according to the present invention selects UAV aerial images as the data source and makes full use of the high spatial resolution, wide imaging range and high overlap rate of UAV images. advantages for image acquisition.

(2) A high-precision large-scene three-dimensional modeling method of UAV images of the present invention avoids the use of traditional oblique photogrammetry three-dimensional modeling methods, but proposes a new reconstruction method by constructing neural radiation fields NeRF For a large-scale three-dimensional model, the UAV aerial image set is divided, feature extraction matching and geometric verification are performed, sub-NeRFs are trained, and sub-NeRFs are finally merged to complete the implicit construction of a large-scene three-dimensional model.

In summary, the present invention discloses a high-precision large-scene three-dimensional modeling method for UAV images, which avoids using the traditional oblique photogrammetry three-dimensional modeling method, and divides the UAV aerial image set into a circular shape. Divide around the track, and after feature extraction, matching and geometric verification of the divided image set, use the SFM algorithm to restore the pose of the UAV camera, then train the sub-NeRF, and finally merge the sub-NeRF around the target perspective. Complete the implicit construction of 3D models of large scenes; through experimental testing, good results have been achieved, and smooth surfaces and small cross-section features can be well reconstructed.

The concepts, specific technical solutions and technical effects produced by the present invention will be further described below in conjunction with specific embodiments to fully understand the purpose, features and effects of the present invention.

Description of the drawings

Figure 1 is a flow chart of a high-precision large-scene three-dimensional modeling method for drone images according to the present invention;

Figure 2 is a schematic diagram of the circular orbit of the UAV;

Figure 3 is a schematic diagram of the scene boundary.

Detailed ways

Several preferred embodiments of the present invention are introduced below to make the technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments. These embodiments are illustrative descriptions, and the protection scope of the present invention is not limited to the embodiments mentioned herein.

If there are experimental methods that do not indicate specific conditions, they are usually implemented according to conventional conditions, such as relevant instructions or manuals.

As shown in Figures 1 to 3, the high-precision large-scene three-dimensional modeling method of drone images of the present invention is implemented as follows:

Step 1. The route design scheme adopted this time is a circumnavigation route (meaning that the drone flies around a circular or elliptical route according to a pre-planned path during flight, as shown in Figure 2). Through automatic route planning The software inputs the flight mission requirements and terrain data, and automatically generates the optimal route plan; the drone's automatic flight system performs the flight mission according to the planned route and completes the image collection work;

After collection, the image set is divided according to the trajectory of the drone around the route. The camera on the drone will point to the same point of interest on each circular route. The point of interest pointed by the camera on the circular route is the origin of each sub-NeRF. Therefore, the drone images taken by each circular route are regarded as a subset;

The SIFT algorithm is used to efficiently extract image features of drone aerial images on a heterogeneous computing system equipped with a GPU. SiftGPU processes pixels in parallel to build a Gaussian pyramid and detect Gaussian difference DoG (Difference of Gaussian) key points. It is generated based on the GPU list. SiftGPU uses The GPU/CPU hybrid method effectively builds a compact keypoint list, and finally processes the keypoints in parallel to obtain their directions and descriptors; the Brute-force algorithm first needs to define a matching threshold, that is, the distance threshold, when performing feature point matching, and then Traverse all feature point pairs and calculate the distance between them; if the distance is less than the threshold, add them to the matching pair list;

First, the RANSAC algorithm is used to eliminate incorrect matching point pairs and calculate the basic matrix. Specifically, eight pairs of matching point pairs are randomly and uniformly sampled. Based on the sampling point pairs, the normalized eight-point method is used to estimate the basic matrix and calculate whether the remaining matching point pairs satisfy the current basis. matrix, count the number of matching point pairs that satisfy the current basis as the current basis matrix score, repeat the above steps a set number of times, and obtain the basis matrix with the highest score; these eight point pairs satisfy the formula p' ^T Fp=0, p and p ' is the projection of the three-dimensional point P in each imaging plane, and F is the basic matrix to be obtained;

Then use the RANSAC algorithm to calculate the homography matrix and eliminate wrong matching point pairs. Specifically, four pairs of matching point pairs are randomly and uniformly sampled. Based on the sampling point pairs, the four-point method is used to estimate the homography matrix and calculate whether the remaining matching point pairs satisfy the current homography. matrix, count the number of matching point pairs that meet the current basis as the current homography matrix score, repeat the above steps a set number of times, and obtain the homography matrix with the highest score; the four-point method formula is p'=Hp, p and p' are The projection of the three-dimensional point P in each imaging plane, H is the homography matrix to be obtained; by substituting the coordinates of the four point pairs into the above formula, four equations can be constructed, and the above four equations are combined to form A system of linear equations. Solve the system of linear equations to obtain the homography matrix;

Incremental SFM is used to calculate the camera pose. First, the trajectory length of each feature point is calculated. The trajectory length represents the number of any feature points appearing in all pictures. Then a scene connected graph is established, and each picture is used as a node of the connected graph. If the number of matching feature point pairs between any two nodes is greater than a certain threshold, then connect the two nodes as an edge of the connected graph, select an edge, and select two intersection angles that are large enough (When triangulating all corresponding points, the median angle of the rays is generally not greater than 60 degrees and not less than 3 degrees). Seed images with a sufficient number of points with the same name and even distribution are used as the two nodes of the edge to robustly estimate this edge. The essential matrix corresponding to the edge (these two seed pictures), decompose the essential matrix, obtain the pose of the camera to which the two seed pictures belong (that is, the external parameters of the camera), and filter out the trajectory length of the feature points on the two seed pictures greater than 2's feature point pair is triangulated to obtain the initial reconstruction result, and this edge is deleted from the scene connected graph. At this point, the initialization of the incremental SFM is completed;

After completing the initialization of incremental SFM, if there are still edges in the scene connected graph, then select an edge from the scene connected graph that satisfies the feature points whose trajectory length is greater than 2 on the corresponding two pictures and has been Maximize the subset of reconstructed 3D points, use the PnP method to estimate the camera pose (external parameters of the camera), and filter out feature point pairs whose trajectory length is greater than 2 on the two pictures corresponding to the new edge. Triangulate, delete the newly added edge in the scene connected graph, and execute the BA algorithm until there is no edge in the scene connected graph. Then the incremental SFM scene incremental reconstruction and camera pose recovery are completed;

Step 2. A spatial scene based on NeRF is represented as a function whose input is a five-dimensional vector, which is implicitly expressed by an MLP network to describe the shape of the three-dimensional model in the spatial scene and the color information observed from different directions; the five-dimensional vector The input of the dimensional vector function is a 3D position vector x = (x, y, z) and a 2D direction vector x=(x,y,z) is the coordinate of the three-dimensional point in the scene,/> Represents the direction angle of the observation direction in the spherical coordinate system (measured from the positive half-axis of the x-axis, rotating along the direction of the yoz plane) and the polar viewing angle (measured from the positive half-axis of the z-axis, rotating along the direction of the xoy plane) ;The output of the function is the color c=(r,g,b) and the volume density σ(x) of the ray emitted by the camera arriving at the three-dimensional position x along the direction d; the volume density σ(x) represents the ray ending at x The differential probability of the infinitesimal particle position; this five-dimensional vector function can be written as:

F _Θ (x, d) = (c, σ)

The training process of this fully connected neural network is to continuously adjust the weight parameter Θ of the network model so that it can finally output the corresponding color and volume density after a given 5D coordinate input; in order to ensure the multi-view consistency of the network output results, The volume density σ is only a function σ(x) of the spatial position x, while the color vector c is a function c(x,d) of x and d;

The specific process of multi-resolution hash encoding is as follows: (1) For a given input coordinate x, find its surrounding voxels at different resolution levels, and map their integer coordinates to each vertex of the voxel through hash Specify the index; (2) Find the feature vector corresponding to each vertex index of the grid in the hash table at different levels; (3) According to the relative position of the input coordinate x in the grid at different levels, divide the grids at different levels into The feature vector of each vertex of the grid is interpolated into a feature vector through trilinear interpolation; (4) The feature vectors of different resolution grids are spliced, that is, multi-resolution hash coding is completed;

The performance in graphics rendering is that when the light source rotates, as long as the transformed generalized Fourier coefficients are calculated synchronously, it can ensure that the lighting effect of the picture will not jitter or jump; when the Laplacian in spherical coordinates After the equation separates the variables, the function about the polar angle θ is the joint Legendre polynomial About direction angle/> The function of is/> Spherical harmonics are defined as:

In the above formula, Represented as a unit vector pointing to a point in spherical coordinates/> l represents the degree, m represents the order, they are both integers, l≥0,-l≤m≤l; A _{l, m} are normalization coefficients, such that/> The integral on the unit sphere is equal to 1;

The spherical harmonic function can be regarded as mapping each point on the unit sphere (or each direction in the three-dimensional space) to a complex function value; for the input direction d, it can be encoded as the basis function and spherical harmonic function of the spherical harmonic function The combination of coefficients is then input into the color MLP network;

In order to enable NeRF to adapt to different lighting changes, GLO technology is used, that is, each image is assigned a corresponding real-valued appearance embedding vector. Its length is - ^(a) ; then the appearance is embedded into the vector/> As the input of the second color MLP network, the weight parameter Θ of the network model is the same as /> Embeddings are optimized together;

Using the large cube surrounding the model set in multi-level hash coding, the values of near and far are determined by calculating the intersection of the camera ray and the large cube, as shown in Figure 3. At this point, the scene boundary range is determined;

For volume rendering, the discrete sampling summation method is used to fit the integral of continuous sampling, that is, t _- to t _f are divided into N uniform intervals, and then a sample point is randomly selected from each interval, then the i-th sample point can be Expressed as:

Discrete sampling summation ensures that MLP's macroscopically continuous positions are estimated during the continuous training and optimization process, thereby ensuring the continuity of scene representation. Based on the above ideas, the form of converting integrals into summation is:

In the above formula, δ _i =t _i+1 -t _i represents the distance between adjacent sample points, σ _i and c _i are the volume density and color of sample point i, calculated from the set of σ _i and c _i values is differentiable, the transparency of sample point i is 1-exp(-σ _i δ _i );

For stratified sampling, NeRF uses a stratified sampling scheme to sample points along rays in space, that is, coarse sampling is performed first, and then the results of coarse sampling are used to guide the next step of fine sampling; coarse sampling is uniform sampling, in Sample N points uniformly within a predefined range from the camera; divide the range from near to far into n equal parts, and place N sampling points evenly on the ray; pass the position information of these N sampling points through multi-level hashing Encoding, the direction information of the point is encoded by spherical harmonic function and spliced with the appearance embedding vector, and then sent to the MLP network. This MLP network is called a coarse network, and the volume density and color of the N sampling points are obtained; here On a ray, the corresponding pixel color can be regarded as the weighted sum of the colors of all sampling points, that is

w _i =T _i (1-exp(-σ _i δ _i ))

The size of the sampling weight reflects the distance between the sampling point and the surface of the three-dimensional model. The second sampling will focus on sampling the area with a larger weight in the first sampling result; that is, more sampling will be done in the area with a large weight value, and the area with a larger weight value will be sampled more. Smaller places need less sampling;

These weights are normalized to

In the above formula, N _c is the number of sampling points for coarse sampling, which It can be regarded as the probability density function of objects along the ray. Through this probability density function, the distribution of objects in the ray direction can be roughly obtained. Next, according to this probability density function, fine sampling, that is, inverse transform sampling ( inversetransformsampling), get N _f new sampling points. These new sampling points are gathered at locations with high coarse sampling weights and sparse at locations with low coarse sampling weights. These new sampling points are close to the object surface; then, N _c + The position and direction information of N _f sampling points are encoded and input into the MLP network. This network is called a fine network. The volume density and color information of the new sampling points are obtained, and then calculated according to the volume rendering formula. Corresponding to the pixel color, the formula is as follows:

The loss function Loss is calculated by the total squared error between the pixel color output by the coarse network and the pixel color output by the fine network and the true pixel color:

In the above formula, R is a set of rays in each training batch, and/> They are the pixel colors obtained by body rendering after coarse sampling and fine sampling of camera rays, /> is the true value of the image pixel color (GroundTruth, GT); even if the final rendering result comes from/> It will also make/> The loss is minimized so that the weight distribution from the coarse network can be used to distribute samples in the fine network;

When the value of Loss reaches a lower value, the training can be regarded as over. At this time, the iteration is stopped. For any perspective in the scene, a virtual camera can be placed. This camera emits multiple rays, and on each ray For coarse sampling, the weight of each sampling point is obtained after sending it to the coarse network, and then fine sampling is performed. Two rounds of sampling points are sent to the fine network, and the color of each pixel is obtained after volume rendering. These pixels together form an image; that is, Images observed from any angle in the scene can be obtained;

Step 3. After training the NeRF model on each subset of the UAV image collection, we obtain each point of interest pointed by the camera on its circular route as the origin, and the large cube constructed within its circular route as the boundary. sub-NeRF, in order to improve efficiency, only render the relevant sub-NeRF for a given target perspective, then render the color information from the filtered sub-NeRF, and use the target camera origin o and the filtered sub-NeRF center point _xi The InverseDistance Weighted (IDW) is interpolated between them, (specifically, each weight is calculated as w _i ∝d(o,xi ₎ ^-p , where p affects the mixing rate between sub-NeRF renderings, d(c, _xi ) is the distance between the target camera origin o and the filtered sub-NeRF center point x _i ), and finally a new view from the target perspective is obtained; views from multiple target perspectives are connected to form a trajectory, You can achieve the effect of roaming in three-dimensional space;

It can be seen from the analysis of Embodiment 1 that the high-precision large-scene three-dimensional modeling method of UAV images of the present invention avoids the use of traditional oblique photogrammetry three-dimensional modeling methods, and divides the UAV aerial image collection into circular shapes. Divide around the track, and after feature extraction, matching and geometric verification of the divided image set, use the SFM algorithm to restore the pose of the UAV camera, then train the sub-NeRF, and finally merge the sub-NeRF around the target perspective. Complete the implicit construction of 3D models of large scenes; through experimental testing, good results have been achieved, and smooth surfaces and small cross-section features can be well reconstructed.

The preferred embodiments of the present invention are described in detail above. It should be understood that those skilled in the art can make many modifications and changes according to the concept of the present invention without creative efforts. Therefore, any technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited testing on the basis of the prior art based on the concept of the present invention should be within the scope of protection determined by the claims.

Claims (2)

1. A high-precision three-dimensional modeling method of large scenes from drone images, which is characterized by: The method includes the following steps: Step 1. Acquisition and processing of UAV images. The acquisition and processing of UAV images can be divided into UAV aerial images, dividing image sets, batch importing images, extracting image features, matching image features, and geometry in order. Verify and extract camera pose; Step 2. Construction of a single NeRF; Step 3. Merge the previously constructed NeRF to obtain a three-dimensional scene from any viewing angle; The criteria for dividing the image set are that it needs to cover a certain scene, and there is a high degree of overlap between the subset and adjacent subsets; the subset is divided by the trajectory of the drone around the route, and each UAV images taken in a circular route are a subset; The extracted image features use the SIFT algorithm to extract the image features of drone aerial images, and use the SIFTGPU graphics card to accelerate to achieve real-time calculation speed; the SIFT algorithm searches for feature points in different scale spaces, calculates the direction of the feature points, and simultaneously Generate descriptors; The matching image features use the Brute-force algorithm for feature point matching. The Brute-force algorithm traverses each pair of feature points, calculates the distance between each pair of feature points, and determines whether each pair of feature points is a matching pair according to the threshold; for For any two images in the UAV aerial image collection, the feature points and descriptors extracted by SiftGPU are used to find matching pairs through the Brute-force algorithm; The geometric verification uses the RANSAC algorithm to randomly select matching pairs, calculates the fitting matrix, and determines whether the matching pairs are reasonable by calculating the fitting error; the geometric verification can effectively improve the matching accuracy and avoid matching errors; The incremental SFM algorithm is used to calculate the camera pose when extracting the camera pose; the incremental SFM algorithm can gradually perform three-dimensional reconstruction and effectively process large-scale image sequences; the incremental SFM algorithm can be divided into It is initialization and incremental reconstruction; the initialization includes triangulation and essential matrix decomposition; The single NeRF is constructed by pre-constructing a fully connected neural network and setting multi-resolution hash coding and spherical harmonic function coding rules; the construction of the single NeRF enables each pixel of the image captured by cameras at different positions and orientations to Emit rays and perform rough sampling on the rays; encode the coordinates of the sampling points and input them into the fully connected neural network together with the appearance embedding vector to perform a round of fine sampling; use the probability density function of the sampling points from one round of fine sampling to guide the second round For fine sampling, the coordinates of the sampling points are encoded and input into the fully connected neural network together with the appearance embedding vector, and the color and volume density of each sampling point are output; the color of the sampling points in the second round of fine sampling is accumulated and integrated through volume rendering to obtain each The pixel color corresponding to the ray is compared with the real value to calculate the LOSS, and the construction process of a single NeRF is continuously iterated until the LOSS is reduced to a lower value; The process of merging previously constructed NeRFs is to select sub-NeRFs and render the target perspective image; the rule for selecting sub-NeRFs is to take the given target perspective as the center of the circle and draw a circle with a preset radius. If the origin of the sub-NeRF is projected on the circle within, then the sub-NeRF is selected; the IDW inverse distance weighting algorithm is used for the rendering target perspective image to interpolate between the images of the rendering target perspective. 2. A high-precision large-scene three-dimensional modeling method for drone images as claimed in claim 1, characterized in that: For step 3, connecting multiple images from the target perspective to form a trajectory can achieve the effect of roaming in the three-dimensional space.