CN115661345A

CN115661345A - Three-dimensional reconstruction method and system based on multi-track SLAM information

Info

Publication number: CN115661345A
Application number: CN202211295422.7A
Authority: CN
Inventors: 张双力; 丛林
Original assignee: Hangzhou Yixian Advanced Technology Co ltd
Current assignee: Hangzhou Yixian Advanced Technology Co ltd
Priority date: 2022-10-21
Filing date: 2022-10-21
Publication date: 2023-01-31

Abstract

The application relates to a three-dimensional reconstruction method and a three-dimensional reconstruction system based on multi-track SLAM information, wherein the method comprises the following steps: acquiring a track image, extracting and matching the features of the image to obtain an effective image pair set, and calculating an epipolar geometry and an interior point feature matching pair set of the image pair; calculating the quality of the evaluation track, selecting a reference track, and reconstructing according to the reference track to obtain a reference map; setting a track screening formula, and calculating and selecting a track to be registered; screening seed images from the track to be registered, and registering the seed images into a reference map; and according to the pose of the seed image in the reference map and the SLAM pose, calculating the similarity transformation between the reference map and an SLAM track coordinate system through a random sampling consistency algorithm, optimizing the similarity transformation to obtain optimal similarity transformation, and registering the whole track to be registered into the reference map through the optimal similarity transformation. The method solves the problems of low reconstruction speed, low accuracy and precision of a reconstructed model and incapability of multi-track three-dimensional reconstruction in the prior art.

Description

Three-dimensional reconstruction method and system based on multi-track SLAM information

Technical Field

The application relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional reconstruction method and a three-dimensional reconstruction system based on multi-track SLAM information.

Background

In the related art, a method of three-dimensional reconstruction includes: first, three-dimensional reconstruction is based on a conventional set of unordered images. However, due to lack of time-series information between images, if there are overlapping points, devices, and textures with similar appearances in a scene, partial images are likely to be registered at wrong positions, and a model after three-dimensional reconstruction is likely to be deformed or displaced. And secondly, performing three-dimensional reconstruction based on point cloud, and being mainly used for RGBD sensors. However, the method depends on the shape of the track and the quality of the point cloud, and the reconstruction speed is greatly influenced by the scale of the point cloud; in addition, if a structured repetitive region exists in a scene, ambiguity easily occurs in point cloud alignment, and the accuracy of a reconstructed model is affected. And thirdly, performing three-dimensional reconstruction based on single-track SLAM information. The method only supports single-acquired information of a single track, and relies on information covariance calculation of acquisition equipment and the acquisition track in an SLAM algorithm. Thus, the method does not support processing trajectories acquired multiple times separately, nor does it support optimizing joint factor graphs of trajectories obtained from multiple devices.

Therefore, in order to solve the above problems in the three-dimensional reconstruction technology, it is urgently needed to provide an effective solution.

Disclosure of Invention

The embodiment of the application provides a three-dimensional reconstruction method and a three-dimensional reconstruction system based on multi-track SLAM information, and aims to at least solve the problems that in the related technology, the reconstruction speed is low, the accuracy and precision of a reconstructed model are not high, and multi-track three-dimensional reconstruction cannot be carried out.

In a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method based on multi-track SLAM information, where the method includes:

acquiring a track image, performing feature extraction and image matching on the image to obtain an effective image pair set, calculating epipolar geometry, and screening out an interior point feature matching pair set which accords with the epipolar geometry relationship from each group of effective image pairs according to the relative position relationship of two standard cameras;

calculating and evaluating track quality according to the effective image pair in each track, selecting an optimal track as a reference track, and performing three-dimensional reconstruction according to the SLAM pose of the reference track to obtain a reference map;

according to the association matching information between the tracks and the reference map, a track screening formula is customized, and according to the track screening formula, the track with the highest score in all the tracks is calculated and selected as the track to be registered;

screening seed images from the track to be registered, registering the seed images into a reference map, and optimizing the pose of the seed images through a bundle adjustment optimization algorithm;

and according to the pose of the seed image in the reference map and the SLAM pose, calculating the similarity transformation between the reference map and the SLAM track coordinate system through a random sampling consistency algorithm, optimizing the similarity transformation to obtain the optimal similarity transformation, and registering the whole track to be registered into the reference map through the optimal similarity transformation.

In some of these embodiments, the acquiring a trajectory image comprises:

and respectively acquiring a plurality of SLAM tracks through various devices, and extracting the images and the poses of the key frames in the SLAM tracks.

In some embodiments, image matching the images to obtain the set of valid image pairs comprises:

and presetting a matching threshold, screening and matching images according to the matching threshold, and if the matching number is greater than the matching threshold, taking the matched image pair as an effective image pair.

In some embodiments, the calculating the evaluation track quality according to the effective image pair in each track, and the selecting the optimal track as the reference track includes:

and selecting different calculation and evaluation methods according to the SLAM pose information quality of the original track data, when the SLAM pose information quality of the original track data is poor, calculating and evaluating the internal characteristics through the track effective image pair and the image pair, and otherwise, when the SLAM pose quality of the original track data is high, calculating and evaluating the SLAM pose of the original track data.

In some embodiments, the computationally evaluating the internal features from the valid image pair and the image pair comprises:

acquiring the number of matching pairs in the interior point feature matching pair set of the effective image pair;

acquiring adjacent frames of effective images, calculating the distance between the adjacent frames of the images and the rotation component of the SLAM pose, and calculating the relevance of the adjacent frames of the images according to the distance and the component;

and calculating the track quality through a self-defined evaluation function according to the matching pair number and the adjacent frame association degree to obtain the optimal track.

In some of these embodiments, the computationally evaluating SLAM pose from raw trajectory data comprises:

calculating a conversion matrix between the effective image pairs according to the SLAM pose of the effective image pairs, and decomposing to obtain a rotation component of the SLAM pose;

comparing the rotation component of the SLAM pose with the epipolar geometry rotation component of the corresponding effective image pair, and calculating the angle difference between the two;

and calculating the track quality through a self-defined evaluation function according to the rotation component and the angle difference of the SLAM pose, and acquiring the optimal track.

In some of these embodiments, reconstructing the three-dimensional pose from the SLAM pose of the reference trajectory comprises:

triangularizing the map points, optimizing the positions and postures of the triangularized map points and the reference tracks through a light beam adjustment optimization algorithm, and finally combining the optimized map points.

In some of these embodiments, merging map points comprises:

and selecting merging points according to the space Euclidean distance between map points and the similarity between image observations corresponding to the map points, and merging the map points through the merging points.

In some embodiments, the association matching information between the track and the reference map comprises:

the number of effective image pairs formed between all images in the track to be selected and all images in the reference map and the total number of corresponding interior point feature matching pairs;

the variance of an interior point feature matching number set of each image pair in an effective image pair formed between all images in the trajectory to be selected and all images in the reference map;

and effective image pairs formed between all the images in the trajectory to be selected and all the images in the reference map: the area of the image of the reference map in space and the area of the track image to be selected in space.

In some embodiments, screening out seed images from the track to be registered comprises:

selecting different screening methods for screening the seed images according to the difference of the tracks to be registered, wherein the specific screening method comprises the following steps: and screening based on the uniformity of the image spatial distribution, screening based on the image correlation degree, and screening by combining the spatial distribution and the correlation degree.

In some of these embodiments, screening based on the uniformity of the spatial distribution of the image comprises:

and dividing the track to be registered into grids according to the spatial distribution of the camera frame positions, and uniformly selecting images from each grid according to the quantity required to be selected.

In some of these embodiments, screening based on image relevance includes:

and calculating the association degree between the track to be registered and the reference map according to the number of effective image pairs formed between each image in the track to be registered and the existing image in the reference map and the total number of corresponding interior point feature matching pairs, and selecting the image with the highest association degree.

In some of these embodiments, screening in combination with spatial distribution and relevance comprises:

dividing a grid sampling area according to the spatial distribution of the camera frame positions;

calculating the association degree between the track to be registered and the reference map and sequencing the association degree;

and sequentially selecting the image with the highest association degree in each sampling region through an octree sampling algorithm.

In some embodiments, optimizing the pose of the seed image by a bundle adjustment optimization algorithm comprises:

fixing the image pose in the reference map;

fixing 3D points not observed in the seed image;

and optimizing the positions and postures of map points observed in the seed images and the seed images.

In some embodiments, optimizing the similarity change to obtain an optimal similarity transformation includes:

and minimizing the pose error after the similarity transformation by using nonlinear optimization and a robust kernel function, and solving to obtain the optimal similarity transformation.

In some embodiments, optimizing the similarity change to obtain an optimal similarity transformation further includes:

and calculating the similarity transformation of the track image by setting different weight constraint terms for the original track data with different confidences acquired by different equipment and generated by different algorithms, thereby obtaining the optimal similarity transformation.

In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction system based on multi-track SLAM information, where the system includes:

the track image processing module is used for acquiring track images, performing feature extraction and image matching on the images to obtain an effective image pair set, calculating epipolar geometry, and screening out an interior point feature matching pair set which accords with the epipolar geometry from each group of effective image pairs according to the relative position relationship of two standard cameras;

the reference map building module is used for calculating and evaluating the track quality according to the effective image pairs in each track, selecting the optimal track as a reference track, and performing three-dimensional reconstruction according to the SLAM pose of the reference track to obtain a reference map;

the track registration module is used for customizing a track screening formula according to the correlation matching information between the track and the reference map, calculating and selecting the track with the highest score from all the tracks as a track to be registered according to the track screening formula,

screening out seed images from the track to be registered, registering the seed images into a reference map, optimizing the pose of the seed images by using a bundle adjustment optimization algorithm,

In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor, when executing the computer program, implements the method according to the first aspect.

In a fourth aspect, embodiments of the present application provide a storage medium, on which a computer program is stored, which when executed by a processor implements the method according to the first aspect.

Compared with the related technology, the three-dimensional reconstruction method based on the multi-track SLAM information, provided by the embodiment of the application, comprises the steps of obtaining track images, carrying out feature extraction and image matching on the images to obtain an effective image pair set, calculating epipolar geometry, and screening out an interior point feature matching pair set which accords with the epipolar geometry from each group of effective image pairs according to the relative position relationship of two standard cameras; calculating and evaluating track quality according to the effective image pair in each track, selecting an optimal track as a reference track, and performing three-dimensional reconstruction according to the SLAM pose of the reference track to obtain a reference map; according to the association matching information between the tracks and the reference map, customizing a track screening formula, and according to the track screening formula, calculating and selecting the track with the highest score from all the tracks as the track to be registered; screening seed images from a track to be registered, registering the seed images into a reference map, and optimizing the pose of the seed images through a bundle adjustment optimization algorithm; according to the pose of the seed image in the reference map and the SLAM pose, the similarity transformation between the reference map and the SLAM track coordinate system is calculated through a random sampling consistency algorithm, the similarity transformation is optimized to obtain the optimal similarity transformation, and the whole track to be registered is registered in the reference map through the optimal similarity transformation, so that the problems of low reconstruction speed, low accuracy and precision of a reconstructed model and incapability of multi-track three-dimensional reconstruction in the related technology are solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a flowchart of a three-dimensional reconstruction method based on multi-track SLAM information according to an embodiment of the present application;

FIG. 2 is a block diagram of a three-dimensional reconstruction system based on multi-track SLAM information according to an embodiment of the present application;

fig. 3 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to only those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but rather can include electrical connections, whether direct or indirect. Reference herein to "a plurality" means greater than or equal to two. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

The present embodiment provides a three-dimensional reconstruction method based on multi-track SLAM information, and fig. 1 is a flowchart of the three-dimensional reconstruction method based on multi-track SLAM information according to the embodiment of the present application, and as shown in fig. 1, the flowchart includes the following steps:

step S101, acquiring a track image, performing feature extraction and image matching on the image to obtain an effective image pair set, calculating epipolar geometry, and screening out an interior point feature matching pair set which accords with the epipolar geometry from each group of effective image pairs according to the relative position relationship of two standard cameras;

preferably, in this embodiment, a plurality of SLAM trajectories are respectively acquired by various devices, and an image and a pose of a key frame in the SLAM trajectories are extracted, where the acquired trajectories include: VISLAM trajectory (whose image pose has a true physical dimension), VSLAM trajectory (whose image pose does not have a true physical dimension), and VSLAM/VISLAM trajectory in combination with sensor readings, where sensors include GPS, bluetooth, WIFI, UWB, etc., and sensor readings include location and uncertainty, etc. It should be noted that the camera internal parameters of the image frames in the present embodiment have been calibrated, and participate in the three-dimensional reconstruction, but do not participate in the optimization.

Further, feature extraction and image matching are carried out on the obtained track images, the images are screened and matched according to a preset matching threshold, if the matching number is larger than the preset matching threshold, the matched image pairs are effective image pairs, and therefore an effective image pair set in the track is obtained. Calculating epipolar geometry, and screening an interior point feature matching pair set M meeting the epipolar geometry relationship from each group of effective image pairs according to the relative position relationship E of the two standard cameras _pq Where image p and image q are a set of valid image pairs.

Step S102, calculating and evaluating track quality according to effective images in each track, selecting an optimal track as a reference track, and performing three-dimensional reconstruction according to the SLAM pose of the reference track to obtain a reference map;

preferably, in this embodiment, different track quality calculation and evaluation methods are selected according to the SLAM pose information quality of the original track image data acquired in step S101, and an optimal track is selected from all tracks as a reference track. The evaluation method comprises the following steps:

1. when the SLAM pose information quality of the original data is poor, for example, the scene change is large, the illumination is low, the texture of a partial region is low, and the like, the quality of the SLAM track is judged in an auxiliary mode due to the lack of information, and the internal features are calculated and evaluated through a track effective image pair and an image pair. The specific evaluation steps are as follows:

s1: acquiring the number of matching pairs in the interior point feature matching pair set of the effective image pair;

s2: acquiring adjacent frames of effective images, calculating the distance between the adjacent frames of the images and the rotation component of the SLAM pose, and calculating the relevance of the adjacent frames of the images according to the distance and the component;

s3: and calculating the track quality through a self-defined evaluation function according to the matching pair number and the association degree of the adjacent frames, and taking the track group with the highest evaluation score as a reference track. The user-defined evaluation function is composed of matching items and adjacent frame association items, specifically, the scores of the matching items are obtained by calculating the number of average matching pairs, the scores of the adjacent frame association items are obtained by calculating the inter-frame distance and the rotation angular velocity of the image pose, and finally, when the total evaluation score is calculated, different weights need to be set and normalized.

2. When the SLAM pose quality of the original data is high, the scene texture is rich, and at the moment, if the first method is continuously adopted for evaluation and judgment, the scores of all tracks after evaluation are very close, and the optimal track cannot be selected. Therefore, at this time, the calculation evaluation is performed by the SLAM pose of the original trajectory data. The specific evaluation steps are as follows:

s1: calculating a transformation matrix between the effective image pairs according to the SLAM pose of the effective image pairs, and decomposing to obtain a rotation component of the SLAM pose

Wherein, visual aspect is abbreviated as vo and called SLAM pose;

s2: rotating component of SLAM pose with epipolar geometry rotation component of corresponding effective image pair

Comparing and calculating the angle difference between the two, wherein the epipolar geometry is abbreviated as eg and is also named as antipoleGeometry;

s3: and calculating the track quality through a self-defined evaluation function according to the rotation component and the angle difference of the SLAM pose, and acquiring the track with the minimum difference as a reference track.

It should be noted that the evaluation methods in this embodiment are all exemplary methods, and are not specifically limited, and the track quality may also be evaluated and calculated according to other information.

After the reference track is selected and obtained through the method, three-dimensional reconstruction is carried out according to the SLAM position and pose of the reference track, and a reference map is obtained. The specific three-dimensional reconstruction step comprises the following steps: triangularizing the map points, optimizing the poses of the triangularized map points and the reference tracks through a light beam adjustment optimization algorithm, and finally combining the optimized map points. Preferably, in this embodiment, a merging point is selected according to the spatial euclidean distance between the map points and the similarity between the image observations corresponding to the map points, and the map points are merged through the merging point. The method specifically comprises the following steps of:

firstly, traversing map points in a reference map, searching all map points in a certain radius around each point P to form a candidate set S, and marking the points in the set S as points Q;

and traversing the respective image observations of P and Q, acquiring an image I in the image observation from P and an image J in the image observation from Q, if I and J are valid image matching pairs, and the feature Fip corresponding to the point P in the image I and the feature Fjq corresponding to the point Q in the image J are within an inner point feature matching pair set of the image matching pairs of I and J, adding 1 to the combined value of P and Q, and when the combined value of P and Q exceeds a preset threshold value, combining P and Q.

It should be noted that the merging operation is a conventional map point merging method, and details are not described in this embodiment.

Compared with the three-dimensional reconstruction without prior (namely, without an initial value), the three-dimensional reconstruction is input with a better initial value through the track pose, so that the mapping result can be obtained more quickly, and the three-dimensional reconstruction is accelerated. In addition, the map point merging strategy in the embodiment can effectively avoid the problem that optimization falls into local optimization due to inaccurate SLAM pose, so that a map with higher precision can be obtained compared with a method only performing triangulation in the prior art. In addition, the three-dimensional reconstruction in the embodiment is established on the image information, and point cloud information is not needed.

Step S103, customizing a track screening formula according to the correlation matching information between the track and a reference map, and calculating and selecting the track with the highest score from all the tracks as a track to be registered according to the track screening formula;

preferably, in this embodiment, the track screening formula is customized according to the association matching information between the track and the reference map, where the association matching information between the track and the reference map includes:

1. the correlation strength is as follows: the number of effective image pairs formed between all images in the to-be-selected track and all images in the reference map and the total number of corresponding interior point feature matching pairs;

2. matching uniformity: the variance of an interior point feature matching number set of each image pair in an effective image pair formed by all images in the trajectory to be selected and all images in the reference map;

3. spatial uniformity: and valid image pairs formed between all the images in the trajectory to be selected and all the images in the reference map: the area of the image of the reference map in the space, the area of the track image to be selected in the space and the product of the two.

The track screening formula can be set through the associated information, for example, linear weighting calculation is directly performed on the associated information, and it should be noted that the formula is an exemplary formula and is not limited specifically.

And finally, screening out the track with the highest score as the track to be registered according to the calculation result.

S104, screening seed images from a track to be registered, registering the seed images into a reference map, and optimizing the pose of the seed images through a bundle adjustment optimization algorithm;

preferably, in this embodiment, different screening methods are selected to screen the seed images according to different tracks to be registered under actual conditions, and the specific screening method includes:

1. based on the spatial distribution: and screening based on the uniformity of the image spatial distribution. The method comprises the following specific steps: and dividing the track to be registered into grids according to the spatial distribution of the camera frame positions, and uniformly selecting images from each grid according to the quantity required to be selected. The method has the advantages of small calculation complexity, uniform spatial distribution of the selected images and suitability for the condition of more track total length and frame number. Although a poor quality image may be selected, the uniformly distributed seed image selected may provide more uniform constraints for subsequent registration of the entire track.

2. Based on the degree of association: and screening based on the image relevance. The method comprises the following specific steps: and calculating the association degree between the track to be registered and the reference map according to the number of effective image pairs formed between each image in the track to be registered and the existing image in the reference map and the total number of the corresponding interior point feature matching pairs, sequencing the association degrees, and selecting the image with the highest association degree. Although the method has higher calculation complexity, the selected seed image has better quality, is more favorable for the next image registration and is easier to obtain an accurate registration result

3. Combining spatial distribution and degree of association: and simultaneously, screening by combining spatial distribution and correlation. The method comprises the following specific steps: dividing a grid sampling area according to the spatial distribution of the camera frame positions; calculating the association degree between the track to be registered and the reference map according to the method 2 and sequencing; and sequentially selecting the image with the highest association degree in each sampling area through algorithms such as octree sampling and the like. The method can obtain the best seed image of each block under the condition of ensuring the uniform distribution of the seed space.

It should be noted that the screening methods provided in this embodiment are all exemplary methods, and are not specifically limited, and other methods may also be used to screen the seed image.

Further, registering the seed image into a reference map, and optimizing the pose of the seed image through a bundle adjustment optimization algorithm. Preferably, in this embodiment, when the pose of the seed image is optimized by using a bundle adjustment optimization algorithm, the following steps need to be performed in the optimization process:

s1, fixing the image pose in a reference map;

s2, fixing 3D points which are not observed in the seed image;

and S3, optimizing the position and posture of the map point observed in the seed image and the seed image.

Through the process, the scale of the optimized variable and factor graph is limited, so that the problem that the optimization time exponential type rises along with the increase of the scale of the whole map in the classical three-dimensional reconstruction can be avoided. The three-dimensional reconstruction speed can be effectively accelerated.

And S105, calculating the similarity transformation between the reference map and the SLAM track coordinate system through a random sampling consistency algorithm according to the pose of the seed image in the reference map and the SLAM pose, optimizing the similarity transformation to obtain the optimal similarity transformation, and registering the whole track to be registered into the reference map through the optimal similarity transformation.

In the embodiment, according to the pose of the seed image in the reference map and the SLAM pose, the similarity transformation between the reference map and the SLAM track coordinate system is calculated through a random sampling consistency algorithm and is used as a value to be optimized, the pose error after the similarity transformation is minimized by using nonlinear optimization and a robust kernel function, and the optimal similarity transformation is obtained through solving.

Preferably, in this embodiment, the similarity transformation of the trajectory image is calculated by setting different weight constraint terms for the original trajectory data with different confidence degrees acquired by different devices and generated by different algorithms, so as to obtain the optimal similarity transformation. For example, the error term for each image is defined as: (SLAM track location-semblance transformed map location) — data confidence of the SLAM track weight of SLAM track quality + (location provided by sensor 1-semblance transformed map location) — data confidence provided by the location sensor 1 + uncertainty weight of sensor 1 + (location provided by sensor 2-semblance transformed map location) — data confidence provided by the location sensor 2 — uncertainty weight of sensor 2. The position refers to a position where an image is captured. In addition, the "map position subjected to similarity transformation" in the sensor specifically refers to: the position provided by each sensor x is aligned to the coordinate system of the SLAM track in advance, and then the position of the sensor x in the coordinate system of the SLAM track is subjected to similarity transformation to obtain the data position in the coordinate system of the reference map.

It should be noted that, in this embodiment, the raw trajectory data of different confidences acquired by different devices includes: images shot by unmanned planes and mobile devices are attached with GPS information, and position information from Bluetooth, wi-Fi, UWB, wheel speed meters and the like are attached to some multi-sensor combination hardware.

The embodiment fully considers the inter-frame continuity and consistency information in the track, ensures the accuracy of transformation, and is superior to the independent image registration in the prior art. In addition, the implementation also supports the multi-track combined three-dimensional reconstruction of any equipment, and compared with the single-track three-dimensional reconstruction, the application range is wider.

Through the steps S101 to S105, the three-dimensional reconstruction is performed based on the multi-track SLAM information, so that the problems of slow reconstruction speed, low accuracy and precision of the reconstructed model and incapability of multi-track three-dimensional reconstruction in the related art are solved, the reconstruction speed and the accuracy and precision of the reconstructed model are improved, and the application range is expanded.

It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.

The present embodiment further provides a three-dimensional reconstruction system based on multi-track SLAM information, where the system is used to implement the foregoing embodiments and preferred embodiments, and details are not repeated for what has been described. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware or a combination of software and hardware is also possible and contemplated.

Fig. 2 is a block diagram of a three-dimensional reconstruction system based on multi-trajectory SLAM information according to an embodiment of the present application, and as shown in fig. 2, the system includes a trajectory image processing module 21, a reference map construction module 22, and a trajectory registration module 23:

the track image processing module 21 is configured to acquire a track image, perform feature extraction and image matching on the track image to obtain an effective image pair set, calculate epipolar geometry, and screen out an interior point feature matching pair set that meets the epipolar geometry relationship from each group of effective image pairs according to the relative position relationship between the two standard cameras; the reference map building module 22 is configured to calculate and evaluate track quality according to the effective image pairs in each track, select an optimal track as a reference track, and perform three-dimensional reconstruction according to the SLAM pose of the reference track to obtain a reference map; the track registration module 23 is configured to customize a track screening formula according to the association matching information between the track and the reference map, calculate and select a track with the highest score from all tracks as a track to be registered according to the track screening formula, screen out a seed image from the track to be registered, register the seed image in the reference map, optimize the pose of the seed image through a beam adjustment optimization algorithm, calculate a similarity transformation between the reference map and a SLAM track coordinate system through a random sampling consistency algorithm according to the pose of the seed image in the reference map and the SLAM pose, optimize the similarity transformation to obtain an optimal similarity transformation, and register the whole track to be registered in the reference map through the optimal similarity transformation.

Through the system, the three-dimensional reconstruction is carried out based on the multi-track SLAM information, the problems that the reconstruction speed is low, the accuracy and precision of the reconstructed model are not high, and the multi-track three-dimensional reconstruction cannot be carried out in the related technology are solved, the reconstruction speed and the accuracy and precision of the reconstructed model are improved, and the application range is expanded.

It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiment and optional implementation manners, and details of this embodiment are not described herein again.

Note that each of the modules may be a functional module or a program module, and may be implemented by software or hardware. For a module implemented by hardware, the above modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.

The present embodiment also provides an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

In addition, in combination with the multi-track SLAM information-based three-dimensional reconstruction method in the foregoing embodiments, the embodiments of the present application may provide a storage medium to implement. The storage medium has a computer program stored thereon; when being executed by a processor, the computer program realizes the three-dimensional reconstruction method based on multi-track SLAM information in any one of the above embodiments.

In one embodiment, a computer device is provided, which may be a terminal. The computer device comprises a processor, a memory, a network interface, a display screen and an input device which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a three-dimensional reconstruction method based on multi-track SLAM information. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

In one embodiment, fig. 3 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application, and as shown in fig. 3, there is provided an electronic device, which may be a server, and its internal structure diagram may be as shown in fig. 3. The electronic device comprises a processor, a network interface, an internal memory and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used for providing calculation and control capabilities, the network interface is used for being connected and communicated with an external terminal through a network, the internal memory is used for providing an environment for an operating system and the running of a computer program, the computer program is executed by the processor to realize a three-dimensional reconstruction method based on multi-track SLAM information, and the database is used for storing data.

Those skilled in the art will appreciate that the architecture shown in fig. 3 is a block diagram of only a portion of the architecture associated with the subject application, and does not constitute a limitation on the electronic devices to which the subject application may be applied, and that a particular electronic device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct Rambus Dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM), among others.

It should be understood by those skilled in the art that various technical features of the above-described embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above-described embodiments are not described, however, so long as there is no contradiction between the combinations of the technical features, they should be considered as being within the scope of the present description.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is specific and detailed, but not to be understood as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent application shall be subject to the appended claims.

Claims

1. A three-dimensional reconstruction method based on multi-track SLAM information is characterized by comprising the following steps:

customizing a track screening formula according to the association matching information between the track and the reference map, and calculating and selecting the track with the highest score from all the tracks as a track to be registered according to the track screening formula;

2. The method of claim 1, wherein the acquiring a track image comprises:

and respectively acquiring a plurality of SLAM tracks through various devices, and extracting the images and the poses of key frames in the SLAM tracks.

3. The method of claim 1, wherein image matching the images to obtain a set of valid image pairs comprises:

4. The method of claim 1, wherein the computing the estimated track quality from the valid image pairs within each track, and the selecting the optimal track as the reference track comprises:

5. The method of claim 4, wherein computationally evaluating the internal features through the pairs of trajectory-valid images and images comprises:

6. The method of claim 4, wherein the computational evaluation of the SLAM pose from the raw trajectory data comprises:

calculating a transformation matrix between the effective image pairs according to the SLAM pose of the effective image pairs, and decomposing to obtain a rotation component of the SLAM pose;

7. The method of claim 1, wherein reconstructing the three-dimensional pose of the reference trajectory from the SLAM pose comprises:

8. The method of claim 7, wherein merging map points comprises:

9. The method of claim 1, wherein the association match information between a track and the reference map comprises:

the number of effective image pairs formed between all images in the to-be-selected track and all images in the reference map and the total number of corresponding interior point feature matching pairs;

10. The method of claim 1, wherein screening the trajectory to be registered for a seed image comprises:

11. The method of claim 10, wherein screening based on the uniformity of the spatial distribution of the image comprises:

12. The method of claim 10, wherein filtering based on image relevance comprises:

13. The method of claim 10, wherein the screening in combination with spatial distribution and correlation comprises:

and sequentially selecting the image with the highest association degree in each sampling region by using an octree sampling algorithm.

14. The method of claim 1, wherein optimizing the pose of the seed image by a bundle adjustment optimization algorithm comprises:

fixing the image pose in the reference map;

fixing 3D points not observed in the seed image;

15. The method of claim 1, wherein optimizing the similarity variation to obtain an optimal similarity transformation comprises:

16. The method of claim 1, wherein optimizing the similarity variation to obtain an optimal similarity transformation further comprises:

and calculating the similarity transformation of the track image by setting different weight constraint terms for the original track data with different confidence degrees, which are acquired by different equipment and generated by different algorithms, so as to obtain the optimal similarity transformation.

17. A three-dimensional reconstruction system based on multi-track SLAM information, the system comprising:

the track registration module is used for self-defining a track screening formula according to the correlation matching information between the track and the reference map, calculating and selecting the track with the highest score from all the tracks as the track to be registered according to the track screening formula,

18. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 16.

19. A storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any one of claims 1 to 16 when executed.