CN112507778B - Loop detection method of improved bag-of-words model based on line characteristics - Google Patents

Abstract

The invention relates to a loop detection method of an improved bag-of-words model based on line characteristics, which comprises the following steps: LSD (Line Segment Detector) features are extracted from the offline image dataset and corresponding LBD descriptors are computed, which are used as the raw data of the clustering generation dictionary. And constructing an LSD characteristic word bag model by using an improved word bag model construction method, and constructing a visual dictionary tree with self-adaptive branches. And converting the bag-of-words model vector. And optimizing visual word weight. And (3) similarity calculation: and calculating the similarity by adopting an L1 norm according to the visual bag-of-word vector between the current frame and the historical key frame, and obtaining the appearance similarity score between the images. And acquiring loop candidate frames, grouping the loop candidate frames, and removing isolated loop candidate frames with similar appearances. The continuity verification can only consider that the loop is a reliable loop candidate if the loop is continuously detected, and the loop candidate is reserved. And (5) verifying geometric consistency.

Description

A Loop Closure Detection Method Based on Line Feature Based on Improved Bag of Words Model

technical field

The present invention relates to the field of visual SLAM (Simultaneous Localization And Mapping: simultaneous localization and mapping), in particular to a visual SLAM loop closure detection method based on an improved word bag model based on line features.

Background technique

Loop closure detection is an indispensable part of visual SLAM, which can eliminate the cumulative error generated by the visual odometry part, thereby constructing a globally consistent map. The loop closure detection algorithm based on the bag-of-words model is the current main method, which judges whether there is a loop by constructing a bag-of-words model to compare the similarity between images. The bag-of-words model originated from text analysis and determined the similarity of text by comparing the frequency of each word in the text. Correspondingly, the bag-of-visual-words model also measures the similarity of two images by comparing the frequency of "visual words" in the images.

In 2008, Cummins et al. (Cummins M, Newman P. FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance [M]. Sage Publications, Inc. 2008.) proposed a feature based on SURF (Speeded Up Robust Features) and Chou- The bag-of-words model of Liu tree, and the camera position recognition based on image appearance is well realized through the bag-of-words model. However, the vector of the bag of words model is a binary vector, that is, it only considers whether the visual word appears in the image, and does not consider the different frequencies of different words.

In 2011, Galvez-Lopez et al. (Galvez-Lopez D, Tardos J D. Real-time loop detection with bags of binary words[C]. International Conference on Intelligent Robots and Systems, 2011: 25-30.) adopted FAST (Features from Accelerated Segment Test) key points and Brief (Binary Robust Independent Elementary Features) binary descriptors implement the extraction and description of point features, and introduce the data structure of k-d tree for dictionary construction. Using the method of hierarchical K-means clustering, a visual bag-of-words model of binary descriptors based on point features is constructed. The dictionary structure of the k-d tree also leads to the fact that the K-means clustering used in the dictionary construction process adopts the same parameter k, but not any data is clustered with the same k value, and the obtained clustering results are all it's the best.

Subsequently, in 2015, Mur-Artal et al. (Mur-Artal R, Montiel J M M, Tardos J D. ORB-SLAM: a versatile and accurate monocular SLAM system [J]. IEEE Transactions on Robotics, 2015, 31(5): 1147- 1163) proposed ORB-SLAM, a visual bag of words model based on ORB (Oriented FAST and Rotated BRIEF) point features is constructed. The ORB point feature solves the rotation invariance and scale invariance of FAST keypoints, and achieves good results in experiments. However, the visual dictionary still adopts the dictionary structure of K-means clustering and k-d tree, and the construction process of the bag-of-words model has not been improved.

The detection effect of the above-mentioned bag-of-words model based on point features depends on the number of point features extracted from the environment. When enough point features cannot be extracted from the environment, and the point features are easy to appear together, the bag-of-words vector of the video frame cannot be calculated. and the appearance similarity between video frames.

In structured low-texture environments, although sufficient point features are often not extracted, there are abundant line features in such scenes that can be exploited.

Lee et al. (Lee J H, Zhang G, Lim J, et al. Place recognition using straightlines for vision-based SLAM [C]., 2013 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2013, pp.3799-3806) A bag-of-words model based on MSLD (Mean-Standard deviation Line Descriptor) line feature descriptor is proposed, and good results are obtained in the experiment. However, MSLD line feature descriptors do not have scale invariance, and the computational complexity is high, which is not conducive to real-time operation.

Lin Limeng et al. (Lin Limeng, Wang Mei. Improved binocular vision SLAM algorithm for point and line features [J]. Computer Measurement and Control, 2019(9): 156-162) Construct point and line features in a visual word bag In the model, the point and line features of the image are extracted, the bag of words model is used to convert the two features into a bag of words vector, and the image similarity is calculated by the bag of words vector.

Patent 201811250049.7 (a tightly coupled binocular visual-inertial SLAM method for point-line feature fusion) builds a point-feature word bag model and a line-feature word-bag model respectively, and calculates the point feature similarity score and line feature similarity between the two frames. and take its weighted sum as the similarity score of the final two frames. Both methods use line features to build a bag of words model, but the K-means clustering and k-d tree dictionary structure are still used in the construction of the visual bag of words model. The essential difference is also unable to obtain better visual word clustering results. Moreover, the word weight calculation method in the bag of words model adopts the TF-IDF (Term Frequency-Inverse Document Frequency) method, which considers the frequency of visual words in the current image and its importance in the training data set, but Its importance on the loop closure detection query dataset is not considered.

To sum up, the line feature is a local feature that can replace the point feature in a structured environment, and a real-time visual SLAM loop closure detection algorithm based on the line feature bag-of-words model is constructed, which can effectively solve the problem in structured low-texture. In the environment, the loop closure detection algorithm based on point feature cannot effectively detect the loop closure problem. In this paper, a visual SLAM loop closure detection algorithm based on the improved bag of words model based on line features is proposed, and the construction process of the bag of words model and the weight of visual words are improved.

SUMMARY OF THE INVENTION

Aiming at the problem that it is difficult to extract enough point features to realize visual SLAM loop detection in a structured low-texture environment, the invention proposes a loop detection method based on an improved bag of words model based on line features. The algorithm can realize vision-based loop closure detection in a structured environment by using rich line features as visual local features. And by improving the construction method of the bag of words model and the visual word weight calculation method, the accuracy and recall rate of loop closure detection are improved. The technical solution is as follows:

A method for loop closure detection based on a line feature-based improved bag-of-words model, comprising the following steps:

Step 1: Extract the LSD (Line Segment Detector) feature from the offline image dataset and calculate the corresponding LBD (Line Band Descriptor) descriptor, and use the descriptor as the original data for clustering to generate a dictionary.

Step 2: Use the improved bag of words model construction method to build the LSD feature bag of words model: Before constructing each clustering of the dictionary tree, first determine the optimal clustering k value k' for the current data, and then cluster the current data. for class k'. This cycle is repeated until a visual dictionary tree with adaptive branches is finally constructed.

Step 3: Vector transformation of bag-of-words model: LSD-LBD line features are extracted from the image. According to the constructed bag-of-words model based on LBD descriptors, as well as the Hamming distance between the line feature descriptors and visual words, each line in the image A line feature is quantized to the corresponding visual word, thereby transforming the entire image into a corresponding numerical vector.

根据视觉单词在历史关键帧数据集上的分布情况，对词袋模型向量中的视觉单词权重进行优化，计算视觉单词的权重优化参数，并结合TF-IDF法计算出的单词权重，得到权重优化后的视觉词袋向量。Step 4: Visual word weight optimization: Introduce a weight optimization parameter in loop closure detection

According to the distribution of visual words on the historical key frame data set, optimize the visual word weight in the bag of words model vector, calculate the weight optimization parameters of the visual word, and combine the word weight calculated by the TF-IDF method to obtain the weight optimization. Post visual bag of words vector.

Step 5: Similarity calculation: According to the visual word bag vector between the current frame and the historical key frame, the L1 norm is used to calculate the similarity, and the appearance similarity score between the images is obtained.

Step 6: Obtain and group loopback candidate frames: Set the historical key frames that meet the similarity threshold requirements as loopback candidate frames, group the loopback candidate frames, and group the loopback candidate frames with similar timing into a group, and then according to the whole group The similarity score of , and given a threshold, those isolated loop closure candidate frames with similar appearance are eliminated.

Step 7: Continuity Verification: At this stage, it is checked whether the loopback can be detected continuously for a period of time in these loopback candidate frames. Only when loopbacks are continuously detected can it be considered as a reliable loopback candidate, and the loopback candidate is retained.

Step 8: Geometric consistency verification: In order to ensure the accuracy of the loopback, the visual word distribution of the current frame and the loopback candidate frame is verified. Only when the line feature distributions corresponding to these visual words are the same, can the two frames be considered to constitute a loopback .

At present, visual SLAM still mainly uses point features as visual features. Compared with point feature loop closure detection, the present invention adopts more abundant line features in a structured environment as local visual features for loop closure detection. The key points of the present invention are: 1) First, a visual dictionary tree with adaptive branch numbers is constructed by using line features, which improves the discrimination of visual words and reduces the quantization error of converting local features into visual words. 2) Then, according to the distribution of visual words in the loopback detection query data set, the weight optimization parameters of each word are calculated, and the visual word bag vector is optimized through the parameters, so that the similarity calculation result of the word bag vector has a higher degree of discrimination. Compared with the unoptimized visual word bag model, the present invention can obtain a higher recall rate at 100% accuracy, which shows that the present invention can detect loop closures more accurately and effectively.

Description of drawings

Figure 1 is the flow chart of the improved bag-of-words model construction

Figure 2 shows the result of LSD line feature extraction in structured environment with low texture

Figure 3 shows the result of ORB point feature extraction in the low texture of the structured environment

FIG. 4 is a schematic diagram of constructing a visual dictionary from LBD descriptors according to an example of the present invention.

Detailed ways

The present invention will be further described below in conjunction with specific examples. It should be noted that the described embodiments are only intended to facilitate the understanding of the present invention without any limitation thereto.

Step 1: For a large amount of image data obtained offline, use the LSD algorithm to extract the line segment features, and use the LBD algorithm to calculate the descriptors of the line features. The LSD algorithm can quickly realize the detection of line features, and the LBD descriptor can generate binary descriptors and can achieve fast matching. This method of line feature extraction and description can meet the real-time requirements of loop closure detection.

Step 2: Build a line-featured visual word bag model with adaptive branch number. The traditional bag of words construction method adopts the K-means clustering algorithm and the data structure of the k-d tree, so it retains the disadvantage of manually specifying the k value of the K-means algorithm. Before constructing each clustering of the dictionary tree, the present invention adds a link of determining the optimal clustering k value of the current data, and determines the optimal k value of the clustering by introducing the contour coefficient of the clustering evaluation index. First, calculate the silhouette coefficient of the current data under different k values (k is 5-15), and select the most reasonable k value under the current data according to the silhouette coefficient, that is, the corresponding k value k' when the silhouette coefficient is the largest, and then Take k' as the cluster k value of the current node of the final constructed visual dictionary tree. This cycle repeats until the fifth layer of the bag of words model is constructed.

The contour coefficient of the clustering evaluation index combines two factors, the degree of aggregation within a class (a) and the degree of separation between classes b (i). Assuming that the current data has been clustered into k categories, the current silhouette coefficient S can be obtained as follows:

First, calculate the silhouette coefficient of each element after clustering

s(i)=(b(i)-a(i))/(max{a(i), b(i)}) (1)

The intra-class aggregation degree a(i) represents the average distance between the current element m _i and other elements m _j of its category, and the inter-class separation degree b(i) represents the minimum value of the average distance between the current element _mi and other clusters. It can be seen that s(i)∈[-1,1], if s(i) is close to 1, it means that the sample elements m _i are reasonably clustered; if s(i) is close to -1, it means that the sample elements m _i should be classified into In other clusters; s(i) is close to 0, indicating that the sample element m _i is located on the boundary of the two clusters.

After calculating the silhouette coefficient of each element, use the average of the silhouette coefficients of all elements as the silhouette coefficient of the current clustering result, that is, S=1/k*∑ _0<i≤k s(i)

Finally, according to the silhouette coefficient S calculated under different k values, the k value corresponding to the maximum silhouette coefficient is selected as the final number of cluster branches at the current node in the visual dictionary tree. By analogy, the optimal number of clustering branches is calculated for all intermediate nodes in the process of building a visual dictionary, so as to obtain a relatively good clustering effect and make visual words more distinguishable.

Step 3: bag-of-words model vector transformation. According to the constructed bag-of-words model based on LBD descriptors and the Hamming distance between line feature descriptors and visual words, each line feature in the image is quantified into a corresponding visual word, thereby converting the entire image into a corresponding visual word. numeric vector, such as

Among them, _wi represents the ith visual word, and η _i is its corresponding weight. The weight calculation method of TF-IDF is adopted, that is, η _i =TF _i *IDF _i . In practice, each image contains only a few visual words in the visual dictionary, so most of the numerical vectors η _i = 0, ie v _a is a sparse vector.

Step 4: Visual word weight optimization. The TF-IDF weight calculation method considers the frequency of the visual word in the current image and its importance on the training dataset, but does not consider its importance on the historical keyframe dataset for loop closure detection. The TF-IDF method believes that the smaller the text frequency of a word (that is, the number of texts containing a certain word), the greater its ability to distinguish different types of texts. In the same way, it can be considered that the smaller the text frequency of a visual word in the historical key frame dataset, the greater its ability to distinguish different images in the historical key frame dataset.

统计历史关键帧数据集中，每一个视觉单词在关键帧中出现的数量I_i。使参数

随着关键帧数量I_i的增大而减小，从而降低在此过程中重复出现的视觉单词的权重。具体如下：Therefore, a repetition factor is introduced in the weight calculation

Count the number I _i of each visual word appearing in the key frame in the historical key frame data set. make parameter

It decreases as the number of keyframes I _i increases, thereby reducing the weight of repeated visual words in the process. details as follows:

并根据对应的关键帧数量计算视觉单词的重复因子

其中n为历史关键帧数据集中关键帧的数量，

为其中出现视觉单词w_i的关键帧数量。1) In loop closure detection, while establishing the mutual index between visual words and key frames, count the number of key frames in which each visual word appears

And calculate the repetition factor of the visual word according to the corresponding number of keyframes

where n is the number of keyframes in the historical keyframe dataset,

is the number of keyframes in which the visual word _wi appears.

和TF-IDF，生成视觉单词w_i新的权重

并据此生成新的词袋模型向量v′_a。2) Binding repeat factors

and TF-IDF, generate new weights for visual words _wi

And generate a new bag-of-words model vector v' _a accordingly.

v' _a = {(W ₁ , η' ₁ ), (w ₂ , η' ₂ ), ..., (w _N , η' _N )} (3)

为单词i的权重优化参数，TF_i为单词i在当前图像中的词频，IDF_i为单词i在训练数据集上的逆文档频率。where η′ _i is the optimized weight of visual word i,

The parameters are optimized for the weight of word i, TF _i is the word frequency of word i in the current image, and IDF _i is the inverse document frequency of word i on the training dataset.

Step 5: Image similarity calculation. Image similarity is calculated using the new bag-of-words model vector calculated from the current image and historical keyframes. For the bag-of-words model vector of any two images, the L1 norm is used to evaluate the similarity of the images, as follows:

The similarity calculation result is between 0 and 1. When the two images are completely unrelated, the similarity score is 0; when the two images are completely identical, the similarity score is 1.

式中v_k为该组中第k个关键帧对应的词袋模型向量，v_c为当前关键帧对应的词袋模型向量。Step 6: Obtain and group loopback candidate frames. Among the historical key frames, if there is a key frame whose similarity with the current key frame satisfies a certain threshold α, it can be set as a loopback candidate frame. After all the loopback candidate frames are obtained, the loopback candidate frames are grouped, and the loopback candidate frames with similar timings are grouped into a group, and the group similarity score is calculated. For each candidate group, use I ₁ , I ₂ , I ₃ , .., In to _denote the key frame in it, and s ₁ , s ₂ , s ₃ , . . . , _sn denote its similarity to the current key frame degree, the group similarity score can be expressed by the sum of these similarities, that is,

where v _k is the bag of words model vector corresponding to the kth key frame in the group, and v _c is the bag of words model vector corresponding to the current key frame.

By grouping the loop closure candidate frames, the corresponding group similarity score is obtained, and according to the given group score threshold β, the loop closure candidate key frames with lower group scores are eliminated. Because of the correct loopback key frame, the key frame with similar timing and the current key frame will also have a high similarity, and also belong to the loopback candidate frame, so that some incorrect loopback candidate frames can be excluded.

Step 7: Continuity check. At this stage, we believe that only when a loopback is detected in several consecutive frames at the same time, the loopback is considered reliable, and the loopback candidate is reserved.

Step 8: Geometric Consistency Verification. Due to the spatial information of visual features ignored by the visual bag of words model, in the final stage, the geometric consistency verification between the loop closure candidate frame and the current key frame is required to ensure the accuracy of loop closure detection.

The line feature reprojection error is calculated by calculating the matching line features between the current frame and the loop closure candidate frame, and then the pose transformation between the current frame and the loop closure candidate frame is obtained through local BA optimization. And by calculating the inlier number of the line features under the pose transformation, it is judged whether the pose transformation is reasonable, so as to determine whether the loop closure candidate frame has passed the geometric consistency verification.

After the similarity of the image appearance reaches the threshold α, it is required to extract the loopback candidate frames, and after passing a series of verification steps to ensure the loopback accuracy, it can be determined that a loopback has occurred, and the global map can be corrected and adjusted according to the detected loopback. renew.

Claims (1)

1. A loop detection method of an improved bag-of-words model based on line features comprises the following steps:

step 1: extracting LSD (Line Segment Descriptor) features through an offline image data set, calculating corresponding LBD (Line Band Descriptor) Line feature descriptors, and taking the LBD Line feature descriptors as original data of a clustering generation dictionary;

and 2, step: constructing a bag-of-words model based on LBD descriptors: before each clustering of the dictionary tree is constructed, determining an optimal clustering k value k 'for current data, and then clustering the current data into a k' class; the steps are circulated until a visual dictionary tree with self-adaptive branches is finally constructed;

and step 3: bag of words model vector transformation: extracting LSD line features from the image, quantizing each line feature in the image into a corresponding visual word according to the constructed bag-of-words model based on the LBD descriptor, the line feature descriptor and the Hamming distance between the visual words, and converting the whole image into a corresponding numerical vector;

and 4, step 4: visual word weight optimization: in the loop detection, the mutual index between the visual words and the key frames is established, and the number of the key frames with each visual word is counted

And calculating the repetition factor of the visual word according to the number of the corresponding key frames

Where n is the number of keyframes in the historical keyframe data set,

the number of key frames in which the visual word wi appears;

optimizing the visual word weight in the bag-of-words model vector according to the distribution condition of the visual words on the historical key frame data set, calculating the weight optimization parameters of the visual words, and combining the word weight calculated by the TF-IDF method to obtain the weight-optimized visual bag-of-words vector;

repetition factor in conjunction with visual words

And TF-IDF algorithm to generate visual word w _i New weights

And generates therefrom a new bag-of-words model vector v' _a ：

v′ _a ＝{(w ₁ ，η′ ₁ )，(w ₂ ，η′ ₂ )，...，(w _N ，η′ _N )}

TF _i As visual words w _i Word frequency, IDF, in the current picture _i As visual words w _i Inverse document frequency on the training data set;

and 5: and (3) similarity calculation: calculating similarity by adopting an L1 norm according to visual bag-of-word vectors between the current frame and the historical key frame, and obtaining an appearance similarity score between the images;

step 6: loop candidate frames are acquired and grouped: setting the historical key frames meeting the requirement of the similarity threshold as loop candidate frames, grouping the loop candidate frames, dividing the loop candidate frames with similar time sequence into a group, and then rejecting the loop candidate frames with low group scores according to the similarity scores of the whole group and the given threshold;

and 7: and (3) verifying the continuity: at this stage, when a loop is continuously detected in the loop candidate frames, the corresponding loop candidate frames are retained;

and step 8: and (3) verifying geometric consistency: and performing geometric consistency verification on the loop candidate frame and the current key frame to ensure the accuracy of loop detection.

Images