CN110852327A - Image processing method, device, electronic device and storage medium - Google Patents

Abstract

The embodiment of the disclosure discloses an image processing method, an image processing device, an electronic device and a storage medium, wherein the method comprises the following steps: filtering dynamic objects in the images to be processed in the image set to be processed to obtain static images; extracting local features in the static image by using a CNN network, and clustering the local features to obtain a plurality of clustering central points; and constructing a visual bag-of-words model of the static image according to the plurality of clustering center points, mapping local features of the static image to the constructed visual bag-of-words model, expressing feature vectors of the image set to be processed according to the visual bag-of-words model, and performing closed-loop detection according to the result expressed by the feature vectors.

Description

Image processing method, device, electronic device and storage medium

technical field

The present disclosure relates to the field of image technology, and in particular, to an image processing method, an apparatus, an electronic device, and a storage medium.

Background technique

In recent years, with the rapid development of robotics and autonomous driving technologies, visual Simultaneous Localization and Mapping (vSLAM) plays an increasingly important role in them. Among them, closed-loop detection, as an important link in the vSLAM system, is mainly used to determine whether the current position of the robot is in the space area that has passed before. The resulting map creation failure and position loss, as well as map redundant data or duplicate structure problems, improve the accuracy and robustness of vSLAM.

The existing closed-loop detection methods can be divided into two categories: traditional closed-loop detection methods based on artificially designed features and closed-loop detection methods based on deep learning. Traditional closed-loop detection algorithms rely on artificially designed feature descriptors for image similarity matching, which can be divided into closed-loop detection methods based on local feature descriptors (SIFT, SURF, and ORB) and closed-loop detection based on global feature descriptors (GIST). method. The most commonly used method for closed-loop detection based on local feature descriptors is the Bag of Visual Words (BoVW) model. According to the visual dictionary tree, the image is represented as a K-dimensional numerical vector, and then it is judged whether there is a closed loop by measuring the distance between the image feature vectors. Because the traditional artificially designed feature closed-loop detection algorithm extracts regional features such as edges, corners, lines, curves and special attributes from the local area of the image, it has good accuracy when the surrounding environment does not change significantly. However, since most of the artificially designed feature algorithms rely on human expertise and experience, they cannot accurately express images. Therefore, when the surrounding environment changes significantly (such as in scenes with obvious changes in illumination), it is difficult to provide accurate images. Moreover, the robust image feature description has poor stability, which leads to an increase in the mismatch rate in complex environments.

With the rapid development of deep learning in recent years, because CNN (Convolutional Neural Network) can extract the deep-level features of the image, it has richer data information and higher feature expression ability, so CNN is used to extract from the image. Learning features from a large amount of data replace the traditional artificially designed features to solve the closed-loop detection problem. CNN-based image descriptors perform better than artificially designed descriptors when the illumination changes significantly, but since the CNN-based closed-loop detection algorithm extracts global image features, in a stable environment without illumination changes, The accuracy of CNN feature extraction closed-loop detection algorithm based on global features is lower than that of traditional artificially designed features based on local features.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide an image processing method, an apparatus, an electronic device, and a computer-readable storage medium.

In a first aspect, the embodiments of the present disclosure provide an image processing method, including:

Filter the dynamic objects in the to-be-processed images in the to-be-processed image set to obtain a static image;

Use CNN network to extract local features in the static image, and cluster the local features to obtain a plurality of cluster center points;

The visual word bag model of the static image is constructed according to the plurality of cluster center points, the local features of the static image are mapped to the constructed visual word bag model, and the visual word bag model is constructed according to the visual word bag model. The to-be-processed image set is represented by a feature vector, and closed-loop detection is performed according to the result represented by the feature vector.

Further, the dynamic objects in the to-be-processed images in the to-be-processed image set are filtered to obtain a static image, including:

Detecting the dynamic object in the to-be-processed image, and extracting the area information of the dynamic object;

The static image is obtained by filtering the dynamic object from the to-be-processed image according to the area information.

Further, using the CNN network to extract the local features in the static image, including:

Process the static image by using a multi-scale dense fully convolutional network, and return key point information in the static image;

locally cropping the static image according to the key point information to obtain a plurality of local image blocks;

A local feature network is used to perform feature extraction on a plurality of the local image blocks, respectively, to obtain the feature point positions and feature descriptors of the local graphics.

Further, mapping the local features of the static image to the constructed visual word bag model, including:

determining a first similarity between the local feature of the static image and the cluster center point in the bag of visual words model;

The local feature of the static image is mapped to a bag-of-words category to which one of the cluster center points belongs according to the first similarity.

Further, performing feature vector representation on the to-be-processed image set according to the visual word bag model, including:

Statistical mapping of the local features of the static image to the distribution of the bag-of-words categories in the visual bag-of-words model;

The feature vector of the static image is obtained according to the distribution.

Further, performing closed-loop detection according to the result represented by the feature vector, including:

determining the second similarity between the feature vectors of the static images corresponding to the two to-be-processed images in the to-be-processed image set;

A closed-loop detection result of the two images to be detected is determined according to the second similarity.

In a second aspect, an embodiment of the present invention provides an image processing apparatus, including:

a filtering module, configured to filter dynamic objects in the to-be-processed images in the to-be-processed image set to obtain a static image;

a feature extraction module, configured to extract local features in the static image by using a CNN network, and cluster the local features to obtain a plurality of cluster center points;

The closed-loop detection module is configured to construct a visual word bag model of the static image according to the plurality of cluster center points, map the local features of the static image to the constructed visual word bag model, and according to The bag-of-visual-words model performs feature vector representation on the to-be-processed image set, and performs closed-loop detection according to the result represented by the feature vector.

The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

In a possible design, the structure of the image processing apparatus includes a memory and a processor, where the memory is used to store one or more computer instructions that support the image processing apparatus to perform the method in the first aspect, and the processor is configured to execute computer instructions stored in the memory. The image processing apparatus may further include a communication interface for the image processing apparatus to communicate with other devices or a communication network.

In a third aspect, embodiments of the present disclosure provide an electronic device, including a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are processed by the The device executes to implement the method described in the first aspect.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium for storing computer instructions used by a security authentication device for an enterprise account, including computer instructions for executing the method described in the first aspect.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

In the embodiment of the present disclosure, the convolutional neural network is used to segment the local area of the image and feature extraction, and the extracted local CNN feature descriptor of the image is combined with the traditional closed-loop detection word bag model algorithm, and the obtained image is obtained in the word bag model. The local CNN feature descriptor uses the clustering center algorithm to cluster, and builds a visual dictionary tree according to the clustered multiple center points, and then obtains the feature vector representing the image, and compares the similarity of the image to determine whether a loop closure occurs. This method can simultaneously improve the accuracy and stability of closed-loop detection in complex environments.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

Other features, objects and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the attached image:

FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure;

Fig. 2 shows a flowchart of step S101 according to the embodiment shown in Fig. 1;

FIG. 3 shows a flowchart of step S102 according to the embodiment shown in FIG. 1;

FIG. 4 shows a schematic flowchart of a method for combining the advantages of deep learning with the advantages of traditional closed-loop detection in an embodiment of the present disclosure;

FIG. 5 shows a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure.

Detailed ways

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Also, for the sake of clarity, parts unrelated to describing the exemplary embodiments are omitted from the drawings.

In the present disclosure, it should be understood that terms such as "comprising" or "having" are intended to indicate the presence of features, numbers, steps, acts, components, parts, or combinations thereof disclosed in this specification, and are not intended to exclude a or multiple other features, numbers, steps, acts, components, parts, or combinations thereof may exist or be added.

In addition, it should be noted that the embodiments of the present disclosure and the features of the embodiments may be combined with each other under the condition of no conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure. As shown in Figure 1, the image processing method includes the following steps:

In step S101, filtering the dynamic objects in the to-be-processed images in the to-be-processed image set to obtain a static image;

In step S102, the CNN network is used to extract the local features in the static image, and the local features are clustered to obtain a plurality of cluster center points;

In step S103, a visual word bag model of the static image is constructed according to the plurality of cluster center points, the local features of the static image are mapped to the constructed visual word bag model, and according to the The visual bag of words model performs feature vector representation on the to-be-processed image set, and performs closed-loop detection according to the result represented by the feature vector.

In this embodiment, the set of images to be processed may be a set composed of multiple images to be processed that are to be subjected to closed-loop detection. Dynamic objects are objects that are dynamically changing in the image to be processed, such as people, animals, vehicles, and the like.

In the embodiment of the present disclosure, before the closed-loop detection of the to-be-processed images in the to-be-processed image set is performed, the dynamic objects in the to-be-processed images are filtered to obtain a static image. In some embodiments, the pixel in the image area where the dynamic object in the image to be processed is located may be set to a preset color, such as black, because the completely black area will not have feature points, which excludes the dynamic object feature point interference.

For multiple static images obtained from the set of images to be processed, a CNN network can be used to extract local features in the static images. In the process of using the CNN network to extract local features from the static image, the first part of the CNN network is used to extract the key point information from the static image, and based on the key points, the static image is cropped into multiple local area image blocks, each local area image block. Include a keypoint extracted from a static image. After that, for each local area image block, the second part of the CNN network is used to extract the key point information from each local area image block, and the key point information extracted from the multiple local area image blocks corresponding to the static image is used as the static image. Local features of the image, such as feature vectors of key points extracted from multiple local area image blocks corresponding to the static image, can be used as the local features of the static image.

After the local features of the static images corresponding to each to-be-processed image in the to-be-processed image set are obtained, the central clustering algorithm can be used to cluster these local features to obtain a plurality of cluster center points. The central clustering algorithm may use the Kmeans++ algorithm, which is a known algorithm, and the specific clustering process will not be repeated here.

After the clustering is completed, a visual word bag model of the static images corresponding to all the to-be-processed images in the to-be-processed image set can be constructed according to the plurality of cluster center points. The constructed visual word bag model includes multiple word bag categories, each word bag category represents a class, and the center point of each word bag category is one of the cluster center points. When a local feature is relatively similar to the cluster center point, the local feature can be classified into the bag of words category where the cluster center point is located.

After the visual word bag model is constructed, the local features of the static image can be mapped to the visual word bag model according to the similarity with each cluster center point. That is, the local features of static images are divided into various word bag categories included in the visual word bag model. After the mapping is completed, the static image is represented by a feature vector according to the mapping result, and the feature vector representation result of the static image can be used as the feature vector representation result of the image to be processed corresponding to the static image. In some embodiments, the feature vector representation of the static image can be obtained based on the distribution results of these local adjustments on the respective bag-of-words categories after the local features of the static image are mapped to the visual bag-of-words model.

After the feature vector representation of each to-be-processed image in the to-be-processed image set is obtained, the closed-loop detection result between the two to-be-processed images can be determined according to the feature vector representation.

In an optional implementation manner of this embodiment, as shown in FIG. 2 , the step S101, that is, the step of filtering the dynamic objects in the to-be-processed images in the to-be-processed image set to obtain a static image, further includes the following steps:

In step S201, a dynamic object in the to-be-processed image is detected, and region information of the dynamic object is extracted;

In step S202, the dynamic object is filtered from the to-be-processed image according to the area information to obtain the static image.

In the image preprocessing stage, the image is scene recognized and the dynamic and static scenes are separated. Before the scene separation, the YOLOv3 model is pre-trained using the PASCAL VOC2007 and VOC2012 datasets. The dataset contains most outdoor moving objects, such as people, animals, vehicles, etc., as well as the location and classification of these objects in the picture. Among them, PASCALVOC 2007 contains 20 categories, 9963 images and 24,640 objects annotated therein, and PASCAL VOC 2012 contains 20 categories, 11,530 images and 27,450 objects annotated in them.

The object classification in the PASCAL data set completely covers the moving objects that the autonomous mobile robot may encounter when working outdoors, and most of the outdoor moving objects in the data set used in the embodiments of the present disclosure are people, vehicles, etc. The convolutional neural network can easily identify these dynamic objects, and then filter out dynamic objects from the image to be processed.

After the training of the convolutional neural network for recognizing dynamic objects in the image is completed, the dynamic and static scene separation algorithm based on object detection can be used to recognize and separate the dynamic and static scenes for the to-be-processed images in the to-be-processed image set.

In the embodiment of the present disclosure, the pre-trained target detection YOLOv3 model is used to perform dynamic object detection on the image to be processed, and the dynamic object area information in the to-be-processed image is extracted, and the area information includes the rectangular area where the dynamic object is located and the area size, etc. . The dynamic object area information is separated from the corresponding position in the original image, and the dynamic object information such as pedestrians and vehicles in the original image is excluded.

In some embodiments, the adopted scene information separation method is to process based on the matrix corresponding to the image pixel points, and set the dynamic object area to be black in the original image, because the completely black area will not have feature points, so the dynamic object is excluded. feature point interference. Therefore, when the local feature extraction is performed on the image subsequently, the feature extraction will not be performed for the separated dynamic scene area. The other location areas obtained are the filtered pure static scene images that exclude dynamic objects that we need to solve.

The embodiment of the present disclosure adopts the dynamic and static scene separation algorithm based on image semantic segmentation in the image preprocessing stage, and performs scene semantic segmentation and separation on the image. The model is pre-trained.

After that, the pre-trained DeepLabv3 Plus model is used to semantically segment the input image to be processed, and the semantic information of the dynamic and static scenes of the image is obtained in the result. According to the obtained results, the semantic information of the dynamic object in the image to be processed (including the information of the image pixels in the area where the dynamic object is located, etc.) is extracted, and the overall semantic information of the image is traversed based on the image pixel matrix. When the pixel matrix is training When labeling the object classification information, the semantic information of the dynamic object is excluded from the original image, that is, the semantic information of dynamic objects such as pedestrians and vehicles is directly blacked out from the corresponding position in the original image, and only the information of the dynamic object itself is removed. static image.

In some embodiments, in order to prevent the dynamic object edge information from not being completely separated, and to ensure that the dynamic object semantic information is removed more cleanly and completely, the separated dynamic object region information can be further processed by image expansion (that is, the image region where the dynamic object is located). After blackening, the black area where the dynamic object is located is expanded), and the number of iterations is 2.

In an optional implementation manner of this embodiment, as shown in FIG. 3 , the step S102, that is, the step of extracting local features in the static image by using a CNN network, further includes the following steps:

In step S301, the static image is processed by a multi-scale dense full convolution network, and the key point information in the static image is returned;

In step S302, locally cropping the static image according to the key point information to obtain a plurality of local image blocks;

In step S303, a local feature network is used to perform feature extraction on a plurality of the local image blocks, respectively, to obtain feature point positions and feature descriptors of the local image blocks.

In this optional implementation, the CNN network can use a deep architecture sparse matching method LF-Net network. The CNN network can be divided into two parts. The first part of the CNN network can be a multi-scale dense fully convolutional network, which returns The position, scale and direction of the key points of the static image are cropped according to the position of the key points, and a plurality of local area image blocks corresponding to each static image are obtained. The second part of the CNN network may be a local descriptor network used to extract the keypoint-based cropped image patches produced by the first part of the network. When extracting local features from static images, a fully convolutional network is first used to generate rich feature maps o from the image I to be processed. The generated feature map o is subjected to scale-invariant key point detection, and the top M pixels are selected as feature points according to the scale-invariant map, and finally the feature point position and feature descriptor of the local image block are obtained.

Among them, the multi-scale dense fully convolutional network is a fully convolutional network structure designed based on the MSDNet (multi-scale dense network) idea. The network has no fully connected layer and adopts three simple ResNet layouts, each block contains 5*5 Convolution filters followed by batch normalization, leaky-ReLU activation and another set of 5*5 convolutions. And adopt MSDNet (multi-scale dense network) idea: multiple scales are used to obtain abstract features of graphics to generate feature maps, which are divided into two parts in series: 1. The convolution of the features of the same scale in the upper layer and the scale of the upper layer Downsampling of feature maps (diagonal connection). 2. Each layer has connections to other convolutional layers, and during backpropagation, the weights are updated in a direction that is better for each feature extraction result.

After the key points are selected, the normalized image (the image obtained by normalizing the static image) is cropped based on the position around the key point using the bilinear sampling method (to preserve the difference). The bilinear sampling method, also known as the bilinear interpolation algorithm, is a relatively good image scaling algorithm. It makes full use of the four real pixel values around the virtual point in the image to jointly determine a pixel value in the target image. So as to achieve the purpose of image cropping.

After using the first part of the network to obtain multiple local area image blocks of the static image, you can use the local area image blocks cropped by the second part of the network to extract features (including the key point position, size and feature vector of the local area image blocks, etc.), and finally A subset of local feature descriptors of static images are obtained for subsequent clustering and building of a visual bag of words model. Compared with the traditional manual feature extraction algorithm, the LF-Net network can extract richer features and is hardly affected by the environment such as illumination, with stronger stability and higher accuracy.

When the local feature extraction is performed on the image by using the above-mentioned manner in the embodiment of the present disclosure, the feature extraction will not be performed on the separated dynamic scene region in the original image. That is, when the region is detected as a dynamic scene region during feature extraction, the region is bypassed and local feature extraction is no longer performed. Therefore, the dynamic scene area no longer participates in the feature expression of the image. Finally, the filtered image feature point locations and feature descriptors are determined.

In an optional implementation manner of this embodiment, the step of mapping the local features of the static image to the constructed visual word bag model in step S103 further includes the following steps:

determining a first similarity between the local feature of the static image and the cluster center point in the bag of visual words model;

The local feature of the static image is mapped to a bag-of-words category to which one of the cluster center points belongs according to the first similarity.

In this optional implementation, the obtained local features of the static image are combined with the closed-loop detection bag-of-words model algorithm. A visual bag of words model is constructed by classifying multiple cluster center points.

In some embodiments, by performing a central clustering algorithm on the local features corresponding to the static images corresponding to each to-be-processed image in the to-be-processed image set, after obtaining multiple cluster center points, the local features corresponding to the static images can be mapped to multiple On the cluster center point, the specific mapping process may be performed by calculating the first similarity between the local feature and the multiple cluster center points, and performing mapping according to the first similarity. For example, for a certain local feature B of the static image A, the first similarity between the feature descriptor of the local feature B and the feature descriptor of each cluster center point Ci can be calculated, and the first similarity can be determined by the Euclidean distance. , and map the local feature B to the bag of words category corresponding to the cluster center point Ci with the first maximum similarity. In this way, all local features extracted from all static images can be mapped to the visual word bag model, and finally a visual dictionary tree can be obtained.

In an optional implementation manner of this embodiment, the step of performing feature vector representation on the to-be-processed image set according to the visual word bag model in step S103 further includes the following steps:

Statistical mapping of the local features of the static image to the distribution of the bag-of-words categories in the visual bag-of-words model;

The feature vector of the static image is obtained according to the distribution.

In this optional implementation, after the local features on the static image are mapped to the visual dictionary model, the distribution of the local features on each static image on each word bag category of the visual dictionary model can be counted according to the mapping result, For example, the distribution frequency of local features on a static image on different word bag categories, and then the feature vector of the static image is obtained based on the distribution. For example, the constructed visual word bag model includes 3 word bag categories, and the static image A has a total of 10 local features. After the 10 local features are mapped to the 3 word bag categories, the first word bag category There is 1 local feature mapped on the top, 6 local features are mapped on the second bag of words category, and 3 bag of words categories are mapped on the third bag of words category, the feature vector of static image A can be expressed as (1 , 6, 3), of course, the feature vector can also be normalized as the feature vector of the static image A. It can be understood that the dimension of the feature vector of the static image is the same as the number of bag-of-words categories in the visual bag-of-words model, and is also the same as the number of cluster center points.

In an optional implementation manner of this embodiment, the step of performing closed-loop detection according to the result represented by the feature vector in step S103 further includes the following steps:

determining the second similarity between the feature vectors of the static images corresponding to the two to-be-processed images in the to-be-processed image set;

A closed-loop detection result of the two images to be detected is determined according to the second similarity.

In this optional implementation manner, the feature vector of each to-be-processed image in the to-be-processed image set may be represented by the feature vector of the corresponding static image. Therefore, in determining the closed-loop detection result between two to-be-processed images in the to-be-processed image set , the second similarity between the feature vectors of the static images corresponding to the two images to be processed can be calculated, and when the second similarity is greater than or equal to a preset threshold, it can be considered that the closed-loop detection results of the two images to be processed If yes, that is, a closed loop is generated between the two to-be-processed images, otherwise, no closed-loop is generated between the two to-be-processed images. The second similarity may be determined by using the cosine distance of the two images to be processed.

Because the traditional closed-loop detection algorithm based on local feature extraction has high accuracy in stable environments, the deep learning closed-loop detection algorithm based on global feature extraction in complex environments has better stability. In order to simultaneously improve the stability and accuracy of closed-loop detection in complex environments, an embodiment of the present disclosure proposes a closed-loop detection method based on dynamic and static scene separation and local CNN feature representation. First, in the image preprocessing stage, the input image is separated by a dynamic and static scene separation algorithm based on object detection or semantic segmentation, and the interference of dynamic objects such as pedestrians and vehicles in the image is excluded, and a filtered pure static scene image is obtained. Then, the CNN feature extraction algorithm is used to extract local features from the filtered images. When performing local feature extraction on an image, feature extraction will not be performed on the separated dynamic scene regions in the original image. Finally, the feature point positions and feature descriptors of the filtered image are determined, and combined with the traditional closed-loop detection word bag model algorithm, the set of extracted CNN local feature descriptors is clustered using the word bag model and a visual dictionary tree is constructed. . The feature vector of the image is obtained according to the constructed dictionary tree, and the similarity of the feature vector is compared to determine whether a closed loop is generated.

The beneficial effects produced by the embodiments of the present disclosure mainly lie in:

1) A closed-loop detection algorithm based on dynamic and static scene separation, which solves the problem that the visual word bag model reduces the recognition of image semantic information under the interference of pedestrians, vehicles and other movable objects in the environment, and cannot effectively extract the global scene in the image. In the image preprocessing stage, the embodiments of the present disclosure separate dynamic and static scenes in the original image, filter out non-static scene objects in the image, and reduce interference to image similarity matching. Thus, the accuracy of closed-loop detection is improved.

2), a closed-loop detection method based on local CNN feature representation. Since the traditional closed-loop detection algorithm extracts the local features of the image, the deep-learning-based closed-loop detection algorithm extracts the global image features. Therefore, under the same stable environment, the traditional closed-loop detection algorithm based on local features has more advantages than the closed-loop detection algorithm based on deep learning. High accuracy is high. However, in complex environments (such as drastic changes in illumination), the closed-loop detection algorithm based on deep learning is significantly better than the traditional closed-loop detection algorithm.

Embodiments of the present disclosure propose a method that combines the advantages of deep learning with the advantages of traditional closed-loop detection. In the embodiment of the present disclosure, the convolutional neural network is used to segment the local area of the image and feature extraction, and the extracted local CNN feature descriptor of the image is combined with the traditional closed-loop detection word bag model algorithm, and the obtained image is obtained in the word bag model. The local CNN feature descriptor is clustered by the Kmeans++ algorithm, and a visual dictionary tree is constructed according to the K center points clustered, and then the feature vector representing the image is obtained, and the similarity of the image is compared to determine whether a loop is generated. This method can simultaneously improve the accuracy and stability of closed-loop detection in complex environments.

FIG. 4 shows a schematic flowchart of a method for combining the advantages of deep learning with the advantages of traditional closed-loop detection in an embodiment of the present disclosure. As shown in Figure 4, in the image preprocessing stage, the original image is separated from dynamic and static scenes to obtain a static image after scene separation. Perform local CNN feature detection on static images to obtain image local feature sets and cluster feature points. In the bag-of-words model, the method of statistical histogram is used to count the frequency distribution of feature points on the bag of words in each image to obtain The vector of is the feature vector of the image, and compare the similarity between any two images to determine whether a closed loop is generated.

The following are the apparatus embodiments of the present disclosure, which can be used to execute the method embodiments of the present disclosure.

FIG. 5 shows a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure. The apparatus may be implemented by software, hardware, or a combination of the two to become part or all of an electronic device. As shown in Figure 5, the image processing device includes:

The filtering module 501 is configured to filter the dynamic objects in the to-be-processed images in the to-be-processed image set to obtain a static image;

The feature extraction module 502 is configured to extract local features in the static image by using a CNN network, and perform clustering on the local features to obtain a plurality of cluster center points;

The closed-loop detection module 503 is configured to construct a visual word bag model of the static image according to the plurality of cluster center points, map local features of the static image to the constructed visual word bag model, and Feature vector representation is performed on the to-be-processed image set according to the visual word bag model, and closed-loop detection is performed according to the result represented by the feature vector.

The above-mentioned image processing apparatus proposed in the embodiments of the present disclosure corresponds to the image processing method described above. For details, please refer to the above description of the image processing method, which will not be repeated here.

FIG. 6 is a schematic structural diagram of an electronic device suitable for implementing the image processing method according to an embodiment of the present disclosure.

As shown in FIG. 6 , the electronic device 600 includes a central processing unit (CPU) 601 that can be loaded into a random access memory (RAM) 603 according to a program stored in a read only memory (ROM) 602 or a program from a storage section 608 Instead, various processes in the embodiments of the above-described methods of the present disclosure are performed. In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are also stored. The CPU 601 , the ROM 602 and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc. ; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

In particular, according to embodiments of the present disclosure, the methods in the above-referenced embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program comprising program code for performing the methods of the embodiments of the present disclosure. In such an embodiment, the computer program may be downloaded and installed from the network via the communication section 609 and/or installed from the removable medium 611 .

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the diagram or block diagram may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function. executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units or modules involved in the embodiments of the present disclosure can be implemented in software or hardware. The described units or modules may also be provided in the processor, and the names of these units or modules do not constitute a limitation on the units or modules themselves in certain circumstances.

As another aspect, the present disclosure also provides a computer-readable storage medium, and the computer-readable storage medium may be a computer-readable storage medium included in the apparatus described in the foregoing embodiments; A computer-readable storage medium that fits into a device. The computer-readable storage medium stores one or more programs used by one or more processors to perform the methods described in the present disclosure.

The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, and should also cover the above-mentioned technical features without departing from the inventive concept. Other technical solutions formed by any combination of its equivalent features. For example, a technical solution is formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) with similar functions.

Claims (9)

1. An image processing method, comprising:

filtering dynamic objects in the images to be processed in the image set to be processed to obtain static images;

extracting local features in the static image by using a CNN network, and clustering the local features to obtain a plurality of clustering central points;

and constructing a visual bag-of-words model of the static image according to the plurality of clustering center points, mapping local features of the static image to the constructed visual bag-of-words model, expressing feature vectors of the image set to be processed according to the visual bag-of-words model, and performing closed-loop detection according to the result expressed by the feature vectors.

2. The method of claim 1, wherein filtering dynamic objects in the to-be-processed images in the to-be-processed image set to obtain a static image comprises:

detecting a dynamic object in the image to be processed, and extracting the region information of the dynamic object;

and filtering the dynamic object from the image to be processed according to the region information to obtain the static image.

3. The method according to claim 1 or 2, wherein extracting local features in the static image using a CNN network comprises:

processing the static image by utilizing a multi-scale dense full convolution network, and returning key point information in the static image;

locally cutting the static image according to the key point information to obtain a plurality of local image blocks;

and respectively extracting the features of the local image blocks by using a local feature network to obtain the feature point positions and the feature descriptors of the local image.

4. The method of claim 1 or 2, wherein mapping local features of the static image onto the constructed visual bag-of-words model comprises:

determining a first similarity between the local features of the static image and the cluster center point in the visual bag of words model;

and mapping the local features of the static images to a bag of words category to which one of the cluster center points belongs according to the first similarity.

5. The method of claim 4, wherein performing feature vector representation on the set of images to be processed according to the visual bag-of-words model comprises:

counting the distribution condition of mapping the local features of the static images to each bag type in the visual bag-of-words model;

and obtaining the characteristic vector of the static image according to the distribution condition.

6. The method of claim 5, wherein performing closed-loop detection based on the result of the eigenvector representation comprises:

determining a second similarity between the feature vectors of the static images corresponding to the two images to be processed in the image set to be processed;

and determining the closed-loop detection results of the two images to be detected according to the second similarity.

7. An image processing apparatus characterized by comprising:

the filtering module is configured to filter dynamic objects in the images to be processed in the image set to be processed to obtain static images;

the characteristic extraction module is configured to extract local characteristics in the static image by using a CNN network, and cluster the local characteristics to obtain a plurality of cluster central points;

the closed-loop detection module is configured to construct a visual bag-of-words model of the static image according to the plurality of clustering center points, map local features of the static image to the constructed visual bag-of-words model, perform feature vector representation on the image set to be processed according to the visual bag-of-words model, and perform closed-loop detection according to a result of the feature vector representation.

8. An electronic device comprising a memory and a processor; wherein,

the memory is to store one or more computer instructions, wherein the one or more computer instructions are to be executed by the processor to implement the method of any one of claims 1-6.

9. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method of any of claims 1-6.

Images