CN111340090B - Image feature comparison method and device, equipment and computer readable storage medium - Google Patents

Abstract

The application relates to an image feature comparison method, a gait recognition method and equipment. The method comprises the following steps: and calculating the weight of each image feature by using a weight calculation module obtained by combined training with the image feature extraction neural network, and carrying out weighting treatment on each image feature to obtain the distance between the compared videos. The method can improve the precision of gait recognition results.

Description

Image feature comparison method, device, device, and computer-readable storage medium

technical field

The present application relates to the field of computer technology, in particular to an image feature comparison method, a gait recognition method and equipment.

Background technique

In the field of video gait recognition and other image processing technologies, image feature comparison is required, that is, the similarity between two images/videos to be compared is determined according to the extracted image features.

Taking the field of video gait recognition as an example, through the comparison of image features between videos, the distance (ie, similarity) between two videos is calculated, and then it is judged whether the targets (usually pedestrians) in the two videos are the same target .

In the existing implementation, the Euclidean distance of the same image features between videos is calculated, and the sum of the Euclidean distances of each image feature is used as the distance between two videos. But for different image variables, the robust image features are different. For example, if a pedestrian in one video wears a thick coat, while a pedestrian in another video wears a thin coat, then the image features corresponding to the upper garment area are less robust, and the image feature extraction and comparison are performed according to the previous age implementation method. It will lead to lower accuracy of gait recognition results.

In order to solve the problem caused by extracting image features with poor robustness, in another existing implementation, different image features are extracted for various image variables. For example, if a pedestrian in one video wears a thick coat, and a pedestrian in another video wears a thin coat, then the image features corresponding to the upper coat area are not extracted. Although this implementation method can avoid extracting image features with poor robustness to a certain extent, it needs to extract different image features for various image variables and train image feature extraction neural networks separately.

Contents of the invention

In order to solve the above technical problems, a method capable of image feature comparison, a gait recognition method and equipment are proposed.

In the first aspect, the embodiment of the present application provides an image feature comparison method, the method comprising:

Using the image feature extraction neural network to obtain multiple image features of the first video and the second video respectively;

respectively determining the distance of the same image feature between the first video and the second video;

Utilize the weight calculation module to calculate the weight of the same image feature between the first video and the second video, the weight calculation module is obtained through joint training with the image feature extraction neural network, and the weight calculation module uses calculating the weight of image features according to the target image variables of the first video and the second video;

Weighting the plurality of image features with respective weights of the plurality of image features, determining the distance between the first video and the second video according to the weighting result, and determining the distance between the first video and the second video The distance between videos reflects the similarity between the target image of the first video and the target image of the second video.

In a second aspect, an embodiment of the present application provides a computer device, including a processor and a memory;

The memory is used to store a program for executing the method described in the first aspect;

The processor is configured to execute programs stored in the memory.

In a third aspect, the embodiment of the present application provides a gait recognition method, the method comprising:

Using the image feature extraction neural network to obtain a plurality of image features of the first video, and obtain a plurality of image features of each of the plurality of second videos;

separately determining the distance of the same image feature between the first video and each second video;

Utilize the weight calculation module to calculate the weight of the same image feature between the first video and each second video, the weight calculation module is obtained through joint training with the image feature extraction neural network, and the weight calculation module uses calculating the weight of the image features according to the target image variables of the first video and the second video;

For each second video, using the respective weights of the multiple image features to weight the multiple image features, and determining the distance between the first video and the corresponding second video according to the weighting result, the first video The distance between a video and the second video reflects the gait similarity between the target image of the first video and the target image of the second video;

According to the distance between the first video and each second video, at least one object corresponding to the second video is selected as the gait recognition result of the first video.

In a fourth aspect, the embodiment of the present application provides a computer device, including a processor and a memory;

The memory is used to store a program for executing the method described in the third aspect;

The processor is configured to execute programs stored in the memory.

The technical solutions provided by the embodiments of the present application have at least the following technical effects or advantages:

The weight calculation module obtained by joint training with the image feature extraction neural network is used to calculate the weight of each image feature, and carry out weighting processing on each image feature to obtain the distance between the comparison videos. Since the weight of the image feature is calculated according to the target image variable, the importance of each image feature under different target image variables is considered. For different target image variables, the same image feature can still be extracted, and the weight of the image feature is calculated using the calculated weight. Weighting processing is performed to improve the robustness of important image features, and when applied to gait recognition, the accuracy of gait recognition can be improved.

Description of drawings

Fig. 1 is a flow chart of the image feature comparison method provided by the embodiment of the present application;

Fig. 2 is a flow chart of the gait recognition method provided by the embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

In some processes described in the specification and claims of the present application and the above-mentioned drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these processes may include more or fewer operations, and These operations can be performed sequentially or in parallel.

An embodiment of the present invention provides an image feature comparison method, as shown in Figure 1, the method includes the following operations:

Step 101, using the image feature extraction neural network to obtain multiple image features of the first video and the second video respectively.

Wherein, the image feature specifically refers to a feature vector of the target image in the video.

In the embodiment of the present invention, the first video and the second video may be original videos collected by a camera, or may be processed silhouette videos of people. The so-called person profile video means that each frame of image in the video is a person profile image.

The method provided by the embodiment of the present invention is applicable to the image feature extraction neural network training stage, then, the first video and the second video are both sample videos for training the image feature extraction neural network. The method provided by the embodiment of the present invention is also applicable to the application stage after the image feature extraction neural network training is completed, such as the gait recognition process, then, the first video is a video to be processed (for example, to be recognized), and the second video can be a video to be processed. The processed video may also be a video with known recognition results (such as a known gait video of a specific person).

Step 102. Determine the distances of the same image features between the first video and the second video respectively.

By way of example and not limitation, the Euclidean distance of the same image features is determined. Of course, in practical applications, other known distances may also be used instead of the Euclidean distance, which is not limited in this embodiment of the present invention.

Step 103, using the weight calculation module to calculate the weight of the same image feature between the first video and the second video, the weight calculation module is obtained through joint training with the image feature extraction neural network, the weight The calculation module is used for calculating the weight of image features according to the target image variables of the first video and the second video.

In the embodiment of the present invention, the target image in the video refers to an image of a pedestrian in the video (it may be but not limited to a silhouette image). The variable of the target image includes at least one of the following: a shooting angle of the target image, and a type of clothing of the target image.

In the embodiment of the present invention, the image variable of the video is also referred to as the condition of the video.

Wherein, the shooting angle may be a precise numerical value, or may be a numerical interval. If the shooting angle is an accurate value, in practical applications, multiple shooting angles can be pre-determined, such as 0°, 30°, 60°..., to determine the actual shooting angle of the target image, and match the actual shooting angle to the closest Predetermined shooting angle. For example, if the actual shooting angle of the target image is 35°, then the shooting angle as a variable of the target image is 30°.

Wherein, the clothing type of the jacket may be specifically determined according to the needs of the scene in practical applications. As a distance and not a limitation, the types of tops include: thin coats, thick coats, long coats, short coats, etc.

Step 104: Use the respective weights of the multiple image features to weight the multiple image features, determine the distance between the first video and the second video according to the weighting result, and determine the distance between the first video and the second video. The distance between the second videos reflects the similarity between the target image of the first video and the target image of the second video.

As an example but not a limitation, specifically, the sum of the weighted results is used as the distance between the first video and the second video.

In practical applications, the specific content of the image feature comparison is determined according to the application scenario of the image feature comparison result. For example, in the gait recognition scenario, what is compared is the gait similarity of target images between videos.

In the method provided by the embodiment of the present invention, the weight calculation module obtained through joint training with the image feature extraction neural network is used to calculate the weight of each image feature, perform weighting processing on each image feature, and obtain the distance between the comparison videos. Since the weight of the image feature is calculated according to the target image variable, the importance of each image feature under different target image variables is considered. For different target image variables, the same image feature can still be extracted, and the weight of the image feature is calculated using the calculated weight. Weighting processing is performed to improve the robustness of important image features, and when applied to gait recognition, the accuracy of gait recognition can be improved.

The implementation of each step of the image feature comparison method provided by the embodiment of the present invention will be further described below.

There are many ways to implement the above step 101. In the training phase of the image feature extraction neural network, an implementation method is to input the first video and the second video into the image feature extraction neural network respectively, and obtain multiple images of the first video and the second video output by the image feature extraction neural network respectively. Image feature; another kind of realization mode is, the first video and the second video input image feature extraction neural network respectively, obtain the feature map of the first video of the image feature extraction neural network output and the second video, first video respectively and the feature map of the second video are segmented according to predetermined rules, and a pooling operation is performed on each sub-feature map obtained by the segmentation to obtain image features corresponding to each sub-feature map of the first video and the second video respectively. In the image feature extraction neural network application stage, in one implementation, the first video is input to the image feature extraction neural network to obtain multiple image features of the first video output by the image feature extraction neural network, and read from a predetermined storage location A plurality of image features of the predetermined second video; another implementation is to input the first video into the image feature extraction neural network, obtain the feature map of the first video output by the image feature extraction neural network, and extract the first video feature The image is segmented according to predetermined rules, and each sub-feature map obtained by segmentation is pooled to obtain the image features corresponding to each sub-feature map of the first video, and the predetermined second video is read from a predetermined storage location. Multiple image features for video. Wherein, the manner of determining the image feature of the second video may refer to the manner of determining the image feature of the first video.

It can be seen that no matter what the application scenario is, the extraction of image features is realized by using the image feature extraction neural network. For example, inputting the first video into the image feature extraction neural network to obtain the feature map of the first video output by the image feature extraction neural network, the size of the feature map of the first video is N×C ×H×W, N is the number of frames of the first video, C is the number of channels of a single frame of the first video, H is the height of a single frame in the first video, W is the height of the first video The width of a single frame image in a video; the feature map of the first video is segmented along the horizontal direction to obtain M sub-feature maps; the pooling operation is performed on each sub-feature map of the first video respectively to obtain the first video There are M image features of a video, and the size of a single image feature of the first video is N×C×1. For another example, the second video is input into the image feature extraction neural network to obtain the feature map of the second video output by the image feature extraction neural network, and the size of the feature map of the second video is N× C×H×W, N is the frame number of the second video, C is the channel number of a single frame video of the second video, H is the height of a single frame image in the second video, W is the The width of a single frame image in the second video; the feature map of the second video is segmented along the horizontal direction (H dimension) to obtain M sub-feature maps; the pooling operation is performed on each sub-feature map of the second video respectively , to obtain M image features of the second video, where the size of a single image feature of the second video is N×C×1.

In a specific embodiment, the image feature extraction neural network includes 6 convolutional layers and 2 pooling layers.

The embodiment of the present invention does not limit the pooling operation of the sub-feature map, for example, maximum pooling and average pooling may be used.

In the embodiment of the present invention, the feature map can be equally divided into M sub-feature maps, or can be divided according to a predetermined segmentation rule, which is not limited in the embodiment of the present invention.

It should be noted that in actual reference, there may be different feature map segmentation methods according to different conditions of image features, which is not limited in this embodiment of the present invention.

There are many ways to implement the above step 102. Taking determining the Euclidean distance of the same image feature as an example, the existing Euclidean distance calculation method can be used to calculate the Euclidean distance of the same image feature between videos.

There are many ways to realize the above step 103 . A preferred implementation method is to obtain the gait energy map of the first video and the gait energy map of the second video respectively; the gait energy map of the first video and the gait energy map of the second video The gait energy map is input into the weight calculation module, and the weight of the same image feature between the first video and the second video output by the weight calculation module is obtained. Wherein, the gait energy map may be acquired in an existing implementation manner, which is not limited in this embodiment of the present invention. In another implementation manner, the first video and the second video are input to the weight calculation module, and the same image between the first video and the second video output by the weight calculation module is obtained The weight of the feature. In yet another implementation manner, multiple image features of the first video and multiple image features of the second video are input into the weight calculation module, and the first video and the output of the weight calculation module are obtained. Weights for the same image features between the second videos.

For example, if there are M image features, the weight calculation module outputs M weights, and each weight corresponds to an image feature.

There are many specific implementation manners of the weight calculation module, which are not limited in this embodiment of the present invention. As an example and not limitation, through training, the pre-image variable comparison result and the weight mapping relationship, in the application process, first compare the image variables of the target image in the two videos (or gait energy maps, image features), such as comparing Take the shooting angle and/or top clothing type of the image variables, and find a set of weights corresponding to the comparison results. Among them, the implementation of comparing the image variables of the target image in the two videos (or gait energy map, image features) can be, firstly, identify the image variables of the target image in each video (such as identifying the shooting angle, identifying the type of clothing on the top) etc.), and then compare the image variables. Among them, the existing image processing technology can be used to identify the shooting angle and the type of clothing.

In the embodiment of the present invention, the joint training method of the weight calculation module and the image feature extraction neural network is:

Using the first distance between sample videos to determine a first loss function, using the first loss function and the sample video to train the image feature extraction neural network, the first distance between the sample videos is using the sample video Calculated from the distance between the same image features;

Using a second distance between sample videos to determine a second loss function; using the second loss function and the sample video to train the weight calculation module, or using the second loss function and the sample video to train the weight calculation module The weight calculation module and the image feature extraction neural network; the second distance between the sample videos is calculated by using the weight of the same image feature between the sample videos to weight the distance of the image feature.

Through the above joint training method, more accurate weight results can be obtained.

It should be noted that the implementation manners of the above steps can be combined arbitrarily to obtain new embodiments.

On the basis of any of the above method embodiments, the target image variable includes the shooting angle of the target image, and when the shooting angles of the target image of the first video and the second video are consistent, the first video and the second video have the same shooting angle. The weight of the image feature corresponding to the shoulder image area between the second video is greater than the weight of other image features, and when the shooting angles of the target image of the first video and the second video are inconsistent, the first video and the second video The weights of the image features corresponding to the head and foot image regions between the second video are greater than the weights of other image features.

On the basis of any of the above method embodiments, the target image variable includes the upper clothing type of the target image, and when the upper clothing type of the target image in the first video and the second video are different, the first video The weights of the image features corresponding to the upper garment image area between the second video are smaller than the weights of other image features.

Based on the same inventive concept, an embodiment of the present invention also provides a computer device, including a processor and a memory; the memory is used to store a program for executing the method described in any of the above embodiments; the processor is configured to execute the programs stored in memory.

The embodiment of the present invention also provides a gait recognition method, as shown in Figure 2, the method includes:

Step 201, using the image feature extraction neural network to acquire multiple image features of the first video, and acquire multiple image features of multiple second videos respectively.

In this embodiment, the first video is a video to be recognized, and the plurality of second videos are videos of known recognition results (such as sample videos).

For the manner of acquiring multiple image features of the first video, refer to the above-mentioned method embodiments, which will not be repeated here. Wherein, multiple image features of the second video are predetermined, and in this step, the predetermined image features of the second video are read.

Step 202, respectively determine the distance of the same image feature between the first video and each second video.

Assuming that there are X second videos (V ₁ , V ₂ , ... V _X ) and M image features, the distances of the same image features between the first video and each second video are respectively determined to obtain X groups of distances, Each group of distances corresponds to a second video, and the number of a group of distances is M, which are respectively the distances corresponding to each image feature.

Step 203, using the weight calculation module to calculate the weight of the same image feature between the first video and each second video, the weight calculation module is obtained through joint training with the image feature extraction neural network, the weight The calculating module is used for calculating the weight of the image features according to the target image variables of the first video and the second video.

Assuming that there are X second videos (V ₁ , V ₂ , ... V _X ) and M image features, each time the first video and a second video are used as a set of input weight calculation modules to obtain X sets of output results , each set of output results includes M weights, and each weight corresponds to an image feature.

For the specific implementation manner, reference may be made to the description of the foregoing method embodiments, and details are not repeated here.

Step 204. For each second video, weight the plurality of image features using their respective weights, and determine the distance between the first video and the corresponding second video according to the weighting result, The distance between the first video and the second video reflects the gait similarity between the target image of the first video and the target image of the second video.

Calculate the distance between the first video and each second video respectively. When calculating the distance between the first video and each second video, its implementation can refer to the description of the above method embodiment, and will not be repeated here .

Step 205, according to the distance between the first video and each second video, select at least one object corresponding to the second video as the gait recognition result of the first video.

Preferably, the target corresponding to the closest second video is used as the gait recognition result of the first video.

Wherein, the target refers to a person corresponding to the target object.

In the method provided by the embodiment of the present invention, the weight calculation module obtained through joint training with the image feature extraction neural network is used to calculate the weight of each image feature, perform weighting processing on each image feature, and obtain the distance between the comparison videos. Since the weight of the image feature is calculated according to the target image variable, the importance of each image feature under different target image variables is considered. For different target image variables, the same image feature can still be extracted, and the weight of the image feature is calculated using the calculated weight. Weighting processing is performed to improve the robustness of important image features, and when applied to gait recognition, the precision and accuracy of gait recognition can be improved.

Wherein, the weight calculation module using the weight calculation module calculates the weight of the same image feature between the first video and each second video, including:

Obtaining the gait energy map of the first video and the gait energy map of each second video respectively;

input the gait energy map of the first video and the gait energy map of the single second video into the weight calculation module, and obtain the output of the weight calculation module between the first video and the single second video The weights of the same image features between them.

Based on the same inventive concept, an embodiment of the present invention provides a computer device, including a processor and a memory;

The memory is used to store a program for executing the method described in any of the above-mentioned embodiments of the gait recognition method;

The processor is configured to execute programs stored in the memory.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: Read Only Memory (ROM, Read Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

The embodiments described above are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

Claims (9)

1. An image feature comparison method, characterized in that the method comprises: Using the image feature extraction neural network to obtain multiple image features of the first video and the second video respectively; respectively determining the distance of the same image feature between the first video and the second video; Utilize the weight calculation module to calculate the weight of the same image feature between the first video and the second video, the weight calculation module is obtained through joint training with the image feature extraction neural network, and the weight calculation module uses calculating the weight of image features according to the target image variables of the first video and the second video; Weighting the plurality of image features with respective weights of the plurality of image features, determining the distance between the first video and the second video according to the weighting result, and determining the distance between the first video and the second video The distance between videos reflects the similarity between the target image of the first video and the target image of the second video; The target image variable includes the shooting angle of the target image and the upper clothing type of the target image. When the shooting angles of the target image of the first video and the second video are consistent, the first video and the second video The weight of the image feature corresponding to the shoulder image area between the videos is greater than the weight of other image features. When the shooting angles of the target image of the first video and the second video are inconsistent, the first video and the second video The weights of the image features corresponding to the head and foot image regions between the videos are greater than the weights of other image features. When the upper clothing type of the target image of the first video and the second video is different, the first video The weights of the image features corresponding to the upper garment image area between the second video are smaller than the weights of other image features. 2. The method according to claim 1, wherein the weight calculation module using the weight calculation module calculates the weight of the same image feature between the first video and the second video, comprising: Obtaining the gait energy map of the first video and the gait energy map of the second video respectively; Input the gait energy map of the first video and the gait energy map of the second video into the weight calculation module, and obtain the difference between the first video and the second video output by the weight calculation module The weights of the same image features between them. 3. method according to claim 1, is characterized in that, the joint training mode of described weight calculation module and described image feature extraction neural network is: Using the first distance between sample videos to determine a first loss function, using the first loss function and the sample video to train the image feature extraction neural network, the first distance between the sample videos is using the sample video Calculated from the distance between the same image features; Using a second distance between sample videos to determine a second loss function; using the second loss function and the sample video to train the weight calculation module, or using the second loss function and the sample video to train the weight calculation module The weight calculation module and the image feature extraction neural network; the second distance between the sample videos is calculated by using the weight of the same image feature between the sample videos to weight the distance of the image feature. 4. The method according to claim 1, wherein said utilizing image feature extraction neural network to obtain a plurality of image features of the first video and the second video respectively, comprising: Inputting the first video into the image feature extraction neural network to obtain a feature map of the first video output by the image feature extraction neural network, the size of the feature map of the first video is N×C×H ×W, N is the frame number of the first video, C is the channel number of a single frame video of the first video, H is the height of a single frame image in the first video, W is the first video The width of a single frame image in the middle; the feature map of the first video is segmented along the horizontal direction to obtain M sub-feature maps; each sub-feature map of the first video is respectively pooled to obtain the first video M image features of the first video, the size of a single image feature of the first video is N×C×1; Inputting the second video into the image feature extraction neural network to obtain a feature map of the second video output by the image feature extraction neural network, the size of the feature map of the second video is N×C×H ×W, N is the frame number of the second video, C is the channel number of a single frame video of the second video, H is the height of a single frame image in the second video, W is the second video the width of a single frame image; the feature map of the second video is segmented along the horizontal direction to obtain M sub-feature maps; each sub-feature map of the second video is respectively pooled to obtain the second video M image features of the second video, the size of a single image feature of the second video is N×C×1. 5. The method according to any one of claims 1 to 4, characterized in that: The distance between the first video and the second video is the sum of weighted results; and / or, The distance of the same image feature between the first video and the second video is the Euclidean distance; and / or, The distance between the first video and the second video reflects the gait similarity between the target image of the first video and the target image of the second video. 6. A computer device, comprising a processor and a memory; The memory is used to store a program for executing the method according to any one of claims 1 to 5; The processor is configured to execute programs stored in the memory. 7. A gait recognition method, characterized in that the method comprises: Using the image feature extraction neural network to obtain a plurality of image features of the first video, and obtain a plurality of image features of each of the plurality of second videos; separately determining the distance of the same image feature between the first video and each second video; Utilize the weight calculation module to calculate the weight of the same image feature between the first video and each second video, the weight calculation module is obtained through joint training with the image feature extraction neural network, and the weight calculation module uses calculating the weight of the image features according to the target image variables of the first video and the second video; For each second video, using the respective weights of the multiple image features to weight the multiple image features, and determining the distance between the first video and the corresponding second video according to the weighting result, the first video The distance between a video and the second video reflects the gait similarity between the target image of the first video and the target image of the second video; According to the distance between the first video and each second video, select at least one target corresponding to the second video as the gait recognition result of the first video; The target image variable includes the shooting angle of the target image and the upper clothing type of the target image. When the shooting angles of the target image of the first video and the second video are consistent, the first video and the second video The weight of the image feature corresponding to the shoulder image area between the videos is greater than the weight of other image features. When the shooting angles of the target image of the first video and the second video are inconsistent, the first video and the second video The weights of the image features corresponding to the head and foot image regions between the videos are greater than the weights of other image features. When the upper clothing type of the target image of the first video and the second video is different, the first video The weights of the image features corresponding to the upper garment image area between the second video are smaller than the weights of other image features. 8. The method according to claim 7, wherein the weight calculation module using the weight calculation module calculates the weight of the same image feature between the first video and each second video, comprising: Obtaining the gait energy map of the first video and the gait energy map of each second video respectively; input the gait energy map of the first video and the gait energy map of the single second video into the weight calculation module, and obtain the output of the weight calculation module between the first video and the single second video The weights of the same image features between them. 9. A computer device, comprising a processor and a memory; The memory is used to store a program for executing the method according to claim 7 or 8; The processor is configured to execute programs stored in the memory.