CN111062329B - Unsupervised person re-identification method based on augmented network - Google Patents

Abstract

The invention provides an unsupervised pedestrian re-identification method based on an augmentation network, which is characterized in that based on pedestrian image data in an original database, various forms of data augmentation are carried out, and the characteristics of the same label data access parameters which are taken as basic data after the augmentation are respectively extracted by a network with unshared parameters, so that the network is helped to train. The method mainly considers how to utilize the unlabeled data which cannot be directly used as input under the condition that the data set is not abundant, and the main network model obtained by the method can be directly used for testing after the feature extraction is directly carried out on the test set; the method can also be used for pre-training a plurality of augmentation networks and the main network by using the unlabeled data, and then the main network parameters are finely adjusted by using the labeled data, so that the unlabeled information is effectively utilized, and the accuracy of pedestrian re-identification is improved.

Description

Unsupervised person re-identification method based on augmented network

technical field

The present invention relates to the field of deep learning, and more specifically, relates to an unsupervised pedestrian re-identification method.

Background technique

In recent years, deep learning technology has continued to develop, and deep learning methods based on deep neural networks have long been applied in all aspects of our lives. For example, text translation and text classification in the field of Natural Language Processing; image retrieval and face recognition in the field of Computer Vision. The emergence of deep learning methods has brought great convenience to human society.

Pedestrian re-identification method is an important application based on deep learning method. Person re-identification (Person re-identification), also known as pedestrian re-identification, is a technology that uses computer vision technology to judge whether there is a specific pedestrian in the image or video sequence acquired by cameras that do not cover each other. Due to the differences between different camera equipment, pedestrians have both rigid and flexible characteristics, and their appearance is easily affected by clothing, scale, occlusion, posture and viewing angle, etc., making pedestrian re-identification a research value in the field of computer vision. Challenging hot topics.

Pedestrian re-identification has some specialized databases in the academic field, but because the acquisition and calibration of data requires a lot of manpower and financial resources, the number of images in these datasets is very small. Market-1501 and DukeMTMC-reID are two commonly used datasets.

The Market-1501 dataset was collected on the campus of Tsinghua University, with images from 6 different cameras. The training set contains 12,936 images and the test set contains 19,732 images. There are 751 people in the training data and 750 people in the test set. So in the training set, there are an average of 17.2 training data per class (per person).

The DukeMTMC-reID dataset was collected at Duke University, with images from 8 different cameras. The training set contains 16,522 images and the test set contains 17,661 images. There are a total of 702 people in the training data, with an average of 23.5 training data per class (each person).

The above two commonly used pedestrian re-identification datasets only have about 33,000 image data, which is significantly different from the tens of millions of image data in enterprises. If the data set is too small, the training of the neural network will tend to overfitting (Overfitting), resulting in a decrease in the test accuracy of the neural network trained on the original data set on other data sets.

In this case, many data augmentation (Data Augmentation) methods have been used in the field of person re-identification, such as random cropping, random flipping, and so on. However, this method is only a secondary processing of image data on the original labeled dataset, while other unlabeled datasets are still not properly utilized.

Contents of the invention

The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a set of convenient operation and accurate positioning provided by a novel osteotomy logic. , and in the case of meeting the conditions, reduce the material and design difficulty as much as possible, and improve the design efficiency.

In order to achieve the above object, the present invention adopts the following technical solutions:

An unsupervised person re-identification method based on an augmented network, comprising the following steps:

S1: Perform an augmentation operation on the unlabeled original pedestrian image dataset D0, the augmentation operation includes one or more of image scaling, random cropping, random erasing, noise addition, and Gaussian blur to obtain M new Augmented data set D1～DM, M is a positive integer;

S2: Pass the original image data in the original pedestrian image dataset D0 into a convolutional neural network as the main network N0 for forward propagation extraction to obtain the feature F0;

S3: Input the corresponding augmented image data in the M augmented data sets D1~DN into M convolutional neural networks whose parameters are not shared as augmented networks N1~NM, and perform forward propagation to obtain features F1~FM;

S4: Randomly select an image Negative from the original pedestrian image data set D0 as a negative sample, and pass it into the main network N0 for forward propagation to extract the feature Fnegative;

S5: Use the output feature F0 to calculate the Euclidean distance with the output features F1 ~ FM respectively, and obtain M loss values L1 ~ LM;

S6: Use the output feature Fnegative to calculate the Euclidean distance with the output features F0~FM respectively, and obtain M+1 loss values L0nagetive~LMnegative;

S7: Subtract the M loss values L1~LM obtained in S5 from the M loss values L1negative~LMnegative obtained in S6, respectively, and use the result of subtracting the M loss values L1negative~LMnegative obtained in S6 as the loss to perform backpropagation calculation on the augmented network N1~NM to update the gradient. Broad network parameters;

S8: Subtract the sum of the M loss values L1~LM obtained in S5 from the sum of the loss values L0negative~LMnegative obtained in S6 to obtain the total loss value L0;

S9: Use the total loss value L0 obtained in S8 as the loss to perform backpropagation calculation on the main network N0 to update the main network parameters;

S10: Repeat the operations from S2 to S9 until the main network and the augmented network converge;

S11: Take the main network model as an output.

As a preferred technical solution, in step S1, when the augmentation operation includes image scaling processing, the bilinear interpolation method is used to scale the image, thereby simulating images of various resolutions that may appear in natural data sets, specifically It is calculated as follows:

Wherein Q11=(x1, y1), Q12=(x1, y2), Q21=(x2, y1), Q22=(x2, y2) are the four closest pixel points to the point (x, y).

As a preferred technical solution, in step S1, when the augmentation operation includes random clipping processing, the random clipping method is used for augmentation, thereby simulating various local pedestrian images that may appear in the natural data set. The specific method is:

First randomly select a pixel in the image, then use the pixel as the upper left corner to randomly form a rectangle with a certain length and width, and output the pixels in the entire rectangle as the cropping result.

As a preferred technical solution, in step S1, when the augmentation operation includes random erasure processing, the random erasure method is used for augmentation, thereby simulating various missing or incomplete pedestrian images that may appear in the natural data set. The specific method for:

First randomly select a pixel in the image, then use this pixel as the upper left corner to randomly form a rectangle with a certain length and width, and set all the pixel values of the pixels in the entire rectangle to black, that is, the pixel value (0 ,0,0), and then output the whole image after operation as the result of random erasing.

As a preferred technical solution, in step S1, when the augmentation operation includes the noise addition method, the noise addition method is used for augmentation, thereby simulating the image noise that may appear in the natural data set, and the specific operation is:

For each pixel, there is a certain probability that the value becomes a white point, that is, the pixel value (255,255,255)) or a black point, that is, the pixel value (0,0,0), and then the entire image after the operation is used as the noise addition result to output.

As a preferred technical solution, in step S1, when the augmentation operation includes Gaussian blur processing, the Gaussian blur method is used for augmentation, thereby simulating the image blurring that may occur in natural data sets, according to the following formula:

After setting the σ value, the weight matrix can be calculated, so that the matrix operation can be performed with each pixel in the image as the center, and the purpose of blurring the image can be achieved.

As a preferred technical solution, in step S2 and step S3, the respective pedestrian image data are passed into the corresponding convolutional neural network, and the forward propagation method is used for feature extraction. The specific formula of forward propagation is as follows:

Where a represents the output of the middle layer; σ represents the activation function; z represents the input of the activation layer; the superscript represents the number of layers; * represents the convolution operation; W represents the convolution kernel; b represents the bias.

As a preferred technical solution, step S5 is specifically:

Use the feature F0 extracted from the main network N0 to calculate the Euclidean distance with the features F1~FM extracted from the augmented network N1~NM respectively. The specific formula is as follows:

Among them, x is brought into the feature F0 extracted by the main network; y is brought into the features F1~FM extracted from the five augmented networks respectively; xi and yi are the values in each dimension of the corresponding feature;

Step S6 is specifically:

The feature Fnegative is a negative sample, and an image is randomly selected as the negative sample Fnegative to calculate the Euclidean distance with the output features F0-FM.

As a preferred technical solution, step S7 is specifically:

Pass the calculated error value back to the corresponding convolutional neural network, and use the backpropagation algorithm to iteratively update the parameter values of the convolutional neural network. The specific formula is as follows:

Among them, the superscript indicates the number of layers; δ indicates the gradient value; * indicates the convolution operation; W indicates the convolution kernel; rot180 indicates that the matrix is flipped 180 degrees, that is, flipped up and down once, and then flipped left and right once; o means point-to-point multiplication; σ' Denotes the derivative of the activation function.

As a preferred technical solution, step S8 is specifically:

Calculate the Euclidean distance between the features obtained by the augmented network and the features obtained by the main network, and subtract the sum of the error values L1~LM from the sum of L0negative~LMnegative to obtain the total error value L0. The specific formula is as follows:

Where λ _i , i∈[1,M] is the positive sample weight value, L _i , i∈[1,M] corresponds to the positive sample error value, where λ _i =1,i∈[1,M]; λ _negative , negative∈[0,M] is the negative sample weight value, Line _negative , negative∈[0,M] corresponds to the negative sample error value, where λ _ineg =1,ineg∈[0,M].

Compared with the prior art, the present invention has the following advantages and beneficial effects:

The method of the present invention uses the method of augmenting the network to utilize the unlabeled pedestrian image data that cannot be directly used as the input of the deep neural network for training, and can perform end-to-end pair augmentation after the unlabeled data set is augmented. The network is trained with the main network. The deep neural network is trained by using the information that the original data and the features extracted from the augmented data obtained from the original data should be consistent as much as possible. It is of great benefit to the field of pedestrian re-identification where the data set and data volume itself is relatively scarce. In addition, various augmentation operations also simulate to a certain extent the blurred and missing situations that may occur in the pedestrian re-identification data itself. Therefore, the method proposed in the present invention can improve the generalization of the trained deep neural network, alleviate over-fitting, and finally achieve the effect of improving the recognition accuracy

Description of drawings

FIG. 1 is a flow chart of an unsupervised pedestrian re-identification method based on an augmented network in the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

Example

As shown in Figure 1, the present embodiment provides an unsupervised pedestrian re-identification method based on an augmented network, including the following steps:

S1: Perform augmentation operations on the unlabeled pedestrian image data set D0, including image scaling, random cropping, random erasing, noise addition, and Gaussian blur (these five augmentation operations can be selected to combine exhalation volume, or All can be selected, this embodiment takes the augmentation operation in 5 as an example for further description), and five new augmented data sets D1-D5 are obtained.

Image scaling, random cropping, random erasing, noise addition and Gaussian blur in step S1 are specifically:

S11: For the original unlabeled pedestrian image data, the bilinear interpolation method is used to scale the image, thereby simulating various images of different resolutions that may appear in the natural data set. The specific calculation method is shown in the following formula:

Wherein Q11=(x1, y1), Q12=(x1, y2), Q21=(x2, y1), Q22=(x2, y2) are the four closest pixel points to the point (x, y).

S12: For the original unlabeled pedestrian image data, use the random cropping method to augment, so as to simulate various local pedestrian images that may appear in the natural data set. The specific operation is: first randomly select a pixel in the image, and then use the pixel The point is used as the upper left corner to randomly form a rectangle with a certain length and width, and the pixels in the entire rectangle are output as the cropping result.

S13: For the original unlabeled pedestrian image data, use the random erasure method to augment, so as to simulate various missing or incomplete pedestrian images that may appear in the natural data set. The specific operations are:

First randomly select a pixel in the image, then use the pixel as the upper left corner to randomly form a rectangle with a certain length and width, and set all the pixel values of the pixels in the entire rectangle to black (that is, the pixel value (0 ,0,0)), and then output the entire image after the operation as the result of random erasure.

S14: For the original unlabeled pedestrian image data, use the noise addition method to augment, so as to simulate the image noise that may appear in the natural data set. The specific operation is:

For each pixel, there is a certain probability that the value becomes a white point (that is, the pixel value (255,255,255)) or a black point (that is, the pixel value (0,0,0)), and then the entire image after the operation is used as a noise The result is output.

S15: For the original unlabeled pedestrian image data, the Gaussian blur method is used to augment it, so as to simulate the image blurring that may occur in the natural data set. According to the following formula:

After setting the σ value, the weight matrix can be calculated, so that the matrix operation can be performed with each pixel in the image as the center, and the purpose of blurring the image can be achieved.

S2: Pass the original image data in the original pedestrian image data set D0 into a convolutional neural network as the main network N0 for forward propagation extraction to obtain the feature F0; that is, pass the respective pedestrian image data into the corresponding convolutional neural network. The forward propagation method is used for feature extraction, and the specific formula of forward propagation is as follows:

Where a represents the output of the middle layer; σ represents the activation function; z represents the input of the activation layer; the superscript represents the number of layers; * represents the convolution operation; W represents the convolution kernel; b represents the bias.

S3: Input the corresponding augmented image data in the five augmented data sets D1~D5 into five convolutional neural networks whose parameters are not shared as augmented networks N1~N5 for forward propagation extraction to obtain features F1~F5; step The forward propagation in S3 adopts the same method as in step S2.

S4. Randomly select an image Negative from the original pedestrian image data set D0 as a negative sample, and pass it into the main network N0 for forward propagation to extract the feature Fnegative;

S5: Use the output feature F0 to calculate the Euclidean distance with the output features F1~F5 respectively, and get five loss values L1~L5, specifically:

Use the feature F0 extracted by the main network N0 to calculate the Euclidean distance with the features F1-F5 extracted by the augmented network N1-N5 respectively. The specific formula is as follows:

Among them, x is brought into the feature F0 extracted by the main network; y is brought into the features F1-F5 extracted from the five augmented networks; xi and yi are the values in each dimension of the corresponding feature.

S6: Use the output feature Fnegative to calculate the Euclidean distance with the output features F0~F5 respectively, and get 6 loss values L0nagetive~LMnegative; because generally speaking, the amount of data in the data set is large, and the proportion of the same type of data relative to the overall data amount is small, Therefore, it is feasible to use a randomly selected image as a negative sample here in most cases.

S7: Subtract the five loss values L1~L5 obtained in S5 from the five loss values L1negative~LMnegative obtained in S6, respectively, and use the results obtained after subtracting the five loss values L1negative~LMnegative obtained in S6 as the loss to perform backpropagation calculation on the augmented network N1~N5 to update the gradient. wide network parameters.

The calculated error value is sent back to the corresponding convolutional neural network, and the parameter value of the convolutional neural network is iteratively updated using the back propagation algorithm. The specific formula is as follows:

Among them, the superscript indicates the number of layers; δ indicates the gradient value; * indicates the convolution operation; W indicates the convolution kernel; rot180 indicates that the matrix is flipped 180 degrees, that is, flipped up and down once, and then flipped left and right once; o means point-to-point multiplication; σ' Denotes the derivative of the activation function.

S8: Subtract the sum of the five loss values L1~L5 obtained in S4 from the sum of the loss values L0negative~L5negative obtained in S6 to obtain the total loss value L0. The specific formula is as follows:

Where λ _i , i∈[1,5] is the positive sample weight value, L _i , i∈[1,5] corresponds to the positive sample error value, where λ _i =1,i∈[1,5]; λ _negative , negative∈[0,5] is the negative sample weight value, and the negative sample error value corresponding to Line _negative , negative∈[0,5], where λ _negative =1, negative∈[0,5].

S9: Use the total loss value L0 obtained in S6 as the loss to perform backpropagation to the main network N0 to calculate the gradient and update the main network parameters.

S10: Repeat the operations of S2-S10 until the main network and the augmented network converge.

S11: Take the main network model as an output.

The above-mentioned embodiment is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited by the above-mentioned embodiment, and any other changes, modifications, substitutions, combinations, Simplifications should be equivalent replacement methods, and all are included in the protection scope of the present invention.

Claims (10)

1. An unsupervised pedestrian re-identification method based on augmented network, is characterized in that, comprises the following steps: S1: Perform an augmentation operation on the unlabeled original pedestrian image dataset D0, the augmentation operation includes one or more of image scaling, random cropping, random erasing, noise addition, and Gaussian blur to obtain M new Augmented data set D1～DM, M is a positive integer; S2: Pass the original image data in the original pedestrian image dataset D0 into a convolutional neural network as the main network N0 for forward propagation extraction to obtain the feature F0; S3: Input the corresponding augmented image data in the M augmented data sets D1~DN into M convolutional neural networks whose parameters are not shared as augmented networks N1~NM, and perform forward propagation to obtain features F1~FM; S4: Randomly select an image Negative from the original pedestrian image data set D0 as a negative sample, and pass it into the main network N0 for forward propagation to extract the feature Fnegative; S5: Use the output feature F0 to calculate the Euclidean distance with the output features F1 ~ FM respectively, and obtain M loss values L1 ~ LM; S6: Use the output feature Fnegative to calculate the Euclidean distance with the output features F0~FM respectively, and obtain M+1 loss values L0nagetive~LMnegative; S7: Subtract the M loss values L1~LM obtained in S5 from the M loss values L1negative~LMnegative obtained in S6, respectively, and use the result of subtracting the M loss values L1negative~LMnegative obtained in S6 as the loss to perform backpropagation calculation on the augmented network N1~NM to update the gradient. Broad network parameters; S8: Subtract the sum of the M loss values L1~LM obtained in S5 from the sum of the loss values L0negative~LMnegative obtained in S6 to obtain the total loss value L0; S9: Use the total loss value L0 obtained in S8 as the loss to perform backpropagation calculation on the main network N0 to update the main network parameters; S10: Repeat the operations from S2 to S9 until the main network and the augmented network converge; S11: Take the main network model as an output. 2. The unsupervised pedestrian re-identification method based on the augmented network according to claim 1, wherein in step S1, when the augmented operation includes image scaling processing, the image is zoomed using a bilinear interpolation method , so as to simulate various images of different resolutions that may appear in the natural data set, the specific calculation method is as follows:

Wherein Q11=(x1, y1), Q12=(x1, y2), Q21=(x2, y1), Q22=(x2, y2) are the four closest pixel points to the point (x, y). 3. The method for unsupervised person re-identification based on augmented network according to claim 1, characterized in that, in step S1, when the augmentation operation includes random clipping, the random clipping method is used for augmentation, thereby simulating natural Various local pedestrian images that may appear in the data set, the specific method is: First randomly select a pixel in the image, then use the pixel as the upper left corner to randomly form a rectangle with a certain length and width, and output the pixels in the entire rectangle as the cropping result. 4. The unsupervised person re-identification method based on augmented network according to claim 1, characterized in that, in step S1, when the augmentation operation includes random erasure processing, the random erasure method is used for augmentation, thereby Simulate various missing or incomplete pedestrian images that may appear in natural datasets by: First randomly select a pixel in the image, then use this pixel as the upper left corner to randomly form a rectangle with a certain length and width, and set all the pixel values of the pixels in the entire rectangle to black, that is, the pixel value (0 ,0,0), and then output the whole image after operation as the result of random erasing. 5. The method of unsupervised person re-identification based on augmented network according to claim 1, characterized in that, in step S1, when the augmented operation includes noise-adding method processing, use the noise-adding method to augment, thereby simulating Image noise that may appear in natural data sets, the specific operation is: For each pixel, there is a certain probability that the value becomes a white point, that is, the pixel value (255,255,255)) or a black point, that is, the pixel value (0,0,0), and then the entire image after the operation is used as the noise addition result to output. 6. The unsupervised person re-identification method based on augmented network according to claim 1, characterized in that, in step S1, when the augmented operation includes Gaussian blur processing, the Gaussian blur method is used for augmentation, thereby simulating natural The image blurring that may appear in the data set is based on the following formula:

After setting the σ value, the weight matrix can be calculated, so that the matrix operation can be performed with each pixel in the image as the center, and the purpose of blurring the image can be achieved. 7. The unsupervised pedestrian re-identification method based on the augmented network according to claim 1, characterized in that, in step S2 and step S3, the respective pedestrian image data are passed into the corresponding convolutional neural network, and the forward propagation method is used For feature extraction, the specific formula of forward propagation is as follows:

Where a represents the output of the middle layer; σ represents the activation function; z represents the input of the activation layer; the superscript represents the number of layers; * represents the convolution operation; W represents the convolution kernel; b represents the bias. 8. The unsupervised pedestrian re-identification method based on the augmented network according to claim 1, wherein step S5 is specifically: Use the feature F0 extracted from the main network N0 to calculate the Euclidean distance with the features F1~FM extracted from the augmented network N1~NM respectively. The specific formula is as follows:

Among them, x is brought into the feature F0 extracted by the main network; y is brought into the features F1~FM extracted from the five augmented networks respectively; xi and yi are the values in each dimension of the corresponding feature; Step S6 is specifically: The feature Fnegative is a negative sample, and an image is randomly selected as the negative sample Fnegative to calculate the Euclidean distance with the output features F0-FM. 9. The unsupervised pedestrian re-identification method based on the augmented network according to claim 1, wherein step S7 is specifically: Pass the calculated error value back to the corresponding convolutional neural network, and use the backpropagation algorithm to iteratively update the parameter values of the convolutional neural network. The specific formula is as follows:

Among them, the superscript indicates the number of layers; δ indicates the gradient value; * indicates the convolution operation; W indicates the convolution kernel; rot180 indicates that the matrix is flipped 180 degrees, that is, flipped up and down once, and then flipped left and right once; o means point-to-point multiplication; σ' Denotes the derivative of the activation function. 10. The method for unsupervised person re-identification based on augmented network according to claim 1, characterized in that step S8 is specifically: Calculate the Euclidean distance between the features obtained by the augmented network and the features obtained by the main network, and subtract the sum of the error values L1~LM from the sum of L0negative~LMnegative to obtain the total error value L0. The specific formula is as follows:

Where λ _i , i∈[1,M] is the positive sample weight value, L _i , i∈[1,M] corresponds to the positive sample error value, where λ _i =1,i∈[1,M]; λ _negative , negative∈[0,M] is the negative sample weight value, and the negative sample error value corresponding to Line _negative , negative∈[0,M]. Here, λ _negative ＝1, negative∈[0,M].

Images