CN110046599A - Intelligent control method based on depth integration neural network pedestrian weight identification technology - Google Patents

Abstract

The invention provides an intelligent monitoring method based on the deep fusion neural network pedestrian re-identification technology. The method of the invention performs color enhancement preprocessing on the acquired image, extracts traditional manual features and deep residual convolution neural network to extract the image, and then extracts the traditional manual features after dimension reduction and the well-trained neural network. The obtained deep features are fused to complete the recognition of the target pedestrian. Compared with the prior art, the present invention greatly improves the recognition accuracy, and the pedestrian re-identification algorithm included in the present invention increases the recognition success rate to 81.74%, making the technology completely practical. The process of pedestrian re-identification is completed automatically, and details will not be lost due to personnel fatigue.

Description

Intelligent monitoring method based on deep fusion neural network pedestrian re-identification technology

technical field

The invention relates to the technical field of intelligent monitoring, in particular to an intelligent monitoring method based on a deep fusion neural network pedestrian re-identification technology.

Background technique

Pedestrian re-identification, also known as pedestrian re-identification, is a technology that uses computer vision technology to determine whether there is a specific pedestrian in an image or video sequence. Due to the differences between different camera devices, pedestrians have both rigid and flexible characteristics, and their appearance is easily affected by clothing, scale, occlusion, posture, and perspective. Therefore, pedestrian re-identification has become a research value in the field of computer vision. Challenging hot topic.

Currently, there are two main research methods for person re-identification: methods based on handcrafted features and methods based on deep features. Methods based on hand-crafted features usually need to find a robust feature to deal with the effects of illumination and viewing angle changes, which are mostly combined with metric learning; while methods based on deep features focus on matching from training data Based on the self-adaptive structure of the label, it can realize the end-to-end identification of the target pedestrian. In recent years, deep features extracted by convolutional neural networks have been shown to be robust. Since it was introduced into the field of person re-identification in 2014, deep feature learning and deep distance measurement based on convolutional neural networks have become the mainstream of person re-identification. Most of the pedestrian re-identification algorithms since then have been structurally improved, including the triplet loss-based cross neural network proposed by Weihua Chen in 2017, and the singular value neural network based on singular value decomposition proposed by Yifan Sun in 2017 ( SVDNet) and SpindleNet based on human joint structure features. These algorithms have good performance, but there is still room for improvement in robustness. In 2016, Zheng Weishi proposed to re-identify by the fusion of manual features and deep features. The algorithm deduces the back-propagation process of the loss function to each parameter, and also proves the constraints of the traditional hand-crafted features on the parameters of the neural network. However, in practical applications, the neural network is difficult to train, the convergence speed is difficult to guarantee, and the hand-crafted features and The coupling of deep features is weak.

SUMMARY OF THE INVENTION

According to the technical problems raised above, an intelligent monitoring system based on the deep fusion neural network pedestrian re-identification technology is provided. The present invention starts from two aspects of manual feature and depth feature, thereby effectively improving the matching rate. The technical means adopted in the present invention are as follows:

As shown in Figure 1, an intelligent monitoring method based on deep fusion neural network pedestrian re-identification technology includes the following steps:

S1. Obtain images frame by frame from the surveillance video;

S2. Preprocess the image for color enhancement, and then send it to the local maximum structure for extracting hand-crafted features and the neural network for extracting deep features;

S3A, extracting traditional hand-crafted features, specifically: using local maximum pooling algorithm to extract feature vectors, using preset-scale squares to traverse the entire picture with a certain step size, and extracting color features and textures from the images in each square feature, and then perform dimensionality reduction on the entire feature vector;

S3B, image extraction and Gaussian pooling using deep residual convolutional neural networks;

S4. Integrate the traditional manual features after dimensionality reduction and the deep features extracted by the well-trained neural network to complete the identification of the target pedestrian;

S5. Track the identified pedestrians, and repeat the above process for re-identification when the tracking confidence is reduced to a certain threshold, and at the same time, display the identified results in a preset style.

Further, the preprocessing of the color enhancement is specifically: using a multi-scale image enhancement algorithm to perform color enhancement on the obtained image by using Gaussian parameters of three different scales respectively.

Further, in the extraction of the traditional manual features, the extraction of the color features includes: converting the preprocessed image from RGB to the HSV color space, and obtaining the color histogram of the image on this basis; In order to extract using the scale-invariant ternary mode coding technology, the dimensionality reduction process is specifically: using Gaussian pooling, dicing the data with a preset size, and then obtaining the origin moment and the central moment of the diced data respectively to represent The data in the block can be used to reduce the dimension of the entire feature vector.

Further, the S3B is specifically: the dynamic fuzzy neural network randomly cuts off the preprocessed image, then re-adjusts it to a preset size, and transmits it to the deep neural network ResNet50, and the input image sequentially passes through the convolution layer, the local normalization layer. The normalization layer, the modified linear activation function, and the maximum pooling process, and then enter the convolution module for dimensionality reduction processing. After each dimensionality reduction module, the size of the image will be reduced by 2*2.

Further, the S4 is specifically: using a fully connected fusion layer of 4096 dimensions to connect the outputs of the two vectors respectively, then performing operations such as regularization, batch normalization, nonlinear activation, etc., and finally connecting to the classifier, using the softmax function. Accept the result and give the predicted value to complete the identification of the target pedestrian.

Further, the step S4 specifically includes the following steps:

S41. Integrate the traditional handcrafted features after dimensionality reduction with the deep features extracted by the well-trained neural network, and the total features after fusion are expressed as:

F _z = [F _h , F _d ]

Among them, F _d is the deep feature vector obtained from the ResNet50 model, and F _h is the traditional handcrafted feature after Gaussian pooling dimension reduction;

S42. Connect the traditional handcrafted features to the splicing layer, and then connect the batch normalization and nonlinear activation functions in turn, and then perform the connection operation of the classifier, which is specifically expressed as:

where h( ) represents the activation function, and b _f represent the weight coefficient and transfer vector of the connection layer,

μ _z and σ _z represent the characteristics of mean and variance, and then connect to the classifier, where the softmax structure is used as a layer of classification, specifically:

Among them, x represents the vector formed by the output of the previous layer of neural network, θ is the formal parameter vector obtained after training, and the existence of the denominator is to normalize the predicted output.

Using the cross-entropy loss function, denote the cross-entropy loss as J, then we have

Among them, p _k represents the softmax output value corresponding to the kth output (which can also be understood as the probability that the neural network predicts the k result).

Further, the training of the model is based on the gradient freezing training method, which specifically includes the following steps:

Train the ResNet50 deep residual neural network and the corresponding connected 2048-dimensional fully connected layer classifier on the dataset;

After training to the convergence of the model, import the trained ResNet50 parameters into the deep fusion neural network, reconnect the new fully connected classification and softmax network, and then train on the same dataset; The network parameters for extracting deep features initialized by the training model are set to a low learning rate, focusing on training the weight parameters of the fusion link and the classifier, and finally completing the training of the entire fusion network.

Further, the data set is specifically the Market1501 pedestrian re-identification data set.

The present invention has the following advantages:

1. Compared with the prior art, the recognition accuracy of the present invention is greatly improved. The accuracy of the deep fusion neural network after combining the deep feature ResNet50 with the hand-crafted feature LOMO is improved by 30% compared with the simple use of ResNet50. The re-identification algorithm increased the recognition success rate to 81.74%, making the technology fully practical.

2. The present invention uses the gradient freezing training method to reduce the time by 10 intervals, so that the disadvantage of the fusion neural network in training time is alleviated, and the convergence speed is improved on the basis of ensuring the accuracy.

3. The present invention only needs to install a mouse-operable program on the basis of manual monitoring, and the user does not need to worry about the algorithm details involved in the technology, and the pedestrian can be automatically identified and tracked, so the use is relatively simple.

4. The degree of automation in the security field has been improved. Traditional security monitoring requires a lot of manpower for visual inspection, and some criminals even have to keep their eyes open. This technology can automate this process. All processes are completed automatically, and details will not be lost due to personnel fatigue.

Based on the above reasons, the present invention can be widely promoted in the field of intelligent monitoring technology.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a flowchart of an intelligent monitoring method based on the deep fusion neural network pedestrian re-identification technology of the present invention.

FIG. 2 is a specific design flow chart of LOMO and ResNet50 according to an embodiment of the present invention.

FIG. 3 is a detailed detailed diagram of deep fusion according to an embodiment of the present invention.

FIG. 4 is a schematic diagram showing the advantages of the gradient freezing training method according to the embodiment of the present invention.

FIG. 5 is a comparison diagram of the embodiment of the present invention and detection by other methods.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

As shown in FIG. 1 , this embodiment provides an intelligent monitoring method based on the deep fusion neural network pedestrian re-identification technology, including the following steps:

S1. Obtain images frame by frame from the surveillance video, specifically, read the video stream in the camera and obtain information frame by frame; use the Python package of opencv to read the video stream to obtain the image tensor of each frame, Data is reserved for use.

S2. Perform color enhancement preprocessing on the image, and then send it to a neural network for extracting local maximum structures of manual features and extracting depth features; the color enhancement preprocessing is specifically: using a multi-scale image enhancement algorithm The resulting images are color-enhanced using three Gaussian parameters at different scales, respectively.

S3A, extracting traditional hand-crafted features, specifically: using local maximum pooling algorithm to extract feature vectors, using preset-scale squares to traverse the entire picture with a certain step size, and extracting color features and textures from the images in each square feature, and then perform dimensionality reduction processing on the entire feature vector; in the extraction of the traditional manual features, the extraction of color features includes: converting the preprocessed image from RGB to HSV color space, and obtaining the Color histogram; the extraction of texture features is specifically extracted by using a scale-invariant ternary mode coding technology, and the dimensionality reduction process is specifically: using Gaussian pooling to cut the data into pieces with a preset size, and then obtain the pieces separately. The origin moment and central moment of the data are used to represent the data in the block, so as to reduce the dimension of the entire feature vector.

S3B, using a deep residual convolutional neural network to extract and Gaussian pooling; specifically, the dynamic fuzzy neural network randomly cuts the preprocessed image, and then rescales it to a preset size, and transmits it to the deep neural network In ResNet50, the input image goes through the convolution layer, local normalization layer, modified linear activation function, and maximum pooling in sequence, and then enters the convolution module for dimensionality reduction processing. After each dimensionality reduction module, the size of the image will be reduced. 2*2.

As shown in Figure 2 and Figure 3, S4, fuse the traditional manual features after dimensionality reduction with the deep features extracted by the well-trained neural network to complete the identification of the target pedestrian; specifically, use 4096-dimensional full connection fusion The layer connects the outputs of the two vectors respectively, and then performs regularization, batch normalization, nonlinear activation, etc., and finally connects to the classifier, and uses the softmax function to accept the results and give the predicted value, thereby completing the target pedestrian recognition.

S5. Track the identified pedestrians, and repeat the above process for re-identification when the tracking confidence is reduced to a certain threshold. At the same time, the identified results are marked and displayed in a preset style. Specifically, the SSD algorithm is used to identify the pedestrians. The obtained pedestrian is tracked, and when the tracking confidence is reduced to a certain threshold, the above process is used again for re-identification, and the recognition result in this process will be framed by a rectangle. Use pyQt5 to write the user interface and convert it into an operating system executable file (such as an exe file), which is convenient for users to use.

The step S4 specifically includes the following steps:

S41. Integrate the traditional handcrafted features after dimensionality reduction with the deep features extracted by the well-trained neural network, and the total features after fusion are expressed as:

F _z = [F _h , F _d ]

Among them, F _d is the deep feature vector obtained from the ResNet50 model, and F _h is the traditional handcrafted feature after Gaussian pooling dimension reduction;

S42. Connect the traditional handcrafted features to the splicing layer, and then connect the batch normalization and nonlinear activation functions in turn, and then perform the connection operation of the classifier, which is specifically expressed as:

where h( ) represents the activation function, and b _f represent the weight coefficient and transfer vector of the connection layer,

μ _z and σ _z represent the characteristics of mean and variance, and then connect to the classifier, where the softmax structure is used as a layer of classification, specifically:

Among them, x represents the vector formed by the output of the previous layer of neural network, θ is the formal parameter vector obtained after training, and the existence of the denominator is to normalize the predicted output.

Using the cross-entropy loss function, denote the cross-entropy loss as J, then we have

Among them, p _k represents the softmax output value corresponding to the kth output (which can also be understood as the probability that the neural network predicts the k result).

As a preferred embodiment, the training of the model is based on the gradient freezing training method, which specifically includes the following steps:

Train the ResNet50 deep residual neural network and the corresponding connected 2048-dimensional fully connected layer classifier on the dataset;

After training to the convergence of the model, import the trained ResNet50 parameters into the deep fusion neural network, reconnect the new fully connected classification and softmax network, and then train on the same dataset; The network parameters for extracting deep features initialized by the training model are set to a low learning rate, focusing on training the weight parameters of the fusion link and the classifier, and finally completing the training of the entire fusion network.

The dataset is specifically the Market1501 pedestrian re-identification dataset. The model in this paper uses the Market1501 dataset for training, testing and validation. The Market1501 dataset is a pedestrian re-identification dataset released by Liang Zheng in 2015, and it is currently the largest dataset. The dataset is set in Tsinghua University supermarket. There are 38,195 pedestrian images captured by 6 different cameras, involving a total of 1,501 pedestrians. Among them, there are 12,936 training images from 751 pedestrians, and 19,732 test images from other pedestrians. of 750 pedestrians.

Example 1:

S3B. For an original image with a size of 128*64, DFNN first randomly cuts it off, then re-adjusts the size to 256*128 and passes it into the deep neural network ResNet50. The input image is successively processed by convolution layer, local normalization layer, modified linear activation function, max pooling, etc., and then enters 16 convolution modules. Although the features extracted by these convolution modules are becoming more and more complex, DFFN maintains and reduces the dimension of the features at the same time, and the convolution module (convblock) is used to reduce the dimension of the data. There are 3 other dimensionality reduction modules following the same. After each dimensionality reduction module, the size of the image will be reduced by 2*2. Therefore, before the input and output are connected through the adder, the input needs to be reduced in size by the convolution layer to ensure normal addition.

S3A. After the image is preprocessed by the multi-scale image enhancement algorithm, a part of the image will be represented as a local scale-invariant background differential encoding, and the other part will be converted into the color feature of the HSV space, and then the LOMO algorithm is used to abstract the feature description of the two. sub, get a handcrafted feature vector of 70770 dimensions.

The manual feature dimension is 70770. If it is directly connected to the fully connected layer, the amount of parameters will be huge. Even if traditional dimensionality reduction methods are adopted, it is difficult to ensure that the entire traditional features can be accurately represented. Therefore, it is necessary to have a reasonably reliable representation of the data. Considering that the LOMO algorithm used in extracting traditional features uses the means of using the largest component to characterize the distribution of all the components, the maximum pooling is no longer used here, but the average length is used for modules of a certain length. pooling. In order to further and more reliably ensure that the main information is not lost, this paper assumes that the classification of the feature data conforms to the one-dimensional Gaussian distribution, so two data of expectation and variance are used to represent the local distribution, 70 data are divided into 1 group, and its Gaussian parameters are obtained. , and then connected together, a 2022-dimensional feature vector is obtained, and dimensionality reduction is completed while ensuring the accuracy as much as possible.

S4. Integrate the traditional handcrafted features after dimensionality reduction with the deep features extracted by the well-trained neural network, and connect them to the overall splicing layer of 2022+2048=4070 dimensions, realizing strong confrontation and strong coupling of the two features. .

After training based on the gradient freezing training method, the final model is obtained and tested to evaluate its superiority. Specifically, the commonly used evaluation method for person re-identification is CMC (Accumulative Match Characteristic). This evaluation method considers that the task of person re-identification is the task of sorting the similarity between the target image and the image to be selected. CMC(k) is a picture with a similarity degree of k in the re-identification task. The frequency of the computer getting the correct conclusion in the first recognition is called Rank1, and Rank1 can be used to measure the accuracy of pedestrian re-identification.

In multi-label image classification, the average accuracy used in single-label classification cannot be simply used, but the mAP (mean Average Precision) evaluation method is used. The evaluation method is to calculate the average precision rate (AP, Average Precision) by calculating the precision rate and recall rate, and then calculate the mAP value. Generally, the mAP value is slightly lower than the Rank1 value.

As shown in Figure 5, in each row, the image on the left is the sample, and all the recognition results are sorted in descending order, with red boxes representing false matches and blue boxes representing correct matches. That is, in the DFNN of the present application, all except the fourth one indicate a correct match. And SOMAnet only 2 and 7 are correct, others are wrong. For the second set of examples, in the DFNN of this application, the 2nd, 4th, 7th, and 10th are correct, and the others are incorrect, while SOMAnet only has the 6th correct identification, and the others are all incorrect. The 751 pedestrian IDs given in the Market1501 dataset are used for training and a total of 12936 images are used for training, and then an additional 750 pedestrian IDs are used for testing with a total of 19732 images.

During the training process, the specific process of the training method of gradient freezing is as follows:

1) Train the ResNet50 deep residual neural network and the corresponding connected 2048-dimensional fully connected layer classifier on the Market1501 dataset;

2) Import the trained ResNet50 parameters into the deep fusion neural network, reconnect the new fully connected classification and softmax network, and then train on the same dataset.

As shown in Figure 4, when only the ResNet50 network is used for training, the learning rate of the ResNet50 network and the learning rate of the fully connected classifier of the deep fusion neural network are both set to 0.01. When training deep neural networks, the learning rate of ResNet50 is set to a low 0.001. The optimization method in this model is a mini-batch gradient descent method based on momentum correction, the momentum value is 0.9, and the weight delay is 0.0005. After the ResNet was trained, it was tested, and after the deep neural network was trained, a second test was performed. Experiments show that the Rank1 obtained by simply using ResNet50 is 51%, while the Rank1 obtained by using the deep fusion neural network test is 82%, which is a 31% improvement in accuracy.

The performance of the algorithm is verified by comparing the deep fusion neural network algorithm (DFNN) of the present application with the pedestrian re-identification algorithm in recent years. The test results are shown in Table 1.

Table 1

As shown in Table 1, the Rank1 of DFNN is 81.74%, which is higher than any other algorithms in the table. The DFNN model outperforms classic algorithms such as LOMO+XQDA because deep features are extracted using ResNet50. Also, DFNN models are better than models like SpindleNet or Triplet CNN because handcrafted features are imported into the model to constrain the deep neural network. It can be seen that the recognition accuracy has been significantly improved after deep fusion.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements to some or all of the technical features thereof, However, these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims (8)

1. a kind of intelligent control method based on depth integration neural network pedestrian weight identification technology, which is characterized in that including such as Lower step:

S1, image is obtained frame by frame from monitor video；

S2, the pretreatment that color enhancement is carried out to image send it to the local maximum structure for extracting manual feature later In the neural network of extraction depth characteristic；

S3A, traditional-handwork feature is extracted, specifically: feature vector is extracted using local maxima pond algorithm, uses default scale Grid whole picture is traversed with a fixed step size, and to the image zooming-out color characteristic and textural characteristics in each grid, then Dimension-reduction treatment is carried out to entire feature vector；

S3B, image is extracted using depth residual error convolutional neural networks and Gauss pond；

S4, the traditional-handwork feature after dimensionality reduction and the depth characteristic for training intact neural network to extract are merged, Complete the identification to target pedestrian；

S5, the pedestrian recognized is tracked, and repeat when tracking creditability is reduced to some threshold value above-mentioned process into It is capable to identify again, meanwhile, the mark for carrying out the preset style to the result of identification is shown.

2. the intelligent control method according to claim 1 based on depth integration neural network pedestrian weight identification technology, It is characterized in that, the pretreatment of the color enhancement specifically: use three kinds of different rulers respectively using multi-scale image enhancing algorithm The Gaussian parameter of degree carries out color enhancement to obtained image.

3. the intelligent control method according to claim 2 based on depth integration neural network pedestrian weight identification technology, Be characterized in that, in the extraction of the traditional-handwork feature, the extraction of color characteristic include: will carry out pretreated image by RGB is converted into hsv color space, and obtains the color histogram of image on this basis；The extraction of textural characteristics is specially to make It is extracted with three value pattern-coding technologies of Scale invariant, the dimension-reduction treatment specifically: using Gauss pond, by data with default Size carries out stripping and slicing, and the moment of the orign and central moment for obtaining stripping and slicing data respectively later are to indicate data in block, to realize to whole The dimensionality reduction of a feature vector.

4. the intelligent control method according to claim 2 based on depth integration neural network pedestrian weight identification technology, It is characterized in that, the S3B specifically: dynamic fuzzy neural network cuts off pretreated image at random, readjusts later It to pre-set dimension, and is passed in deep neural network ResNet50, input picture passes sequentially through convolutional layer, part normalization The processing of layer, the linear activation primitive of amendment, maximum pond enters convolution module later and carries out dimension-reduction treatment, every by a drop Module is tieed up, the size of image can all reduce 2*2.

5. the intelligence prison according to any one of claims 1 to 4 based on depth integration neural network pedestrian weight identification technology Prosecutor method, which is characterized in that the S4 specifically: be separately connected the defeated of two vectors using the full connection fused layer of 4096 dimensions Out, the operations such as regularization, batch normalization, nonlinear activation are carried out later, classifier are finally coupled to, using softmax letter Number receives result and provides predicted value, to complete the identification to target pedestrian.

6. the intelligent control method according to claim 5 based on depth integration neural network pedestrian weight identification technology, It is characterized in that, the step S4 specifically comprises the following steps:

S41, the traditional-handwork feature after dimensionality reduction and the depth characteristic for training intact neural network to extract are merged, Total character representation after fusion are as follows:

F_z=[F_h, F_d]

Wherein, F_dFor the depth characteristic vector obtained in the ResNet50 model, F_hFor by the tradition after the dimensionality reduction of Gauss pond Manual feature；

S42, traditional-handwork feature is connected in splicing layer, is carried out after being sequentially connected batch normalization, nonlinear activation function The attended operation of classifier, is embodied as:

H () represents activation primitive in formula,And b_fIndicate the weight coefficient and migration vector of articulamentum,

μ_zAnd σ_zThe characteristic for representing mean value and variance, is connected to classifier later, is used herein as softmax structure as classification One layer, specifically:

Wherein, x indicates that the vector that the output of one layer of neural network is constituted, θ are the parameter vector obtained by training, denominator In the presence of be in order to prediction output be normalized.

Using cross entropy loss function, it is J that note, which intersects entropy loss, then has

Wherein, p_kIndicate that (also be understood as neural network prediction is k result to the corresponding softmax output valve of k-th of output Probability).

7. the intelligent control method according to claim 1 based on depth integration neural network pedestrian weight identification technology, It is characterized in that, the training of model is freezed coaching method based on gradient and is trained, and specifically comprises the following steps:

Training ResNet50 depth residual error neural network and the full articulamentum classifier of 2048 dimensions being correspondingly connected on data set；

After training to model convergence, the parameter of trained ResNet50 is imported in depth integration neural network, is connected again New full link sort and softmax network are connect, then is trained on identical data set；In training, will utilize The network parameter of the extraction depth characteristic that pre-training model initialization is crossed is arranged lower learning rate, emphasis training fusion link and The weight parameter of classifier is finally completed the training to entire converged network.

8. the intelligent control method according to claim 7 based on depth integration neural network pedestrian weight identification technology, It is characterized in that, the data set is specially that Market1501 pedestrian identifies data set again.