CN111967433A - Action identification method based on self-supervision learning network - Google Patents

Abstract

The invention provides an action recognition method based on a self-supervised learning network. The first step is to use OpenPose to extract human skeleton information in a video stream, make the human skeleton information into a positive and negative sample data set, and use the data set to train actions The classification model ResNet‑56, the preliminarily trained action classification model is used to preliminarily judge the input action and further self-supervised training; while training the action classifier, the deep learning model YOLOv4 is used to detect the object information in the video stream, and the detected The object information is labeled with positive and negative labels and sent to the action classification model ResNet‑56 for self-supervised learning training. The action classification model after self-supervision has higher detection accuracy and reliability.

Description

An Action Recognition Method Based on Self-Supervised Learning Network

technical field

The invention belongs to the field of computer intelligent learning, and more specifically relates to a self-supervised human action recognition method combining OpenPose and YOLOv4.

Background technique

Computer vision technology is a branch of artificial intelligence, mainly through the processing of visual image information, to achieve the purpose of identifying and understanding image content. Intelligent video surveillance relies on the powerful computing power of computers. Relying on computer vision technology, key information is extracted from surveillance images or videos and quickly analyzed, and the analysis results are fed back to the surveillance system for processing, so as to achieve the purpose of intelligently identifying, understanding and processing surveillance videos.

The analysis of human key points in human action recognition is one of the most important, and is also collectively referred to as human pose estimation. At present, there are many technologies that can read the information of the key points of the human body, such as depth somatosensory cameras, such as Kinect, etc. After people put on the device, they can collect real-time information on the key points of the human body. The disadvantage is that the number of people that can be collected at one time is too small, and the price is relatively expensive. In contrast, it is much more convenient to use OpenPose to collect information on key points of human skeleton, and it can collect multiple people at one time, and it does not require too expensive price.

OpenPose is the current mainstream model algorithm for human pose estimation. Most actions can use OpenPose to collect key point information of human skeletons and train them to recognize actions in video streams. However, compared to Kinect's high accuracy, OpenPose is slightly insufficient in detection accuracy; because most models are based on monotonic OpenPose for action recognition, for example, an OpenPose-based eating behavior recognition method CN201911150648.6, OpenPose-based fencing action acquisition method and computer The storage medium CN201810338998.4, etc., the above two patents are based on the model training of action recognition only based on OpenPose to extract information

In order to improve the detection accuracy of OpenPose in practical applications, this patent provides an action recognition method based on a self-supervised learning network. On the basis of the traditional monotonous OpenPose action recognition, YOLOv4 target detection technology is added to detect the detected objects. Item information is classified into positive and negative samples and then self-supervised model training is performed to improve the accuracy of action recognition.

SUMMARY OF THE INVENTION

The present invention proposes an action recognition method based on a self-supervised learning network based on the low accuracy of the current monotonous OpenPose extracting human skeleton information and then using it for recognition.

The specific content of the present invention is as follows:

Step S1: Collect the feature information of all human skeleton joint points: OpenPose predicts the position reliability S of the human body in the picture through the feedforward network, and at the same time predicts the affinity vector field L of the part (the relationship between each joint of the human skeleton), the set S = ( S1, S2, SJ) J means that each skeleton joint point has J body part position information maps. The set L(L1, L2, LC) has C site affinity vector fields for each limb. After the sets J and L are obtained, the Greedy algorithm is used to find out the joint point information of the human skeleton. The extracted skeleton feature information is made into a positive and negative sample data set of the desired action to be recognized. As shown in Figure 1.

Step S2: Use the positive and negative sample data set made in Step S1.1 as input, and train an action classifier model based on ResNet-56. In the ResNet-56 model, the Softmax loss is used for classification, and the objective function is as follows:

N is the number of samples, λ is the regression loss weight, i is the index of the proposed window in the batch, and P is the predicted probability.

The input of the neural network is x, the expected output is H(x), the residual network F(x)=H(x)-x, and the ResNet-56 model is trained until F(x) is close to 0. The model training ends. The training data set is passed into the model trained in step S2.1, and the action recognition result of the video stream training set is obtained.

Step S3.1: Establishment of Object Data Set: Collect and classify the object data set other than human body included in the action to be detected. For example: set the action to be recognized as A, then the items related to action A are set as B1, B2... Arranged in order, the information of items B1 and B2 can be collected from a variety of different videos or pictures, and made into Standard VOC dataset.

Step S3.2: Input the item VOC data set prepared in the above step S3.1 into the standard YOLOv4 model for training. The convolution cascade structure is used in the YOLOv4 model, and the input layer of the network is designed to be 448*448.

The cost function of logistic regression is:

where h _θ is the sigmoid function as the activation function in the network. When the accuracy rate of item detection reaches more than 95%, the model is trained and can be used to detect and record the information of items and pedestrians in the video stream.

Step S4.1: Divide the item information labels identified in step S3 into positive and negative labels. If the item information and action information are similar, they are regarded as positive samples, and if they are different, they are regarded as negative samples. The similarity index between two functional similarities can be measured according to the formula score(f(x), f(x ⁺ )) >> score(f(x), f(x ^- )).

Construct a softmax classifier to correctly classify positive and negative samples:

Step S4.2: After dividing the item information into positive and negative samples, the positive and negative samples will be passed to ResNet-56 in step S3 for self-supervised training: when there is a lot of positive sample information, the final identified action The confidence will increase, on the contrary, if the number of negative samples is large, the confidence of the predicted change will decrease. After continuous input of positive and negative labels made from item information to train the model, the recognition rate of the action classifier model will gradually increase until it reaches an ideal level.

Description of drawings

Figure 1 shows the process of extracting human skeleton information from OpenPose;

Figure 2 shows the detection of objects around the human body;

Figure 3 is a diagram of the self-supervised training action classification model

Detailed ways

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. It should be understood that the described embodiments are only Some, but not all, embodiments of the present invention should therefore not be construed as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

Example 1: (take a person climbing as an example)

Step S1: Collect the feature information of human skeleton joint points related to climbing when people are climbing walls, climbing ladders, etc.: OpenPose predicts the position reliability S of the key nodes of the climbing human body through a feedforward network, and at the same time predicts the position of the body. The affinity vector field L (the relationship of each joint of the human skeleton), the set S=(S1, S2, SJ)J indicates that each skeleton joint point has J body part position information maps. The set L(L1, L2, LC) has C site affinity vector fields for each limb. After the sets J and L are obtained, the Greedy algorithm is used to find out the joint point information of the human skeleton. The extracted climbing skeleton feature information is made into a positive and negative sample data set for the required recognition action. Skeletal keypoint predictions are shown in Figure 1.

Step S2: Use the climbing positive and negative sample data set made in Step S1 as input, and train a ResNet-56-based action classifier model. In the ResNet-56 model shown in Figure 2, the Softmax loss is used for classification, and the objective function is as follows:

N is the number of samples, λ is the regression loss weight, i is the index of the proposed window in the batch, and P is the predicted probability.

The input of the neural network is x, the expected output is H(x), the residual network F(x)=H(x)-x, and the ResNet-56 model is trained until F(x) is close to 0. The model training ends. The action model at this time can be used to identify the climbing behavior in the video stream, but the accuracy needs to be improved.

Step S3.1: In this patent, the object detection of YOLOv4 is a combination of official weights and self-built dataset weights. Therefore, the first step is to establish a dataset of climbing-related objects: climbing is to measure the objects such as walls, doors, ladders, telephone poles, etc., classify and collect the pictures of these objects, and make them into standard VOC data set.

Step S3.2: Input the item VOC data set prepared in the above step S3.1 into the standard YOLOv4 model for training, as shown in Figure 3.

In the model YOLOv4 model used in this patent, there are multiple convolution layers and one fully connected layer, and the convolution cascade structure is widely used. The input layer of the network is designed to be 448*448. The convolution cascade structure is used for image feature extraction, and the fully connected layer is used to predict class probabilities and bounding boxes. The cost function of logistic regression in the model is:

where h _θ is the sigmoid function as the activation function in the network. When the accuracy rate of item detection reaches more than 95%, the model is trained and can be used to detect and record the information of items and pedestrians in the video stream.

Step S4.1: Divide the item information tags identified in step S3 into positive and negative tags, and items related to climbing such as doors, ladders, telephone poles, etc. are regarded as positive samples, and mobile phones, earphones, food, etc. are regarded as negative samples Samples, and the rest of the recognized objects are classified according to the construction of a softmax classifier to correctly classify positive and negative samples:

The denominator term includes a positive number and n-1 negative samples;

Step S4.2: After the item information is divided into positive and negative samples, the two types of item information will be passed into the ResNet-56 action classifier model, and the accuracy of action recognition will be improved under self-supervised learning.

After the model is trained, the trained model can be used to detect whether there is climbing behavior in the video stream, and the accuracy will be greatly improved compared to the model without self-supervision.

Claims (4)

1. An action recognition method based on an automatic supervision learning network is characterized by comprising the following steps:

step S1: extracting human skeleton characteristic information of a target human body image in the video stream through OpenPose;

step S2: constructing a neural network model, and training the neural network model by taking the human skeleton information extracted by OpenPose as input to obtain an action recognition result corresponding to the action training data set;

step S3: detecting object information in the video stream by using a YOLOv4 model;

step S4: labeling the identified object information according to the detection action, and retraining the neural network based on the article information label and the action identification result corresponding to the pre-training data to obtain a retrained neural network model;

step S5: and performing motion recognition on the video stream by using the retrained model, and outputting a motion prediction result in the video stream.

2. The method for constructing a neural network model according to step S2 in claim 1, comprising the following steps:

step S2.1: extracting human body skeleton characteristic information by using OpenPose, predicting human body part confidence coefficient S in a picture through a feedforward network, and simultaneously predicting an affinity vector field L (relation of each joint of a human body) of the part, wherein the set S is (S1, S2, SJ) J and shows that each bone joint point has J body part confidence maps; set L (L1, L2, LC) has C site affinity vector fields per limb; after the sets J and L are obtained, the information of the human skeleton joint points is found out by using a Greedy algorithm; making the extracted skeleton characteristic information into a positive and negative sample data set of the motion to be recognized;

step S2.2: taking the positive and negative sample data sets produced in the step S1.1 as input, and training a ResNet-56-based action classifier model; in the ResNet-56 model, classification is performed using Softmax loss, and the objective function is as follows:

n is the number of samples, λ is the regression loss weight, i is the index of the proposed window in the batch, and P is the prediction probability;

the input of the neural network is x, the expected output is H (x), the residual network F (x) ═ H (x) — x, and the model training is finished until F (x) is close to 0;

step S2.3: and (4) transmitting the training data set into the model trained in the step (S2.1) to obtain the action recognition result of the video stream training set.

3. The method for the YOLOv4 model detection of claim 1, step S3, comprising the following steps:

step S3.1: establishing an object data set: collecting and classifying object data sets except human bodies contained in the motion to be detected; such as: setting the action to be identified as A, setting the articles related to the action A as B1 and B2 … … to be sequentially arranged, and making the articles into a standard VOC data set;

step S3.2: inputting the article VOC data set manufactured in the step S3.1 into a standard YOLOv4 model for training; the input layer of the network is designed to 448 x 448, a convolution cascade structure is used for carrying out feature extraction on the image, and the full-connection layer is used for predicting class probability and frames;

the cost function of logistic regression is:

wherein h is_θIs a sigmoid function, which is used as an activation function in the network; when the accuracy rate of the article detection reaches more than 95%, the model training is finished;

step S3.3: and detecting the incoming video stream by using the Yolov4 model trained in the step S3.2, and recording and storing object information when detecting that the motion comprises objects B1 and B2 … ….

Labeling the identified object information according to the detection action, and retraining the neural network based on the article information label and the action identification result corresponding to the pre-training data to obtain a retrained neural network model;

4. the method for tagging identified object information according to detection actions of claim 1, step S4, comprising the steps of:

dividing the item information labels identified in step S3 into positive and negative labels, regarding the item information labels as positive samples if the item information and the motion information are similar, regarding the item information labels as negative samples if the item information and the motion information are different, and determining the item information labels as positive samples according to a formula score (f (x), f (x))(x⁺))＞＞score(f(x),f(x^-) To measure a similarity index between two functionally similar; constructing a softmax classifier to correctly classify positive and negative samples:

after the article information is divided into positive and negative samples, the positive and negative samples are transmitted to the ResNet-56 for self-supervision training in step S3: in the transmitted article information, when the number of positive samples is large, the action confidence coefficient finally output by the action classification model ResNet-56 is increased, on the contrary, when the number of negative samples is large, the confidence coefficient of the output action is reduced, and the identification rate of the action classification model to the set action reaches the ideal level through multiple input training of the object information.

Images