WO2017133009A1

WO2017133009A1 - Method for positioning human joint using depth image of convolutional neural network

Info

Publication number: WO2017133009A1
Application number: PCT/CN2016/073695
Authority: WO
Inventors: 陈勇杰; 林倞; 王青; 王可泽
Original assignee: 广州新节奏智能科技有限公司
Priority date: 2016-02-04
Filing date: 2016-02-05
Publication date: 2017-08-10
Also published as: CN105787439B; CN105787439A

Abstract

A method for positioning a human joint using a depth image of a convolutional neural network comprises a training process and a recognition process. The training process comprises the following steps: 1) inputting a training sample; 2) initializing a deep convolutional neural network and a parameter thereof, the parameter including a weight and an offset of each edge; and 3) and obtaining, using a forward algorithm and a backward algorithm, and by means of learning using the training sample, a parameter of a constructed convolutional neural network. The recognition process comprises the following steps: 4) inputting a test sample; 5) and performing regression on the input test sample by using the trained convolutional neural network, to obtain the position of a human joint therein. By using a deep convolutional neural network and big data, the present invention can overcome multiple challenges such as blocking and noise, and achieve a high accuracy rate. In addition, by means of parallel computing, the present invention enables accurate positioning of a human joint in real time.

Description

Depth image human joint positioning method based on convolutional neural network

Technical field

The invention relates to the fields of computer vision, pattern recognition and human-computer interaction, in particular to a depth image human joint positioning method based on a convolutional neural network.

Background technique

Body pose estimation and motion capture are an important research direction in the field of computer vision. Its applications include home entertainment, human-computer interaction, motion recognition, security systems, remote monitoring, intelligent monitoring, and even patient health care. However, performing pose estimation in normal RGB images or video is a very challenging task. Because the natural environment factors such as color, illumination, and occlusion cannot be robust, and the degree of freedom of the human body posture and the observation angle are different, this problem is naturally more difficult.

A depth image is a two-dimensional grayscale image, but unlike a conventional grayscale image, the grayscale value of each pixel of the depth image reflects the millimeter distance of the object corresponding to the point in the real space from the camera. Compared with traditional color two-dimensional images, depth images are not affected by environmental factors such as illumination and shadows, and can effectively express the geometric structure information of objects in the real world. Therefore, research and application in computer vision and human-computer interaction The field has an important place. With the popularity of inexpensive depth cameras, research and application based on depth images has broad market and bright prospects.

Depth image The human joint positioning method refers to determining the position of a joint point of a human body in a depth image containing a person or a human body. Here, the joint points of the human body refer to the bone joints of the hands, elbows, wrists, shoulders, head, ankles, knees, buttocks, and the like. Determining the position of the joint points of the human body allows us to analyze the bone structure of the human body, and then simply judge the posture of the human body, and then recognize the movements and behaviors of the human body, which is of great significance for human-computer interaction entertainment and computer vision.

Depth image Human joint positioning has the following difficulties:

1) The depth image has a defect of low resolution and large mechanical noise. The use of hand-designed features to locate human joints does not yield good results.

2) The positioning of the human joint is very difficult to achieve accurate and robust due to the different angles of the camera, the distance between the camera and the character, and the degree of occlusion of the character itself.

3) There is a constraint between the bones and joints of the human body: when the limbs of the person move, between the limbs and the limbs There are constraints such as linkage and braking, and it is very difficult to learn and express such linkage constraints.

4) The positioning and tracking of the human bone joints is difficult to fuse. At present, the position and posture of the character are directed to a single depth image because it is difficult to express the motion consistency of the bone joint in the time domain.

The above difficulties make the goal of achieving accurate and robust human joint positioning still have a certain gap. Therefore, it is necessary to solve the above difficulties.

Summary of the invention

The main object of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a depth image human joint positioning method based on a convolutional neural network.

In order to achieve the above object, the present invention adopts the following technical solutions:

Compared with the prior art, the present invention has the following advantages and beneficial effects:

1. The present invention relies on the currently well-recognized and promising technology - deep learning, which is to construct a deep convolutional neural network from a large number of training samples (these training samples contain various angles of camera placement, The camera automatically learns the effective features of the various distances of the person and the various occlusions of the character itself, without relying on the features that people manually design. Through the effective features learned, the joint position of the human body is directly returned.

2. The convolutional neural network of the present invention uses a three-dimensional convolutional layer to express the motion consistency of the bone joint in the time domain; at the top layer, a loss function based on the bone relationship tree is used to express the linkage and braking between the human bone joints. Constraint relationship.

DRAWINGS

Figure 1 is a flow chart of the present invention;

2 is a diagram of a convolutional neural network architecture of the present invention;

Figure 3 is a schematic view of the human bone joint points of the present invention.

detailed description

The present invention will be further described in detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

Example

As shown in FIG. 1, the present invention is based on a depth image human joint positioning method based on a convolutional neural network, including a training process and a recognition process;

The steps of the training process are as follows:

1) Input training samples;

2) Initializing a deep convolutional neural network and its parameters, the parameters including the weights and offsets of each layer edge;

3) Using the forward algorithm and the backward algorithm, using the training samples to learn the parameters of the constructed convolutional neural network;

The steps in the identification process are as follows:

4) Enter the test sample;

5) Using the trained convolutional neural network to return the position of the human joint point to the input test sample.

The technical solution of the present invention is further elaborated below in conjunction with a specific technical solution:

1. Convolutional neural network architecture

The present invention proposes a deep level full convolutional neural network (shown in Figure 2) for estimating body pose joint points in depth images and depth image sequences. The network consists of 9 convolution layers connected in series with a downsampling layer and a normalized layer interspersed. The following will introduce each one:

Two-dimensional convolutional layer: Convolutional layer refers to the convolution of an input image or feature in a two-dimensional space, which can extract some important features. Assume that the width and height of the input image are w and h, respectively, and the three-dimensional convolution kernel The size is w' x h' x m', where w', h', m' represent the width, height and number of channels, respectively. A feature map can be obtained after convolution. Where the value at the position of the feature map (x, y) can be expressed as

Where p _{(x+i)(y+j)(s+k)} represents the pixel value of the (x+i,y+j) position in the (s+k)th frame of the input, and ω _ijk represents the parameter of the convolution kernel , b denotes the offset associated with the feature map. Therefore, we can get 1 feature map, the size of each feature map is (w-w'+1, h-h'+1). Since a single convolution kernel can only extract one type of feature, we introduce multiple convolution kernels in each convolution layer to extract many different features.

Downsampling layer: Downsampling We use the max-pooling operation. This operation refers to the process of downsampling the feature map according to a certain strategy (selecting the maximum value). This is an effective process that is widely used to extract features that preserve shape and offset invariance. For a set of feature maps, the max-pooling operation obtains the same number of sets of low-resolution feature maps by downsampling them. More, if a 2 × 2 max-pooling operation is applied to the feature map of a ₁ × a ₂ size, and the maximum value on the 2 × 2 non-overlapping region is extracted, we will get a size of a ₁ /2 × a ₂ New feature map of /2.

ReLU Nonliearity Layer: This layer uses a simple nonlinear threshold function to perform a transformation that allows only non-negative signals to pass through the input. Assume that g is the output of this layer, W is the weight of the edge of this layer, and a is the input of this layer, then we have

g=max(0,W ^T a)

Experiments show that in deep convolutional neural networks, the use of a corrected linear element layer can make the network convergence rate faster than the traditional excitation function.

Fully connected layer: We added two layers of fully connected layers to the model, which can be seen as a perceptron model (hidden layer and logistic regression layer) based on the previous two-dimensional convolutional layer output. We first concatenate the feature maps obtained from the M subnetworks into a long feature vector. This vector represents the feature extracted from the depth image sequence. Each dimension element of it is connected to all sections of the first fully connected layer (hidden layer) Point and further connect all the output units. The output unit has a total of 2K, where K represents the number of bone nodes, and the value of the output unit is the two-dimensional coordinate position of the bone node on the depth image.

Normalized layer: The normalized layer refers to the normalization of the coordinates manually labeled in the dataset. Train a CNN network that detects people and then use them in the normalization layer to crop the targets in the depth map. This can reduce background interference and improve the accuracy of the final detection of bone points.

2. Generation of joint heat map

Let the given data set be {I _n , L _n }, n=1,...,N,N be the total number of data set samples. Where I _n represents the nth image, L _n represents the human bone point corresponding to the nth image, L _n ={l _k }, k=1,...,K,K represents a total of K labeled human bones Point, our model setting K is 19, as shown in Figure 3. l _k = (x _k , y _k ) is the position of the kth skeletal point. Assuming that the heat map of the kth skeletal point is hp _k , the coordinates lk _k ((hh _k , yh _k ) of l _k mapped on hp _k are expressed as follows:

x _k =stride×xh _k +offset (1.1)

y _k =stride×yh _k +offset (1.2)

Where stride is the step size, offset is the offset, and an extra is set.

To determine the size of the orange small diamond. Each value in hp _k represents the probability that the value is at the kth skeletal point position in I _n and takes the value [0, 1]. The algorithm for generating a heat map is as follows.

3. Model training

That is, given a picture, the corresponding K heat maps are obtained by the network proposed by the present invention. We assume that the K heat maps are arranged in a fixed order according to the body parts, so that it is convenient to compare with the real joint heat map and learn the predicted heat map corresponding to the real joint heat map. In order to normalize the size of the input image, the method of determining the size of the heat map and calculating the appropriate size of the input image is used here. The size of the heat map s _hp ×s _hp is determined by experience, and our model sets it to 50×50. Then the size of the input image s _I × s _{I is} defined as follows:

s _I =(s _hp -1)×stride+offset×2+1 (3.1)

For the above formula, since the part of the human body is easily approached to the edge of the image in the real joint heat map training data, a pad of a size of offset is added around the input image. The input to our model is the image and its corresponding K real joint heat maps, and the output is the corresponding predicted heat map for the K bone points. Not only does this reduce the complexity of the model (avoiding training one R-CNN for each heat map), but it also allows the weights of the individual heat maps to be shared.

Forward propagation:

Each frame of the dataset is propagated along our defined R-CNN model. The first layer is a normalized layer that normalizes input images of different sizes into a uniform size for subsequent propagation. Processing. Then through the full convolution network as shown in Figure 2, the output size is (batch_size, K, S _hp , S _hp ), where batch_size is the number of trainings for batch training.

Backpropagation:

After the forward propagation is completed, the output as described above is obtained. Backpropagation requires first finding the residual J(ω) between the heat map of the forward propagation output and the true joint heat map, and then finding its gradient for the parameter ω

The ω is updated by a random gradient descent algorithm to minimize the residual, and the residual loss function J(ω) is defined as follows.

Where ||·|| _F is the Frobenius norm, Y ^pred is the predicted heat map, and Y ^gt is the true joint heat map

However, the final effect of simply backpropagating such errors is not good because the background area is much larger than the foreground area. Therefore, we add a scale factor, that is, to set the residual of the background to 0 in a certain proportion during backpropagation, which can make the ratio of the foreground to the background close. For example, the heat map size is 50×50, and the small diamond area inside is 5×5, then the ratio of foreground to background is 1:100. Our model sets a scale factor ratio of 0.012 on the background. In the backpropagation, the ratio of foreground to background changes from 1:100 to 1:1.2.

The learning process of the model is summarized as Algorithm 2:

4. Model testing

Given a test image, enter the trained model and get a heat map of 19 bone points. For each heat map, find out its maximum response value, which is a bone point of the human body. Finally, by transforming the coordinates back to the original image by equations (1.1) and (1.2), the coordinates of the 19 human skeleton points can be obtained. Evaluation mark The standard is as follows:

Among them, pred_coord is the predicted coordinate, gt_coord is the real coordinate, and the subscript is the index of the skeleton point. The subscript ls (left shoulder) in the denominator indicates the left shoulder, and the rh (right hip) indicates the right hip, that is, the entire denominator indicates the length of the body posture torso. The true meaning of this evaluation is that the distance between the predicted coordinates and the real coordinates should be less than a certain ratio of the true torso length in the human body pose of the image. In our model, r is 20.

The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and combinations thereof may be made without departing from the spirit and scope of the invention. Simplifications should all be equivalent replacements and are included in the scope of the present invention.

Claims

A depth image human joint positioning method based on convolutional neural network, characterized in that it comprises a training process and a recognition process;

The steps of the training process are as follows:

1) Input training samples;

2) Initializing a deep convolutional neural network and its parameters, the parameters including the weights and offsets of each layer edge;

3) Using the forward algorithm and the backward algorithm, using the training samples to learn the parameters of the constructed convolutional neural network;

The steps in the identification process are as follows:

4) Enter the test sample;

5) Using the trained convolutional neural network to return the position of the human joint point to the input test sample.
The depth image human body joint positioning method based on convolutional neural network according to claim 1, wherein the training sample in the step 1) is an original depth image of a person captured by a depth angle depth camera and A collection of annotations.
The method for positioning a human body joint based on a depth image based on a convolutional neural network according to claim 1, wherein the convolutional neural network in the step 2) has a deep hierarchical structure and is composed of a convolutional layer and a downsampling layer. The correcting linear unit layer and the fully connected layer are stacked and normalized by the normalized layer, and the top layer of the convolutional neural network directly outputs the position of the human joint point.
The convolutional neural network-based depth image human joint positioning method according to claim 3, wherein the convolution layer refers to convolution of an input image or feature in a two-dimensional space to extract important features;

The downsampling layer uses a max-pooling operation, which refers to a process of downsampling a feature map according to a set strategy, and is used to extract features that maintain shape and offset invariance;

The correcting linear unit layer uses a simple nonlinear threshold function to perform a transformation that allows only non-negative signals to pass through the input;

In the fully connected layer, the feature maps obtained from the M subnetworks are first connected in series to form a long feature. Vector, the long feature vector represents the feature extracted from the depth image sequence, each dimension element of which is connected to all nodes of the first fully connected layer, and further connected to all output units, the output unit 2K, where K represents the number of bone nodes, and the value of the output unit is the two-dimensional coordinate position of the bone node on the depth image;

The normalized layer is a normalized operation of the coordinates manually labeled in the data set.
The depth image human body joint positioning method based on the convolutional neural network according to claim 1, wherein the step 3) further comprises a step of generating a joint heat map, specifically:

Let the given data set be {I n , L n }, n=1,...,N,N be the total number of data set samples, where I n represents the nth image and L n represents the nth image corresponding Human bone points, L n ={l k },k=1,...,K,K represent a total of K labeled human skeleton points; l k =(x k ,y k ), the kth skeletal point position, assuming that the k-th picture shows the skeleton point heat hp k, l k mapped coordinates on HP LH k k a = (xh k, yh k) as follows:

x k =stride×xh k +offset (1.1)

y k =stride×yh k +offset (1.2)

Where stride is the step size, offset is the offset, and an extra is set.
To determine the size of the orange small diamond, each value in hp k represents the probability that the value is at the kth bone point position in I n , and the value is [0, 1].
The method for positioning a human body joint based on a depth image based on a convolutional neural network according to claim 5, wherein in step 3), it is assumed that K heat maps are arranged in a fixed order according to human body parts during training, and are used for joints. The heat map is compared and the predicted heat map corresponding to the real joint heat map is learned. In order to normalize the size of the input image, the size of the heat map is first determined, and then the appropriate size of the input image is calculated; the size of the heat map s hp × s hp is determined by experience, then the size of the input image s I × s I is defined as follows:

s I= (s hp -1) ×stride+offset×2+1 (3.1)

For the above formula, since the human body part is easily approached to the edge of the image in the real joint heat map training data, a pad of size offset is added around the input image, and the input of the model is the image and its corresponding K The true joint heat map, the output is the corresponding predicted heat map for the K bone points.
The method for positioning a depth image human joint based on a convolutional neural network according to claim 6, The feature is that the forward algorithm is specifically:

Each frame of the data set is propagated along the defined R-CNN model. The first layer is a normalized layer, which can normalize the input images of different sizes into a uniform size, which is convenient for subsequent propagation. Processing, then through the full convolution network, the output size is (batch_size, K, S hp , S hp ), where batch_size is the number of training for batch training;

The backward algorithm is specifically:

Backpropagation requires first finding the residual J(ω) between the heat map of the forward propagation output and the real joint heat map, and then finding its gradient for the parameter ω
And the algorithm of random gradient descent is used to update ω to minimize the residual, and the loss function J(ω) of the residual is defined as follows;

Where ||·|| F is the Frobenius norm, Y pred is the predicted heat map, and Y gt is the true joint heat map.
The depth image human body joint positioning method based on convolutional neural network according to claim 1, wherein in the step 4), the test sample is an original depth image captured by a depth angle depth camera.
The method for positioning a human body joint according to a convolutional neural network according to claim 5, wherein in step 5), the specific method for retrieving the position of the joint point of the human body is:

Given a test image, enter the trained model, you can get the heat map of the bone points, find the maximum response value for each heat map, which is a bone point of the human body, and finally pass the formula (1.1) And (1.2), the coordinates are transformed back to the original image, the coordinates of the body skeleton points can be obtained, and the evaluation criteria are as follows:

Among them, pred_coord is the predicted coordinate, gt_coord is the real coordinate, the subscript is the index of the skeleton point, the subscript ls in the denominator indicates the left shoulder, and rh indicates the right hip, that is, the whole denominator It is the length of the body posture torso.