CN110580461A - A Facial Expression Recognition Algorithm Combining Multi-Level Convolutional Feature Pyramid - Google Patents

Abstract

The invention discloses a human facial expression recognition algorithm combined with a multi-level convolution feature pyramid, which comprises the following steps: inputting the original human facial expression picture into the first feature extraction network for feature extraction, and extracting global features; The enlarged facial expression picture is input to the second feature extraction network for feature extraction, and local features are extracted; a feature pyramid network is constructed in the first feature extraction network and the second feature extraction network; The local features in the feature extraction network are used for regional positioning; the global features and local features of the facial expression pictures are fused using the feature fusion network, and the facial expression pictures are classified through the fully connected layer. To achieve the purpose of improving the accuracy of facial expression recognition technology for facial expression recognition.

Description

A Facial Expression Recognition Algorithm Combining Multi-Level Convolutional Feature Pyramid

technical field

The invention relates to the field of image processing, in particular to a facial expression recognition algorithm combined with a multi-level convolution feature pyramid.

Background technique

Facial expressions are not only an expression of people's own emotions, but also can convey extremely rich information about human behavior. Facial expression recognition technology is the main form of human emotion research. Accurate recognition of human facial expressions can effectively judge people's emotional state. Facial expression recognition can be applied in many fields, such as safe driving, intelligent human-computer interaction, medical monitoring, lie detection in criminal investigation, and psychotherapy. However, the existing facial expression recognition technology has a low accuracy rate for facial expression recognition.

Contents of the invention

The purpose of the present invention is to provide a facial expression recognition algorithm combined with a multi-level convolutional feature pyramid, aiming to solve the technical problem of low accuracy of facial expression recognition technology in the prior art.

In order to achieve the above object, a kind of face expression recognition algorithm that the present invention adopts in conjunction with multi-level convolution feature pyramid comprises the following steps:

Input the original facial expression picture to the first feature extraction network for feature extraction, and extract global features;

Input the cropped and enlarged facial expression picture to the second feature extraction network for feature extraction, and extract local features;

Constructing a feature pyramid network in the first feature extraction network and the second feature extraction network;

Using the attention region localization network to localize the local features in the second feature extraction network;

The feature fusion network is used to fuse the global features and local features of the facial expression pictures, and the facial expression pictures are classified through the fully connected layer.

Wherein, the input samples of the second-level feature extraction network are obtained by cropping and enlarging the input sample pictures in the first-level feature extraction network.

Among them, the first-level feature extraction network and the second-level feature extraction network are both a fully convolutional network.

Among them, the first convolutional layer in extracting global features and extracting local features uses a convolution kernel with a size of 3*3.

Wherein, the original facial expression picture is input into the first-level feature extraction network, and the feature of the facial expression picture is extracted by a convolutional neural network, and then the output category and the probability value of each category, the acquisition of the probability value of each category The method is:

C ⁽ⁱ⁾ ＝f(W _i ⊙X _i )

p ⁽ⁱ⁾ = f(W _i ⊙X _i )

Among them, the input facial expression picture is X, X _i and W _i represent the input and weight parameters of the i-th level convolutional neural network, and ⊙ represents the convolution, pooling, activation or normalization operations performed during the convolution process , f( ) represents the feature extraction network, C ⁽ⁱ⁾ (i=1,2) represents the output category label of the i-th level convolutional neural network, p ⁽ⁱ⁾ (i=1,2) represents the i-th level volume The product neural network outputs the probability value of each category.

Among them, each level of network will generate a feature map about the input facial expression picture, using Represents the set of feature maps generated by each level of network, m represents the number of network series included in the model, where m=(1,2), n represents the number of feature maps output by each level of network, and the input image X is represented by the following The formula means:

Among them, the feature map generated by the shallow convolution layer in the convolutional neural network has a large scale, and the feature map generated by the deep convolution layer of the convolutional neural network has a small scale. Set the convolution operation, and then add the two feature maps to unify the scale of the feature maps.

Among them, the attention area positioning network maps the input feature map to a square attention area with (D _x , D _y ) as the center and D _α as the side length, and then maps the attention area to the original facial expression picture, and then It is sent to the next level network after cropping and amplifying.

Among them, the second feature extraction network can automatically activate the salient areas of facial expressions during the recognition and extraction of facial expressions. The area is mapped to the original facial expression picture, and the important local area of the original facial expression is obtained, and the feature extraction of the local area is carried out to realize the local area recognition of the picture.

Among them, at the time of initialization, all the feature maps generated by the last layer of the first-level feature extraction network are added together, and then the area with a larger value in the added feature map is located, and the located area is fitted as A square, then get the center coordinates of the square and the side length of the square, and set them as the initialization parameters of the attention network, expressed as:

Among them, f represents the feature map generated by the last convolutional layer of the first feature extraction network, d represents the total number of feature maps generated by the network, and F is the sum of each point corresponding to each feature map. The total feature map, h and w represent the height and width of the feature map, and p represents the pixel mean of the total feature map.

A kind of facial expression recognition algorithm combined with multi-level convolution feature pyramid of the present invention, by inputting the original facial expression picture into the first feature extraction network for feature extraction, and extracting global features; Input the expression picture to the second feature extraction network for feature extraction, and extract local features; construct a feature pyramid network in the first feature extraction network and the second feature extraction network; use the attention area positioning network to pair it into the second feature extraction network The local features of the facial expression are used to locate the area; the global features and local features of the facial expression picture are fused using the feature fusion network, and the facial expression picture is classified through the fully connected layer. Among them, the transfer of face features from global to local is realized by adopting the cascaded structure of two-level face feature extraction network, and a feature pyramid network is added to each level of face feature extraction network, thereby improving the algorithm's accuracy for face targets. Size robustness. In this way, the effect of improving the accuracy of facial expression recognition by the facial expression recognition technology is obtained.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a step diagram of the facial expression recognition algorithm combined with the multi-level convolution feature pyramid of the present invention.

Fig. 2 is a flow chart of the facial expression recognition algorithm combined with the multi-level convolutional feature pyramid of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

In describing the present invention, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", The orientation or positional relationship indicated by "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than Nothing indicating or implying that a referenced device or element must have a particular orientation, be constructed, and operate in a particular orientation should therefore not be construed as limiting the invention. In addition, in the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

Please refer to Fig. 1 and Fig. 2, the present invention provides a kind of facial expression recognition algorithm in conjunction with multi-level convolution feature pyramid, comprises the steps:

S100: Input the original facial expression picture to the first feature extraction network for feature extraction, and extract global features;

S200: Input the cropped and enlarged facial expression picture to the second feature extraction network for feature extraction, and extract local features;

S300: Construct a feature pyramid network in the first feature extraction network and the second feature extraction network;

S400: Use the attention area positioning network to perform area positioning on the local features in the second feature extraction network;

S500: Use a feature fusion network to fuse the global features and local features of the facial expression pictures, and classify the facial expression pictures through a fully connected layer.

In this embodiment, the first-level feature extraction network can perform feature extraction on the original facial expression picture of 448*448 pixels, and extract global features; Feature extraction is performed on facial expression pictures, and local features are extracted; the input samples of the second-level feature extraction network are obtained by cropping and enlarging the input sample pictures in the first-level feature extraction network, and the first feature extraction network and the second feature extraction network The network is a fully convolutional network. The structure of the network is similar to the VGG19 network structure. Each module of the network consists of a convolutional layer, a BatchNorm layer, a relu layer and an average pooling layer. There are 16 convolutional modules in total. The first convolutional layer in extracting global features and extracting local features uses a convolution kernel with a size of 3*3.

The original facial expression picture is input into the first-level feature extraction network, and the features of the facial expression picture are extracted through the convolutional neural network, and then the output category and the probability value of each category are expressed by the following formula:

C ⁽ⁱ⁾ ＝f(W _i ⊙X _i )

p ⁽ⁱ⁾ = f(W _i ⊙X _i )

In the above formula, the input facial expression picture is X, Xi and W _i represent the input and weight parameters of the i- _th convolutional neural network, and ⊙ represents the convolution, pooling, activation or normalization performed in the convolution process operation, f( ) represents the feature extraction network, C ⁽ⁱ⁾ (i=1,2) represents the output category label of the i-th convolutional neural network, p ⁽ⁱ⁾ (i=1,2) represents the i-th The level convolutional neural network outputs the probability value for each category.

Each level of the network will generate a feature map of the input facial expression picture, using Represents the set of feature maps generated by each level of network, m represents the number of network series included in the model, where m=(1,2), n represents the number of feature maps output by each level of network, and the input image X is represented by the following The formula means:

The convolutional neural network extracts the features of the input image layer by layer through the convolution kernel of each layer. The feature maps of different convolution layers correspond to different areas and targets of the input image. Make good use of the shallow convolutional layer of the convolutional neural network to extract facial expression information, and construct a feature pyramid network between each convolutional layer of the convolutional neural network, due to the shallow convolutional layer in the convolutional neural network. The feature map of the convolutional neural network has a large scale, while the feature map generated by the convolutional neural network is small. In order to unify the scale of the feature map, the feature map generated by the convolutional layer of the next layer is first transposed and convolved. operation, and then add the two feature maps, and the generated feature map is the final representation of the entire facial expression picture.

The global information of facial expression pictures can provide the overall information of facial expressions, while the local features can provide more refined detailed information of facial expressions. In order to realize the transition and transfer of facial expressions from global features to local features, the attention area The positioning network is between the feature extraction network, responsible for locating the local feature areas (such as eyebrows, eyes, nose, mouth) in the original facial expression picture, and cutting the corresponding area, and then sending it to the next level network . The facial expression recognition network that has been trained can automatically activate the salient area of facial expression (that is, the area that has an important influence on the recognition effect of facial expression) when recognizing facial expression, so the feature must first be The region with the largest response in the figure is precisely positioned, and then these precisely positioned regions are mapped to the original facial expression picture, so that the very important local area of the original facial expression can be obtained, and feature extraction is performed on the local area to realize the picture. Local area recognition. Automatically localize regions that have significant influence on facial expression recognition by employing a hierarchical localization attention network.

The attention area localization network maps the input feature map into a square attention area with (D _x , D _y ) as the center and D _a as the side length, and then maps the attention area to the original facial expression picture, and then After cropping and zooming in, it is sent to the next-level network, so that the refined recognition of facial expression pictures from global feature pictures to local feature areas can be realized.

The key to mapping the strong response area in the feature map obtained by the convolutional neural network to the original facial expression picture is to obtain the coordinates of the strong response area in the feature map, so by designing a 3-layer fully connected neural network attention area positioning network , passing the feature map through the fully connected layer network can realize the automatic positioning of the strong response area in the feature map. In order to speed up the convergence speed of the attention area positioning network, all the feature maps generated by the last layer of the first-level feature extraction network are added together during initialization, and then the areas with larger values in the added feature maps are located. Come out, fit the positioned area into a square, then get the center coordinates of the square and the side length of the square, and set them as the initialization parameters of the attention network. The specific implementation method is:

In the above formula, f represents the feature map generated by the last convolutional layer of the first feature extraction network, d represents the total number of feature maps generated by the network, and F is the sum of each point corresponding to each feature map After the total feature map, h and w represent the height and width of the feature map, p represents the pixel mean value of the total feature map, and then compare each pixel point of the total feature map with the pixel mean value, which is equivalent to the The pixel mean value is the threshold of the total feature map, the pixel points greater than the threshold value are set to 1, and the pixel points less than or equal to the threshold value are set to 0, and then the longest side of the largest connected region set to 1 is selected as a square Side length, and finally initialize the parameters of the attention positioning network with the center coordinates and side length of the square. The attention area positioning network can automatically locate the area with the largest response in the feature map, and then crop and enlarge the attention area after positioning. Assume that the lower left corner of the picture is the origin, the horizontal direction is the x-axis, and it gradually increases from left to right. , the vertical direction is the y-axis, which gradually increases from bottom to top. According to the coordinates of the center point of the square attention area and its side length obtained earlier, we can calculate the four vertices of the square attention area to realize the automatic cropping of the attention area, and finally use bilinear interpolation to the attention area The picture is enlarged. Get the pictures we need.

Each level of feature extraction network of the algorithm proposed above will extract the features of the facial expression picture, the first feature extraction network extracts the global features of the facial expression picture, the second network extracts the local detail features of the facial expression picture, and each The feature map of the channel is added to the corresponding pixels, and then the feature dimension is reduced by using a 1*1 convolution kernel, and then the feature fusion and pooling are performed, and finally the category label of the facial expression is output through the fully connected layer.

After designing the network according to the above method, in order to optimize the feature extraction network and the attention area positioning network at the same time, two loss functions are used to optimize the network, using the cross entropy loss function of these two networks and the intersection of the fusion network The weighted sum of the entropy loss function is used as the overall loss function of the network, which is very effective in distinguishing different facial expressions. At the same time, in order to constrain the difference of similar facial expressions, two features extraction network and attention area positioning network are used. The network penalizes the validation loss, which greatly enhances the recognition performance between similar facial expressions.

To sum up: the algorithm can automatically locate the areas with high discrimination of facial expression pictures, integrate the global and local features of facial expression pictures, and realize the accurate recognition of facial expression pictures. The first feature extraction network and the second feature extraction network are responsible for feature extraction of facial expression pictures, and the attention area positioning network is responsible for locating local feature areas of facial expression pictures, representing and locating local feature areas of faces, and for similar High-resolution facial expression pictures, such as anger and disgust, the algorithm realizes the regional localization of high-resolution facial expression local features, and comprehensively utilizes the superficial high-resolution facial structure information of the convolutional neural network and the deep low-level The high-resolution semantic information obtains better positioning of local feature areas of the face, and greatly improves the recognition rate of facial expression pictures with high similarity.

The feature fusion network is responsible for fusing the global features and local features of facial expressions, and fusing the global feature information of facial expression pictures with the local feature information of facial expression pictures. The local feature information of facial expression pictures can improve the ability to distinguish facial expressions with high similarity, and at the same time retain the global structural information of facial expression pictures, so the feature fusion network is used to extract the facial expression feature extraction network at all levels. The features are fused to obtain more comprehensive feature information of facial expression pictures, thereby improving the accuracy of facial expression recognition. Since the unipolar facial expression feature extraction network cannot obtain the global and local features of facial expression pictures at the same time, a two-level facial expression feature extraction network cascade structure is adopted to realize the transfer of features from global to local. Expression pictures have the characteristics of scale changes. A feature pyramid network is constructed between the feature maps of the facial expression feature extraction network at each level, thereby improving the feature description ability of the network and greatly improving the performance of facial expression pictures with scale changes. , the recognition accuracy of facial expression pictures with occlusion and similarity facial expression pictures.

What is disclosed above is only a preferred embodiment of the present invention, and of course it cannot limit the scope of rights of the present invention. Those of ordinary skill in the art can understand all or part of the process for realizing the above embodiments, and according to the rights of the present invention The equivalent changes required still belong to the scope covered by the invention.

Claims (10)

1. A facial expression recognition algorithm combined with a multilevel convolution feature pyramid is characterized by comprising the following steps:

Inputting the original facial expression picture into a first feature extraction network for feature extraction, and extracting global features;

Inputting the cut and amplified facial expression picture into a second feature extraction network for feature extraction, and extracting local features;

Constructing a feature pyramid network in the first feature extraction network and the second feature extraction network;

Carrying out regional positioning on the local features in the second feature extraction network by using an attention regional positioning network;

And fusing the global features and the local features of the human face expression pictures by using the feature fusion network, and classifying the human face expression pictures through a full connection layer.

2. The algorithm for facial expression recognition in combination with the multi-level pyramid of convolution features of claim 1,

The input sample of the second-level feature extraction network is obtained by cutting and amplifying an input sample picture in the first-level feature extraction network.

3. The algorithm for facial expression recognition in combination with the multi-level pyramid of convolution features of claim 1,

The first level feature extraction network and the second level feature extraction network are both full convolution structure networks.

4. The algorithm for facial expression recognition in combination with the multi-level pyramid of convolution features of claim 3,

The first convolutional layer in both global feature extraction and local feature extraction uses a convolution kernel of size 3 x 3.

5. The algorithm for facial expression recognition in combination with the pyramid of multilevel convolution features of claim 4,

Inputting an original facial expression picture into a first-level feature extraction network, extracting features of the facial expression picture through a convolutional neural network, and then outputting categories and probability values of each category, wherein the probability value of each category is obtained by the following steps:

C⁽ⁱ⁾＝f(W_i⊙X_i)

p⁽ⁱ⁾＝f(W_i⊙X_i)

Wherein, the input facial expression picture is X, X_iAnd W_irepresents the input and weight parameters of the i-th convolutional neural network, indicates the convolution, pooling, activation, or normalization operation performed during convolution, f (-) indicates the feature extraction network, C⁽ⁱ⁾(i ═ 1,2) denotes an output class label of the i-th convolutional neural network, p⁽ⁱ⁾(i ═ 1,2) denotes the probability value of the i-th order convolutional neural network for outputting each class.

6. The algorithm for facial expression recognition in combination with the pyramid of multilevel convolution features of claim 5,

Each level of network generates a feature map about the input facial expression picturerepresenting a set of generated feature maps of each level of network, wherein m represents the number of network levels contained in the model, wherein m is equal to (1,2), n represents the number of the feature maps finally output by each level of network, and an input image X is represented as:

7. The algorithm for facial expression recognition in combination with the multi-level pyramid of convolution features of claim 6,

The feature graph generated by the shallow convolutional layer in the convolutional neural network is large in scale, the feature graph generated by the deep convolutional layer in the convolutional neural network is small in scale, the feature graph generated by the next convolutional layer is transposed and convolved, and then the two feature graphs are added to unify the scales of the feature graphs.

8. the algorithm for facial expression recognition in combination with the multi-level pyramid of convolution features of claim 7,

the attention area positioning network maps the input feature map into (D)_x,D_y) Is a center, D_αthe attention area is a square attention area with side length, then the attention area is mapped to an original facial expression picture, and the original facial expression picture is cut and amplified and then sent to a next-level network.

9. The algorithm for facial expression recognition in combination with the pyramid of multilevel convolution features of claim 8,

The second feature extraction network can automatically activate the salient regions of the facial expressions in the process of identifying and extracting the facial expressions, firstly, accurate positioning must be achieved on the regions with the largest response in the feature images, then the accurately positioned regions are mapped to the original facial expression images, important local regions of the original facial expressions are obtained, and feature extraction is conducted on the local regions to achieve local region identification of the images.

10. The algorithm for facial expression recognition in combination with the multi-level pyramid of convolution features of claim 9,

When initializing, all feature maps generated by the last layer of the first-level feature extraction network are added together, then an area with a larger numerical value in the added feature maps is positioned, the positioned area is fitted into a square, then the central coordinate and the side length of the square are obtained and set as initialization parameters of the attention network, and the initialization parameters are expressed as:

Wherein F represents a feature map generated by the last convolutional layer of the first feature extraction network, d represents the total number of feature maps generated by the network, F is the total feature map obtained by adding each corresponding point of each feature map, h and w represent the height and width of the feature map, and p represents the pixel mean value of the total feature map.