CN101908149A - Method for identifying facial expressions from human face image sequence - Google Patents
Method for identifying facial expressions from human face image sequence Download PDFInfo
- Publication number
- CN101908149A CN101908149A CN2010102185432A CN201010218543A CN101908149A CN 101908149 A CN101908149 A CN 101908149A CN 2010102185432 A CN2010102185432 A CN 2010102185432A CN 201010218543 A CN201010218543 A CN 201010218543A CN 101908149 A CN101908149 A CN 101908149A
- Authority
- CN
- China
- Prior art keywords
- image
- mfv
- expression
- msub
- points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000008921 facial expression Effects 0.000 title claims abstract description 36
- 230000014509 gene expression Effects 0.000 claims abstract description 71
- 230000001815 facial effect Effects 0.000 claims abstract description 29
- 239000011159 matrix material Substances 0.000 claims abstract description 20
- 238000006073 displacement reaction Methods 0.000 claims abstract description 14
- 238000010219 correlation analysis Methods 0.000 claims abstract description 12
- 239000013598 vector Substances 0.000 claims abstract description 6
- 210000004709 eyebrow Anatomy 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 13
- 238000012360 testing method Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 7
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 claims description 5
- 230000007935 neutral effect Effects 0.000 claims description 5
- 210000000887 face Anatomy 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 239000002245 particle Substances 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 3
- 238000011426 transformation method Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 2
- 210000003128 head Anatomy 0.000 claims description 2
- 238000013507 mapping Methods 0.000 claims description 2
- 210000000216 zygoma Anatomy 0.000 claims description 2
- 238000001514 detection method Methods 0.000 claims 1
- 230000008569 process Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 206010063659 Aversion Diseases 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000010195 expression analysis Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 241000557626 Corvus corax Species 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000011960 computer-aided design Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention relates to a method for identifying facial expressions from a human face image sequence, belonging to the technical field of analyzing and identifying human facial expressions. The method of the invention comprises the following steps of: firstly, adopting a method for tracing feature points, sequentially extracting the displacement amount of the normalized facial key point and the length of the special geometrical characteristic for each frame image of the expression image sequence, and combining the data to form a characteristic column vector; secondly, sequentially arranging all characteristic column vectors of the sequence to form a characteristic matrix, wherein each characteristic matrix represents a facial expression image sequence; finally, comparing the similarities among the characteristic matrixes by using a canonical correlation analysis method, thereby determining the human face images to be identified into one of the basic expressions of happiness, sadness, fear, hate, surprise and anger. In the invention, the canonical correlation analysis method is successfully applied to identifying the human facial expressions, the dynamic information in the expression generation course is utilized effectively and the higher recognition rate and the shorter CPU computation time are acquired.
Description
Technical Field
The invention relates to a method for identifying facial expressions from a facial image sequence, and belongs to the technical field of facial expression analysis and identification.
Background
With the rapid development of computer technology, automatic facial expression analysis and recognition technology will make facial expression a new channel for human-computer interaction, and make the interaction process more natural and effective. Facial expression analysis and recognition involves three basic problems: finding and positioning a human face in an image; how to extract effective expression features from the detected facial image or facial image sequence; and designing a proper classification method to identify the expression types. In recent years, there has been much research effort devoted to identifying facial expressions from image sequences: cohn et al propose an Optical flow-based method to identify Subtle changes in Facial Expression in the document Feature-Point Tracking by Optical flow discovery sub Differences Expression (Int' l conf. automatic Face and Gesture Recognition, pp.396-401 (1998)). Laj evardi et al, in the literature, "Facial expression recognition from image sequences using optimal temporal extraction selection" (IVCNZ 2008, New Zealand, Page(s): 1-6(2008)), disclose a method for recognizing Facial expressions using a Naive Bayes (NB) classifier through an optimized feature selection process. Sunkening et al, in the document LSVM algorithm for video sequence expression classification (computer aided design and graphics declaration, volume 21, phase 4 (2009)), disclose a method for extracting expression geometric features from video faces by using an Active Shape Model (ASM) based on point tracking, and classifying the expressions by using a Local Support Vector Machine (LSVM) classifier. The disadvantages of these methods are: the features are only extracted from the peak expression frame, and important time domain dynamic information contained in the expression generation process is ignored, so that the recognition accuracy is not high.
In addition, "Active and Dynamic Information Fusion for Facial expression from Image Sequences" (IEEE TRANSACTIONS ON PATTERNANALYSIS AND MACHINE INTELLIGENCE, VOL.27, NO.5, MAY 2005), "Artificial analysis of Facial expression (Image and Vision Computing 24(2006)605 and 614)," Facial expression from Image Sequences: although the methods proposed by the documents such as temporal and static modeling (Computer Vision and Image interpretation 91(2003)160-187) use the time domain dynamic information contained in the expression generation process, the methods are complex in calculation and high in calculation cost.
An important prior art used in the present invention is: typical correlation Analysis (CCA).
Typical correlation analysis is a classical tool in statistical analysis that can be used to measure linear relationships between two or more data sets. A typical correlation coefficient is defined as two d-dimensional linear subspaces L1And L2Main angle theta betweeniCosine value of (d):
wherein:
the parameter d represents the linear subspace dimension.
In recent years, the typical correlation analysis technology has been successfully applied to the fields of image set matching, face or object recognition and the like, so that the use of the typical correlation analysis technology for solving the expression recognition problem is a simple and effective method in theory. However, in the expression recognition problem, the facial images of different expressions of the same person are not very different, and even the images of two opposite expressions are not very different, so that a good effect cannot be obtained by simply applying the typical correlation analysis technology to the expression recognition. So far, no relevant documents and practical applications for using typical correlation analysis techniques in facial expression recognition have been found.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a method for recognizing facial expressions from a human face image sequence. The invention uses a facial feature point tracking method to sequentially extract normalized displacement of facial key points and the length of specific geometric features for each frame of image in an expression image sequence, and the data are combined into a feature column vector; all the characteristic column vectors in the sequence are arranged in sequence to form a characteristic matrix, and each characteristic matrix represents a facial expression image sequence; then, the similarity between the feature matrixes is compared by using a typical correlation analysis method, so that the face image to be recognized is determined to be one of six basic expressions (happiness, sadness, fear, disgust, surprise and anger).
The purpose of the invention is realized by the following technical scheme.
A method for recognizing facial expressions from a facial image sequence comprises the following specific operation steps:
step one, selecting an image sequence
Selecting image sequences representing six basic expressions such as happiness, sadness, fear, disgust, surprise and anger from a facial expression database, wherein the number of the image sequences of each basic expression is more than 20; and selecting m (m is more than or equal to 10 and m is a positive integer) frame images from each expression image sequence, wherein each expression image sequence starts from a neutral expression image and ends at a peak expression image.
Step two, identifying facial feature points
On the basis of the first step, identifying facial feature points; the method specifically comprises the following steps:
step 1: sequentially identifying 20 facial feature points in a first frame image in each expression image sequence; the 1 st characteristic point and the 2 nd characteristic point are respectively positioned at the eyebrow positions of the right eyebrow and the left eyebrow; the 3 rd and 4 th characteristic points are respectively positioned at the eyebrow tail positions of the right eyebrow and the left eyebrow; the 5 th and 6 th characteristic points are respectively positioned at the inner canthus positions of the right eye and the left eye; the 7 th and 8 th characteristic points are respectively positioned at the lowest points of the right eye and the left eye; the 9 th and 10 th characteristic points are respectively positioned at the external canthus positions of the right eye and the left eye; the 11 th and 12 th characteristic points are respectively positioned at the highest points of the right eye and the left eye; the 13 th characteristic point and the 14 th characteristic point are respectively positioned at the rightmost position of the nasal wing and the leftmost position of the nasal wing; the 15 th characteristic point is positioned at the tip of the nose; the 16 th characteristic point and the 17 th characteristic point are respectively positioned at the rightmost position of the mouth corner and the leftmost position of the mouth corner; the 18 th and 19 th characteristic points are respectively positioned at the highest point and the lowest point of the intersection of the lip center line and the lip contour line; the 20 th feature point is located at the lowest point where the face center line intersects the face contour line.
The method for identifying 20 facial feature points includes but is not limited to: manual marking; secondly, the automatic positioning of 20 face feature points is realized by adopting a Gabor feature-based enhanced classifier method provided by Vukdadinovic et al in the document ' full automatic facial feature point detecting and detecting device and devices ' (Proc. IEEE ' l Conf. on Systems, Man and Cybernetics, pp.1692-1698 (2005)).
Step 2: calculating the positions of the 21 st and 22 nd feature points in the first frame image in each expression image sequence according to the statistical data of the spatial relationship between the eyes and the cheeks, the nose and the cheeks of the human being proposed by Farkas in the Anthropometry of the Head and Face (New York: Raven Press (1994)); the 21 st and 22 nd feature points are located at the position of the cheekbones of the right and left cheeks, respectively.
And 3, step 3: the Particle filter tracking method based on likelihood function factorization proposed in Patras et al, document "Particle filtering with factitious after faces for tracking facial features" (Proc. int' l Conf. automatic face & Gesture Recognition, pp.97-102(2004)) is adopted to track 22 facial feature points in the subsequent frame images in each expression image sequence according to the positions of 22 feature points in the first frame image in each expression image sequence.
And 4, step 4: and (3) adjusting the position and the size of the face in the image by adopting an affine transformation method, so that the face in the same image sequence has the same size and the same position. The method specifically comprises the following steps:
firstly, keeping a connecting line of two inner eye corner points in a first frame of image in each image sequence horizontal; then, according to the positions of 3 points, namely two inner eye corner points in the first frame of image in the image sequence and the uppermost point in the person, mapping and normalizing 22 face feature points in the rest frames; after affine transformation processing, the human faces of all the images in the same image sequence are equal in size, and the positions of 3 points, namely the two inner eye corner points and the uppermost point in the human, are consistent with the positions of the 3 points in the first frame.
Step three, extracting facial expression characteristics
On the basis of the operation of the second step, facial expression characteristics are extracted from each image in sequence; the method specifically comprises the following steps:
step 1: establishing an Xb-Yb coordinate system, wherein the coordinate system takes the lower left corner of each image as an origin, the horizontal right direction is an Xb axis, and the vertical upward direction is an Yb axis; obtaining the coordinate values (Xb) of 22 characteristic points based on an Xb-Yb coordinate system on each image in sequence according to the positions of the pixels where the 22 characteristic points are located on each imagei，ybi) And the coordinate value (x) of the uppermost point in the human face of each imageorigin，yorigin) Wherein i is 1-22, and i is a positive integer;
step 2: establishing an X-Y coordinate system, wherein the coordinate system takes the uppermost point in the human face of each image as an original point, the horizontal right direction is an X axis, and the vertical upward direction is a Y axis; coordinate values (X) of 22 feature points based on the X-Y coordinate system on each image are obtained by formula 1 and formula 2i，yi)；
xi＝xbi-xorigin (1)
yi＝ybi-yorigin (2)
And 3, step 3: the abscissa displacement amounts of 22 feature points of each image are obtained by formula 3 and formula 4
Displacement of ordinate
Wherein,respectively the abscissa value and the ordinate value of the corresponding feature point of the first frame image in the image sequence in which the image is located.
And 4, step 4: obtaining the normalized abscissa displacement quantity delta x of the 22 characteristic points of each image through a formula 5 and a formula 6 in sequenceiThe longitudinal coordinate displacement amount Deltayi；
Wherein x isbase＝x6-x5，ybase＝y6-y5；x5And x6Respectively the abscissa values of the 5 th and 6 th feature points of the image; y is5And y6The ordinate values of the 5 th and 6 th feature points of the image are respectively.
And 5, step 5: obtaining the 10 geometric distance features mfv in each image1～mfv10(ii) a The method specifically comprises the following steps:
obtaining an eye openness value mfv according to equation 71：
mfv1＝((y11-y7)+(y12-y8))/2 (7)
Wherein, y7、y8、y11、y12The ordinate values of the 7 th, 8 th, 11 th, and 12 th feature points of the image are shown.
Obtaining an eye width value mfv according to equation 82：
mfv2＝((x5-x9)+(x10-x6))/2 (8)
Wherein x is5、x6、x9、x10The abscissa values of the 5 th, 6 th, 9 th, and 10 th feature points of the image are shown.
Obtaining a eyebrow height value mfv according to equation 93：
mfv3＝(y1+y2)/2 (9)
Wherein, y1、y2The vertical coordinate values of the 1 st and 2 nd feature points of the image are respectively.
Obtaining the brow-tail height value mfv according to equation 104：
mfv4＝(y3+y4)/2 (10)
Wherein, y3、y4The ordinate values of the 3 rd and 4 th feature points of the image are respectively.
Obtaining the eyebrow width value mfv according to equation 115：
mfv5＝((x1-x3)+(x4-x2))/2 (11)
Wherein x is1、x2、x3、x4The abscissa values of the 1 st, 2 nd, 3 th and 4 th feature points of the image are shown.
The mouth opening degree value mfv is obtained according to equation 126：
mfv6＝y18-y19 (12)
Wherein, y18、y19The ordinate values of the 18 th and 19 th feature points of the image are shown.
The mouth width value mfv is obtained according to equation 137：
mfv7＝x17-x16 (13)
Wherein x is16、x17The abscissa values of the 16 th and 17 th feature points of the image are shown, respectively.
Obtaining a nose tip-mouth angle distance value mfv according to equation 148：
mfv8＝((y15-y16)+(y15-y17))/2 (14)
Wherein, y15、y16、y17The ordinate values of the 15 th, 16 th and 17 th feature points of the image are shown.
Obtaining an eye-cheek distance value mfv according to equation 159：
mfv9＝(((y11+y7)/2-y21)+((y12+y8)/2-y22))/2 (15)
Wherein, y21、y22The ordinate values of the 21 st and 22 nd feature points of the image are respectively.
Nasal tip-chin distance value mfv obtained according to equation 1610：
mfv10＝y15-y20 (16)
Wherein, y15、y20The ordinate values of the 15 th and 20 th feature points of the image are respectively.
And 6, step 6: the 10 geometric distance features mfv in each image are shown by equation 171～mfv10Carrying out normalization processing;
wherein j is 1-10, and j is a positive integer;
and the geometric distance characteristic corresponding to the image in the first frame of image in the image sequence is obtained.
And 7, step 7: using 10 geometric distance features mfv of each image respectively1～mfv10And the abscissa displacement delta x obtained after normalization processing of 22 characteristic points of each imageiThe longitudinal coordinate displacement amount DeltayiForming a 54-dimensional column vector z representing expression information of a facial imagek(ii) a Wherein: z is a radical ofk∈R54K is more than or equal to 1 and less than or equal to m, and R represents a real number);
and 8, step 8: using the feature matrix Z ═ Z1，z2，...，zm}∈R54×mRepresenting a sequence of emoticons; wherein z is1Denotes neutral expression, zmIndicating a peak expression.
Step four, classifying the trial image sequence to be detected;
on the basis of the third step, classifying the image sequence to be tested by adopting typical correlation analysis; the method specifically comprises the following steps:
step 1: randomly dividing the image sequence of each basic expression selected in the step one into 2 parts, wherein one part is used as training data, and the other part is used as test data; the number of the training data is Q, Q is more than or equal to 20, and Q is a positive integer; one piece of training data is an expression image sequence; a piece of test data is also a sequence of expression images.
Step 2: the training data obtained in step 1 is processed by a typical correlation discriminant Analysis method proposed by T.K. Kim et al in the literature, "characterization Learning and correlation of Image Set Classes using canonical Correlations" (IEEE Transactions On Pattern Analysis and machine Analysis, Vol.29, No.6(2007)), to obtain a transformation matrix T ∈ R54×nN is less than 54, and n is a positive integer; and then, converting a feature matrix Z of the images (including training data and test data) in all the expression image sequences selected in the step one by using a transformation matrix T to obtain Z' ═ TTZ。
And 3, step 3: randomly selecting an expression image sequence from the test data in the step 1, and calculating the typical correlation coefficient sum of the feature matrix Z 'of the expression image sequence and the feature matrix Z' of each training data.
And 4, step 4: on the basis of the result of the step 3, respectively calculating the average value of the sum of the typical correlation coefficients of the expression image sequence and each basic expression; and selecting the expression corresponding to the minimum value in the 6 average values as a classification result.
Through the steps, the expression recognition of the image sequence to be tested can be completed.
Advantageous effects
Compared with the existing recognition method, the method for recognizing the facial expression from the facial image sequence successfully applies the typical correlation analysis method to the facial expression recognition, effectively utilizes the dynamic information in the expression generation process, and obtains higher recognition rate and shorter CPU operation time.
Drawings
FIG. 1 is a diagram of 7 images in a 15 image sequence according to an embodiment of the present invention;
FIG. 2 is a diagram of feature points in a face image and Xb-Yb and X-Y coordinate systems according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the general structural framework of the method of the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments.
In this embodiment, a Cohn-Kanade facial expression database is used to select 212 image sequences of 50 persons representing six basic expressions of happiness, sadness, fear, disgust, surprise, anger, and the like. 15 frames of images are selected from each expression image sequence, starting from a neutral expression image and ending at a peak expression image. Fig. 1 shows 7 frames of a 15-frame image sequence. The 22 feature points in each face image, and the Xb-Yb coordinate system and the X-Y coordinate system are shown in FIG. 2. And selecting 35 personal image sequences as a training set and the rest image sequences as a test set so as to ensure the facial expression classification effect independent of individuals. Each test was performed 5 times using a randomly selected test and training set, and the average result was calculated. In the learning and classification process, the parameter d representing the dimension of the linear subspace is set to 10, and the parameter n representing the dimension of the transformation matrix T is set to 20.
The general structural framework of the process of the invention is schematically illustrated in figure 3. The confusion matrix of the recognition result is shown in table 1 by applying the method of the present invention. The diagonal elements of the matrix are the percentage of facial expressions that are correctly classified and the non-diagonal elements correspond to the percentage of misclassifications. The average accuracy of the method herein is over 90%.
TABLE 1 confusion matrix (%) -of the results identified by the method herein
| Happiness | Sadness and sorrow | Fear of | Aversion to | Is surprised | Generating qi | |
| Happiness | 95.2 | 0 | 2.6 | 0 | 2.2 | 0 |
| Sadness and |
0 | 92.5 | 0 | 2.8 | 0 | 4.7 |
| Fear of | 8.8 | 0 | 87.2 | 0 | 0 | 4.0 |
| Aversion to | 0 | 1.5 | 7.4 | 91.1 | 0 | 0 |
| Is surprised | 5.2 | 0 | 1.2 | 0 | 90.5 | 3.1 |
| |
0 | 10.3 | 0 | 4.2 | 0 | 85.5 |
To illustrate the effects of the present invention, the same data were used to perform experiments using the optimization feature selection method and the LSVM method, respectively, and the results are shown in table 2.
TABLE 2 comparison of the recognition rates (%)
| Happiness | Sadness and sorrow | Fear of | Aversion to | Is surprised | Generating qi | Average rate of accuracy | |
| Methods of the invention | 95.2 | 92.5 | 87.2 | 91.1 | 90.5 | 85.5 | 90.3 |
| Optimizing feature selection | 76.3 | 67.6 | 55 | 100 | 78 | 100 | 79.5 |
| LSVM | 91.3 | 86.4 | 89.6 | 88.3 | 92.5 | 86.5 | 89.1 |
Tests show that the method has higher accuracy, and the operation steps of the method are simple and easy to implement.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications may be made or equivalents may be substituted for some of the features thereof without departing from the scope of the present invention, and such modifications and substitutions should also be considered as the protection scope of the present invention.
Claims (3)
1. A method of recognizing facial expressions from a sequence of facial images, characterized by: the specific operation steps are as follows:
step one, selecting an image sequence
Selecting image sequences representing six basic expressions such as happiness, sadness, fear, disgust, surprise and anger from a facial expression database, wherein the number of the image sequences of each basic expression is more than 20; selecting m frames of images from each expression image sequence, wherein: m is more than or equal to 10 and is a positive integer; each expression image sequence starts from the neutral expression image and ends from the peak expression image;
step two, identifying facial feature points
On the basis of the first step, identifying facial feature points; the method specifically comprises the following steps:
step 1: sequentially identifying 20 facial feature points in a first frame image in each expression image sequence; the 1 st characteristic point and the 2 nd characteristic point are respectively positioned at the eyebrow positions of the right eyebrow and the left eyebrow; the 3 rd and 4 th characteristic points are respectively positioned at the eyebrow tail positions of the right eyebrow and the left eyebrow; the 5 th and 6 th characteristic points are respectively positioned at the inner canthus positions of the right eye and the left eye; the 7 th and 8 th characteristic points are respectively positioned at the lowest points of the right eye and the left eye; the 9 th and 10 th characteristic points are respectively positioned at the external canthus positions of the right eye and the left eye; the 11 th and 12 th characteristic points are respectively positioned at the highest points of the right eye and the left eye; the 13 th characteristic point and the 14 th characteristic point are respectively positioned at the rightmost position of the nasal wing and the leftmost position of the nasal wing; the 15 th characteristic point is positioned at the tip of the nose; the 16 th characteristic point and the 17 th characteristic point are respectively positioned at the rightmost position of the mouth corner and the leftmost position of the mouth corner; the 18 th and 19 th characteristic points are respectively positioned at the highest point and the lowest point of the intersection of the lip center line and the lip contour line; the 20 th characteristic point is positioned at the lowest point of the intersection of the face central line and the face contour line;
step 2: calculating the positions of the 21 st and 22 nd feature points in the first frame images in each expression image sequence according to the statistical data of the spatial relationship between the eyes and the cheeks, the nose and the cheeks of the human, which is proposed by Farkas in the Anthropomology of the Head and Face; the 21 st and 22 nd characteristic points are respectively positioned at the cheekbone positions of the right cheek and the left cheek;
and 3, step 3: tracking 22 facial feature points in a subsequent frame image in each expression image sequence according to the positions of 22 feature points in a first frame image in each expression image sequence by adopting a Particle filtering tracking method based on likelihood function factorization, which is proposed by Patras et al in the document of Particle filtering with factitious pixels for tracking facial features;
and 4, step 4: adopting an affine transformation method to adjust the position and the size of the face in the image, so that the face in the same image sequence has the same size and the positions are kept consistent;
step three, extracting facial expression characteristics
On the basis of the operation of the second step, facial expression characteristics are extracted from each image in sequence; the method specifically comprises the following steps:
step 1: establishing an Xb-Yb coordinate system, wherein the coordinate system takes the lower left corner of each image as an origin, the horizontal right direction is an Xb axis, and the vertical upward direction is an Yb axis; obtaining the coordinate values (Xb) of 22 characteristic points based on an Xb-Yb coordinate system on each image in sequence according to the positions of the pixels where the 22 characteristic points are located on each imagei，ybi) And the coordinate value (x) of the uppermost point in the human face of each imageorigin，yorigin) Wherein i is 1-22, and i is a positive integer;
step 2: establishing an X-Y coordinate system, wherein the coordinate system takes the uppermost point in the human face of each image as an original point, the horizontal right direction is an X axis, and the vertical upward direction is a Y axis; coordinate values (X) of 22 feature points based on the X-Y coordinate system on each image are obtained by formula 1 and formula 2i，yi)；
xi＝xbi-xorigin (1)
yi＝ybi-yorigin (2)
And 3, step 3: the abscissa displacement amounts of 22 feature points of each image are obtained by formula 3 and formula 4
Displacement of ordinate
Wherein,
respectively the abscissa value and the ordinate value of the corresponding characteristic point of the first frame image in the image sequence where the image is located;
and 4, step 4: obtaining the normalized abscissa displacement quantity delta x of the 22 characteristic points of each image through a formula 5 and a formula 6 in sequenceiThe longitudinal coordinate displacement amount Deltayi；
Wherein x isbase＝x6-x5，ybase＝y6-y5；x5And x6Respectively the abscissa values of the 5 th and 6 th feature points of the image; y is5And y6The longitudinal coordinate values of the 5 th and 6 th characteristic points of the image are respectively;
and 5, step 5: to obtain10 geometric distance features mfv in each image1～mfv10(ii) a The method specifically comprises the following steps:
obtaining an eye openness value mfv according to equation 71：
mfv1＝((y11-y7)+(y12-y8))/2 (7)
Wherein, y7、y8、y11、y12The ordinate values of the 7 th, 8 th, 11 th and 12 th feature points of the image respectively;
obtaining an eye width value mfv according to equation 82：
mfv2＝((x5-x9)+(x10-x6))/2 (8)
Wherein x is5、x6、x9、x10Respectively the abscissa values of the 5 th, 6 th, 9 th and 10 th feature points of the image;
obtaining a eyebrow height value mfv according to equation 93：
mfv3＝(y1+y2)/2 (9)
Wherein, y1、y2The longitudinal coordinate values of the 1 st and 2 nd characteristic points of the image are respectively;
obtaining the brow-tail height value mfv according to equation 104：
mfv4＝(y3+y4)/2 (10)
Wherein, y3、y4Longitudinal coordinate values of the 3 rd and 4 th characteristic points of the image respectively;
obtaining the eyebrow width value mfv according to equation 115：
mfv5＝((x1-x3)+(x4-x2))/2 (11)
Wherein x is1、x2、x3、x4Respectively the abscissa values of the 1 st, 2 nd, 3 th and 4 th feature points of the image;
the mouth opening degree value mfv is obtained according to equation 126：
mfv6＝y18-y19 (12)
Wherein, y18、y19Longitudinal coordinate values of 18 th and 19 th characteristic points of the image respectively;
the mouth width value mfv is obtained according to equation 137：
mfv7＝x17-x16 (13)
Wherein x is16、x17Respectively the abscissa values of the 16 th and 17 th feature points of the image;
obtaining a nose tip-mouth angle distance value mfv according to equation 148：
mfv8＝((y15-y16)+(y15-y17))/2 (14)
Wherein, y15、y16、y17The ordinate values of the 15 th, 16 th and 17 th feature points of the image are respectively;
obtaining an eye-cheek distance value mfv according to equation 159；
mfv9＝(((y11+y7)/2-y21)+((y12+y8)/2-y22))/2 (15)
Wherein, y21、y22Longitudinal coordinate values of 21 st and 22 th characteristic points of the image respectively;
nasal tip-chin distance value mfv obtained according to equation 1610：
mfv10＝y15-y20 (16)
Wherein, y15、y20Longitudinal coordinate values of 15 th and 20 th characteristic points of the image respectively;
and 6, step 6: the 10 geometric distance features mfv in each image are shown by equation 171～mfv10Carrying out normalization processing;
wherein j is 1-10, and j is a positive integer;
the image is the corresponding geometric distance characteristic in the first frame image in the image sequence where the image is located;
and 7, step 7: using 10 geometric distance features mfv of each image respectively1～mfv10And the abscissa displacement delta x obtained after normalization processing of 22 characteristic points of each imageiThe longitudinal coordinate displacement amount DeltayiForming a 54-dimensional column vector z representing expression information of a facial imagek(ii) a Wherein: z is a radical ofk∈R54K is more than or equal to 1 and less than or equal to m, and R represents a real number);
and 8, step 8: using the feature matrix Z ═ Z1，z2，...，zm}∈R54×mRepresenting a sequence of emoticons; wherein z is1Denotes neutral expression, zmRepresents a peak expression;
step four, classifying the trial image sequence to be tested
On the basis of the third step, classifying the image sequence to be tested by adopting typical correlation analysis; the method specifically comprises the following steps:
step 1: randomly dividing the image sequence of each basic expression selected in the step one into 2 parts, wherein one part is used as training data, and the other part is used as test data; the number of the training data is Q, Q is more than or equal to 20, and Q is a positive integer; one piece of training data is an expression image sequence; one piece of test data is also an expression image sequence;
step 2: aiming at the training data obtained in the step 1, a typical correlation discriminant analysis method provided by T.K. Kim et al in the literature, "discrete Learning and Recognition of Image Set Classes using canoni cal Correlations" is adopted for processing to obtain a transformation matrix T ∈ R54×n，n＜54，And n is a positive integer; and then, converting the feature matrix Z of the images in all the expression image sequences selected in the step one by using a transformation matrix T to obtain Z' ═ TTZ；
And 3, step 3: randomly selecting an expression image sequence from the test data in the step 1, and calculating the sum of typical correlation coefficients of a feature matrix Z 'of the expression image sequence and a feature matrix Z' of each training data;
and 4, step 4: on the basis of the result of the step 3, respectively calculating the average value of the sum of the typical correlation coefficients of the expression image sequence and each basic expression; selecting the expression corresponding to the minimum value in the 6 average values as a classification result;
through the steps, the expression recognition of the image sequence to be tested can be completed.
2. A method of recognizing facial expressions from a sequence of facial images as claimed in claim 1, wherein: step two, step 1, the method for identifying 20 facial feature points in the first frame image in each expression image sequence includes but is not limited to: manual marking; secondly, a Gabor feature-based enhanced classifier method proposed by Vukdadinovic et al in the document 'full automatic facial feature point detection using Gabor feature based classifiers' is adopted to realize the automatic positioning of 20 facial feature points.
3. A method of recognizing facial expressions from a sequence of facial images as claimed in claim 1, wherein: step two and step 4, the position and the size of the face in the image are adjusted by adopting an affine transformation method, so that the face in the same image sequence has the same size and the positions are kept consistent, specifically:
firstly, keeping a connecting line of two inner eye corner points in a first frame of image in each image sequence horizontal; then, according to the positions of 3 points, namely two inner eye corner points in the first frame of image in the image sequence and the uppermost point in the person, mapping and normalizing 22 face feature points in the rest frames; after affine transformation processing, the human faces of all the images in the same image sequence are equal in size, and the positions of 3 points, namely the two inner eye corner points and the uppermost point in the human, are consistent with the positions of the 3 points in the first frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102185432A CN101908149A (en) | 2010-07-06 | 2010-07-06 | Method for identifying facial expressions from human face image sequence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010102185432A CN101908149A (en) | 2010-07-06 | 2010-07-06 | Method for identifying facial expressions from human face image sequence |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101908149A true CN101908149A (en) | 2010-12-08 |
Family
ID=43263604
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010102185432A Pending CN101908149A (en) | 2010-07-06 | 2010-07-06 | Method for identifying facial expressions from human face image sequence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101908149A (en) |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102945361A (en) * | 2012-10-17 | 2013-02-27 | 北京航空航天大学 | Facial expression recognition method based on feature point vectors and texture deformation energy parameter |
| CN103400145A (en) * | 2013-07-19 | 2013-11-20 | 北京理工大学 | Voice-vision fusion emotion recognition method based on hint nerve networks |
| CN103679143A (en) * | 2013-12-03 | 2014-03-26 | 北京航空航天大学 | Method for capturing facial expressions in real time without supervising |
| CN103971137A (en) * | 2014-05-07 | 2014-08-06 | 上海电力学院 | Three-dimensional dynamic facial expression recognition method based on structural sparse feature study |
| CN103996029A (en) * | 2014-05-23 | 2014-08-20 | 安庆师范学院 | Expression similarity measuring method and device |
| CN104866807A (en) * | 2014-02-24 | 2015-08-26 | 腾讯科技（深圳）有限公司 | Face positioning method and system |
| CN105559804A (en) * | 2015-12-23 | 2016-05-11 | 上海矽昌通信技术有限公司 | Mood manager system based on multiple monitoring |
| CN105740688A (en) * | 2016-02-01 | 2016-07-06 | 腾讯科技（深圳）有限公司 | Unlocking method and device |
| WO2017045404A1 (en) * | 2015-09-16 | 2017-03-23 | Intel Corporation | Facial expression recognition using relations determined by class-to-class comparisons |
| CN106650555A (en) * | 2015-11-02 | 2017-05-10 | 苏宁云商集团股份有限公司 | Real person verifying method and system based on machine learning |
| CN106940792A (en) * | 2017-03-15 | 2017-07-11 | 中南林业科技大学 | The human face expression sequence truncation method of distinguished point based motion |
| CN107203734A (en) * | 2016-03-17 | 2017-09-26 | 掌赢信息科技（上海）有限公司 | A kind of method and electronic equipment for obtaining mouth state |
| CN107341460A (en) * | 2017-06-26 | 2017-11-10 | 北京小米移动软件有限公司 | Face tracking method and device |
| CN108074203A (en) * | 2016-11-10 | 2018-05-25 | 中国移动通信集团公司 | A kind of teaching readjustment method and apparatus |
| CN108108651A (en) * | 2016-11-25 | 2018-06-01 | 广东亿迅科技有限公司 | The non-wholwe-hearted driving detection method of driver and system based on video human face analysis |
| CN108540863A (en) * | 2018-03-29 | 2018-09-14 | 武汉斗鱼网络科技有限公司 | Barrage setting method, storage medium, equipment and system based on human face expression |
| CN108875335A (en) * | 2017-10-23 | 2018-11-23 | 北京旷视科技有限公司 | The method and authenticating device and non-volatile memory medium of face unlock and typing expression and facial expressions and acts |
| CN109034099A (en) * | 2018-08-14 | 2018-12-18 | 华中师范大学 | A kind of expression recognition method and device |
| CN109063679A (en) * | 2018-08-24 | 2018-12-21 | 广州多益网络股份有限公司 | A kind of human face expression detection method, device, equipment, system and medium |
| CN109255322A (en) * | 2018-09-03 | 2019-01-22 | 北京诚志重科海图科技有限公司 | A kind of human face in-vivo detection method and device |
| CN109409273A (en) * | 2018-10-17 | 2019-03-01 | 中联云动力(北京)科技有限公司 | A kind of motion state detection appraisal procedure and system based on machine vision |
| CN105095827B (en) * | 2014-04-18 | 2019-05-17 | 汉王科技股份有限公司 | Facial expression recognition device and method |
| CN109784123A (en) * | 2017-11-10 | 2019-05-21 | 浙江思考者科技有限公司 | The analysis and judgment method of real's expression shape change |
| CN109948569A (en) * | 2019-03-26 | 2019-06-28 | 重庆理工大学 | A kind of three-dimensional hybrid expression recognition method using particle filter frame |
| CN110909680A (en) * | 2019-11-22 | 2020-03-24 | 咪咕动漫有限公司 | Facial expression recognition method and device, electronic equipment and storage medium |
| CN110941332A (en) * | 2019-11-06 | 2020-03-31 | 北京百度网讯科技有限公司 | Expression driving method and device, electronic equipment and storage medium |
| CN111382648A (en) * | 2018-12-30 | 2020-07-07 | 广州市百果园信息技术有限公司 | Method, device and equipment for detecting dynamic facial expression and storage medium |
| CN111626253A (en) * | 2020-06-02 | 2020-09-04 | 上海商汤智能科技有限公司 | Expression detection method and device, electronic equipment and storage medium |
| CN111951930A (en) * | 2020-08-19 | 2020-11-17 | 陈霄 | Emotion identification system based on big data |
| CN112132084A (en) * | 2020-09-29 | 2020-12-25 | 上海松鼠课堂人工智能科技有限公司 | Eye micro-expression analysis method and system based on deep learning |
-
2010
- 2010-07-06 CN CN2010102185432A patent/CN101908149A/en active Pending
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102945361B (en) * | 2012-10-17 | 2016-10-05 | 北京航空航天大学 | Feature based point vector and the facial expression recognizing method of texture deformation energy parameter |
| CN102945361A (en) * | 2012-10-17 | 2013-02-27 | 北京航空航天大学 | Facial expression recognition method based on feature point vectors and texture deformation energy parameter |
| CN103400145B (en) * | 2013-07-19 | 2016-08-10 | 北京理工大学 | Voice based on clue neutral net-vision merges emotion identification method |
| CN103400145A (en) * | 2013-07-19 | 2013-11-20 | 北京理工大学 | Voice-vision fusion emotion recognition method based on hint nerve networks |
| CN103679143A (en) * | 2013-12-03 | 2014-03-26 | 北京航空航天大学 | Method for capturing facial expressions in real time without supervising |
| CN103679143B (en) * | 2013-12-03 | 2017-02-15 | 北京航空航天大学 | Method for capturing facial expressions in real time without supervising |
| CN104866807A (en) * | 2014-02-24 | 2015-08-26 | 腾讯科技（深圳）有限公司 | Face positioning method and system |
| CN104866807B (en) * | 2014-02-24 | 2019-09-13 | 腾讯科技（深圳）有限公司 | A kind of Face detection method and system |
| CN105095827B (en) * | 2014-04-18 | 2019-05-17 | 汉王科技股份有限公司 | Facial expression recognition device and method |
| CN103971137A (en) * | 2014-05-07 | 2014-08-06 | 上海电力学院 | Three-dimensional dynamic facial expression recognition method based on structural sparse feature study |
| CN103996029A (en) * | 2014-05-23 | 2014-08-20 | 安庆师范学院 | Expression similarity measuring method and device |
| CN103996029B (en) * | 2014-05-23 | 2017-12-05 | 安庆师范学院 | Expression method for measuring similarity and device |
| US10339369B2 (en) | 2015-09-16 | 2019-07-02 | Intel Corporation | Facial expression recognition using relations determined by class-to-class comparisons |
| WO2017045157A1 (en) * | 2015-09-16 | 2017-03-23 | Intel Corporation | Facial expression recognition using relations determined by class-to-class comparisons |
| WO2017045404A1 (en) * | 2015-09-16 | 2017-03-23 | Intel Corporation | Facial expression recognition using relations determined by class-to-class comparisons |
| CN106650555A (en) * | 2015-11-02 | 2017-05-10 | 苏宁云商集团股份有限公司 | Real person verifying method and system based on machine learning |
| CN105559804A (en) * | 2015-12-23 | 2016-05-11 | 上海矽昌通信技术有限公司 | Mood manager system based on multiple monitoring |
| CN105740688A (en) * | 2016-02-01 | 2016-07-06 | 腾讯科技（深圳）有限公司 | Unlocking method and device |
| CN107203734A (en) * | 2016-03-17 | 2017-09-26 | 掌赢信息科技（上海）有限公司 | A kind of method and electronic equipment for obtaining mouth state |
| CN108074203A (en) * | 2016-11-10 | 2018-05-25 | 中国移动通信集团公司 | A kind of teaching readjustment method and apparatus |
| CN108108651A (en) * | 2016-11-25 | 2018-06-01 | 广东亿迅科技有限公司 | The non-wholwe-hearted driving detection method of driver and system based on video human face analysis |
| CN106940792A (en) * | 2017-03-15 | 2017-07-11 | 中南林业科技大学 | The human face expression sequence truncation method of distinguished point based motion |
| CN107341460B (en) * | 2017-06-26 | 2022-04-22 | 北京小米移动软件有限公司 | Face tracking method and device |
| CN107341460A (en) * | 2017-06-26 | 2017-11-10 | 北京小米移动软件有限公司 | Face tracking method and device |
| CN108875335A (en) * | 2017-10-23 | 2018-11-23 | 北京旷视科技有限公司 | The method and authenticating device and non-volatile memory medium of face unlock and typing expression and facial expressions and acts |
| US10922533B2 (en) | 2017-10-23 | 2021-02-16 | Beijing Kuangshi Technology Co., Ltd. | Method for face-to-unlock, authentication device, and non-volatile storage medium |
| CN108875335B (en) * | 2017-10-23 | 2020-10-09 | 北京旷视科技有限公司 | Method for unlocking human face and inputting expression and expression action, authentication equipment and nonvolatile storage medium |
| CN109784123A (en) * | 2017-11-10 | 2019-05-21 | 浙江思考者科技有限公司 | The analysis and judgment method of real's expression shape change |
| CN108540863B (en) * | 2018-03-29 | 2021-03-12 | 武汉斗鱼网络科技有限公司 | Bullet screen setting method, storage medium, equipment and system based on facial expressions |
| CN108540863A (en) * | 2018-03-29 | 2018-09-14 | 武汉斗鱼网络科技有限公司 | Barrage setting method, storage medium, equipment and system based on human face expression |
| CN109034099A (en) * | 2018-08-14 | 2018-12-18 | 华中师范大学 | A kind of expression recognition method and device |
| CN109034099B (en) * | 2018-08-14 | 2021-07-13 | 华中师范大学 | Expression recognition method and device |
| CN109063679A (en) * | 2018-08-24 | 2018-12-21 | 广州多益网络股份有限公司 | A kind of human face expression detection method, device, equipment, system and medium |
| CN109255322A (en) * | 2018-09-03 | 2019-01-22 | 北京诚志重科海图科技有限公司 | A kind of human face in-vivo detection method and device |
| CN109409273A (en) * | 2018-10-17 | 2019-03-01 | 中联云动力(北京)科技有限公司 | A kind of motion state detection appraisal procedure and system based on machine vision |
| CN111382648A (en) * | 2018-12-30 | 2020-07-07 | 广州市百果园信息技术有限公司 | Method, device and equipment for detecting dynamic facial expression and storage medium |
| CN109948569B (en) * | 2019-03-26 | 2022-04-22 | 重庆理工大学 | Three-dimensional mixed expression recognition method using particle filter framework |
| CN109948569A (en) * | 2019-03-26 | 2019-06-28 | 重庆理工大学 | A kind of three-dimensional hybrid expression recognition method using particle filter frame |
| CN110941332A (en) * | 2019-11-06 | 2020-03-31 | 北京百度网讯科技有限公司 | Expression driving method and device, electronic equipment and storage medium |
| CN110909680A (en) * | 2019-11-22 | 2020-03-24 | 咪咕动漫有限公司 | Facial expression recognition method and device, electronic equipment and storage medium |
| CN111626253A (en) * | 2020-06-02 | 2020-09-04 | 上海商汤智能科技有限公司 | Expression detection method and device, electronic equipment and storage medium |
| CN111951930A (en) * | 2020-08-19 | 2020-11-17 | 陈霄 | Emotion identification system based on big data |
| CN111951930B (en) * | 2020-08-19 | 2021-10-15 | 中食安泓(广东)健康产业有限公司 | Emotion identification system based on big data |
| CN112132084A (en) * | 2020-09-29 | 2020-12-25 | 上海松鼠课堂人工智能科技有限公司 | Eye micro-expression analysis method and system based on deep learning |
| CN112132084B (en) * | 2020-09-29 | 2021-07-09 | 上海松鼠课堂人工智能科技有限公司 | Eye micro-expression analysis method and system based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101908149A (en) | Method for identifying facial expressions from human face image sequence | |
| Li et al. | A comprehensive survey on 3D face recognition methods | |
| Agrawal et al. | Facial expression detection techniques: based on Viola and Jones algorithm and principal component analysis | |
| CN105894047B (en) | A kind of face classification system based on three-dimensional data | |
| Poon et al. | Performance evaluation and comparison of PCA Based human face recognition methods for distorted images | |
| Li et al. | Overview of principal component analysis algorithm | |
| Wang et al. | Feature representation for facial expression recognition based on FACS and LBP | |
| Liu et al. | Gait recognition based on outermost contour | |
| Bedre et al. | Comparative study of face recognition techniques: a review | |
| CN106096517A (en) | A kind of face identification method based on low-rank matrix Yu eigenface | |
| JP2000030065A (en) | Pattern recognizing device and its method | |
| Angadi et al. | Face recognition through symbolic modeling of face graphs and texture | |
| CN107220627A (en) | Pose-varied face recognition method based on cooperation fuzzy mean discriminatory analysis | |
| Bekhouche | Facial soft biometrics: extracting demographic traits | |
| Kim et al. | On-line learning of mutually orthogonal subspaces for face recognition by image sets | |
| Zhou et al. | Real-time gender recognition based on eigen-features selection from facial images | |
| Li et al. | A group of facial normal descriptors for recognizing 3D identical twins | |
| Chang et al. | Applications of Block Linear Discriminant Analysis for Face Recognition. | |
| Sagonas et al. | Back to the future: A fully automatic method for robust age progression | |
| Praseeda Lekshmi et al. | Analysis of facial expressions from video images using PCA | |
| Belle | Detection and recognition of human faces using random forests for a mobile robot | |
| CN111444860A (en) | Expression recognition method and system | |
| Lo Presti et al. | Ensemble of Hankel matrices for face emotion recognition | |
| Wang et al. | Expression robust three-dimensional face recognition based on Gaussian filter and dual-tree complex wavelet transform | |
| Naruniec | A survey on facial features detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20101208 |