research-article

3D facial expression modeling based on facial landmarks in single image

Authors:

Mingquan ZhouAuthors Info & Claims

Volume 355, Issue C

Pages 155 - 167

https://doi.org/10.1016/j.neucom.2019.04.050

Published: 25 August 2019 Publication History

Abstract

Facial expression modeling is important for many applications such as human emotional analysis and facial animation. Generally, facial expression modeling from single 2D facial image is difficult. Different head poses and scales of facial data in images affect the accuracy of the modeling results. We propose a new 3D facial expression modeling method which is based on facial landmarks from single image. Using the facial landmarks, expression modeling can be processed in Kendall shape space. The Kendall shape space is mathematic space, the facial expression modeling process in Kendall shape space can be regarded as a geodesic path search between different faces. The modeling result is more accurate. The 3D facial expression modeling result is convenient to obtain from 2D facial image with different head poses. In experiments, we show the 3D facial expression modeling performance by our method, which include expression editing and evaluation in public facial database: JAFFE, LFW, Helen and RAF-DB.

References

[1]

A. Bas, W.A.P. Smith, T. Bolkart, S. Wuhrer, Fitting a 3D morphable model to edges: a comparison between hard and soft correspondences, Proceedings of the Asian Conference on Computer Vision, 2016, pp. 377–391.

[2]

C. Cao, Y. Weng, S. Zhou, Y. Tong, K. Zhou, Facewarehouse: a 3D facial expression database for visual computing, IEEE Trans. Vis. Comput. Graph. 20 (3) (2014) 413–425.

[3]

T.F. Cootes, C.J. Taylor, D.H. Cooper, J. Graham, Active shape models-their training and application, Comput. Vis. Image Underst. 61 (1) (1995) 38–59.

Digital Library

[4]

G.J. Edwards, T.F. Cootes, C.J. Taylor, Face recognition using active appearance models, Proceedings of the IEEE Signal Processing and Communications Applications Conference, 1998, pp. 940–943.

[5]

T. Hong, Y.B. Lee, Y.G. Kim, H. Kim, Facial expression recognition using active appearance model, Proceedings of the International Symposium on Neural Networks, 2006, pp. 69–76.

[6]

C.C. Hsieh, M.K. Jiang, A facial expression classification system based on active shape model and support vector machine, Proceedings of the International Symposium on Computer Science and Society, 2011, pp. 311–314.

[7]

C.L. Huang, Y.M. Huang, Facial expression recognition using model-based feature extraction and action parameters classification, J. Vis. Commun. Image Represent. 8 (3) (1997) 278–290.

[8]

P.S. Penev, L. Sirovich, The global dimensionality of face space, Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, 2000, p. 264.

[9]

X.W. Chen, T. Huang, Facial expression recognition: a clustering-based approach, Pattern Recognit. Lett. 24 (9) (2003) 1295–1302.

[10]

P.P. Paul, M. Gavrilova, PCA based geometric modeling for automatic face detection, Proceedings of the International Conference on Computational Science and ITS Applications, 2011, pp. 33–38.

[11]

D. Decarlo, D. Metaxas, M. Stone, An anthropometric face model using variational techniques, Proceedings of the Conference on Computer Graphics and Interactive Techniques, 1998, pp. 67–74.

[12]

K.S. Lee, K.H. Wong, S.H. Or, Y.F. Fung, 3D face modeling from perspective-views and contour-based generic-model, Real-Time Imaging 7 (2) (2001) 173–182.

[13]

N.A. A, A.M. Mohamed, Automatic facial feature extraction and 3d face modeling using two orthogonal views with application to 3D face recognition, Pattern Recognit. 12 (38) (2005) 2549–2563.

[14]

J. Wang, L. Yin, X. Wei, Y. Sun, 3d facial expression recognition based on primitive surface feature distribution, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2006, pp. 1399–1406.

[15]

H. Soyel, H. Demirel, Facial expression recognition using 3d facial feature distances, Proceedings of the International Conference Image Analysis and Recognition, 2007, pp. 831–838.

[16]

W. Peng, C. Xu, Z. Feng, 3d face modeling based on structure optimization and surface reconstruction with b-spline, Neurocomputing 179 (12) (2016) 228–237.

[17]

S. Zhan, L. Chang, J. Zhao, T. Kurihara, H. Du, Y. Tang, J. Cheng, Real-time 3d face modeling based on 3d face imaging, Neurocomputing 252 (2017) 42–48.

[18]

Y. Zhenbo, L. Guangcan, L. Qingshan, D. Jiankang, Spatio-temporal convolutional features with nested LSTM for facial expression recognition, Neurocomputing 317 (2018) 50–57.

[19]

J. Wu, Z. Lin, W. Zheng, H. Zha, Locality-constrained linear coding based bi-layer model for multi-view facial expression recognition, Neurocomputing 239 (C) (2017) 143–152.

[20]

V. Blanz, T. Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the Conference on Computer Graphics and Interactive Techniques, 1999, pp. 187–194.

[21]

V. Blanz, T. Vetter, Face recognition based on fitting a 3D morphable model, IEEE Trans. Pattern Anal. Mach. Intell. 25 (9) (2003) 1063–1074.

Digital Library

[22]

P. Paysan, R. Knothe, B. Amberg, S. Romdhani, T. Vetter, A 3d face model for pose and illumination invariant face recognition, Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance, 2009, pp. 296–301.

[23]

J.P. Mena-Chalco, I. MacÃªdo, L. Velho, R.M.C. Jr, 3D face computational photography using PCA spaces., Vis. Comput. 25 (10) (2009) 899–909.

[24]

F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, Synthesizing realistic facial expressions from photographs, Proceedings of the ACM SIGGRAPH, 2005, p. 9.

[25]

J. Booth, A. Roussos, A. Ponniah, D. Dunaway, S. Zafeiriou, Large scale 3D morphable models, Int. J. Comput. Vis. 126 (2–4) (2018) 1–22.

[26]

F. Duan, D. Huang, T. Yun, L. Ke, Z. Wu, M. Zhou, 3d face reconstruction from skull by regression modeling in shape parameter spaces, Neurocomputing 151 (2015) 674–682.

[27]

X. Lu, A.K. Jain, Deformation modeling for robust 3d face matching, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2006, pp. 1377–1383.

[28]

A.E. Ichim, S. Bouaziz, M. Pauly, Dynamic 3D avatar creation from hand-held video input, ACM Trans. Graph. 34 (4) (2015) 1–14.

[29]

H. Jin, X. Wang, Z. Zhong, J. Hua, Robust 3d face modeling and reconstruction from frontal and side images, Comput. Aided Geom. Des. 50 (2017) 1–13.

[30]

D. Vlasic, M. Brand, H. Pfister, Face transfer with multilinear models, Proceedings of the ACM SIGGRAPH, 2005, pp. 426–433.

[31]

I. Mpiperis, S. Malassiotis, M.G. Strintzis, Bilinear models for 3-D face and facial expression recognition, IEEE Trans. Inf. Forensics Secur. 3 (3) (2008) 498–511.

[32]

C. Cao, Q. Hou, K. Zhou, Displaced dynamic expression regression for real-time facial tracking and animation, ACM Trans. Graph. 33 (4) (2014) 1–10.

[33]

X. Gao, C. Tian, Multi-view face recognition based on tensor subspace analysis and view manifold modeling, Neurocomputing 72 (16–18) (2009) 3742–3750.

[34]

X. Song, Z.H. Feng, X. Yang, X. Wu, J. Yang, Towards multi-scale fuzzy sparse discriminant analysis using local third-order tensor model of face images, Neurocomputing 185 (C) (2016) 53–63.

[35]

S. Elaiwat, M. Bennamoun, F. Boussaid, A spatio-temporal RBM-based model for facial expression recognition, Pattern Recognit. 49 (C) (2015) 152–161.

[36]

T. Zhang, W. Zheng, Z. Cui, Y. Zong, J. Yan, K. Yan, A deep neural network-driven feature learning method for multi-view facial expression recognition, IEEE Trans. Multimed. 18 (12) (2016) 2528–2536.

[37]

A.T. Lopes, E.D. Aguiar, A.F.D. Souza, T. Oliveira-Santos, Facial expression recognition with convolutional neural networks: coping with few data and the training sample order, Pattern Recognit. 61 (2016) 610–628.

Digital Library

[38]

H. Li, J. Sun, Z. Xu, L. Chen, Multimodal 2d+3d facial expression recognition with deep fusion convolutional neural network, IEEE Trans. Multimed. 19 (12) (2017) 2816–2831.

[39]

A. Brunton, A. Salazar, T. Bolkart, S. Wuhrer, Review of statistical shape spaces for 3d data with comparative analysis for human faces, Comput. Vis. Image Underst. 128 (11) (2014) 1–17.

[40]

S. Kurtek, H. Drira, A comprehensive statistical framework for elastic shape analysis of 3d faces, Comput. Graph. 51 (2015) 52–59.

[41]

T. Alashkar, B.B. Amor, M. Daoudi, S. Berretti, A Grassmann framework for 4d facial shape analysis, Pattern Recognit. 57 (C) (2016) 21–30.

[42]

A. Patel, W.A.P. Smith, Manifold-based constraints for operations in face space, Pattern Recognit. 52 (2016) 206–217.

[43]

D.G. Kendall, Shape manifolds, procrustean metrics, and complex projective spaces, Bull. Lond. Math. Soc. 16 (2) (1985) 81–121.

[44]

V. Kazemi, J. Sullivan, One millisecond face alignment with an ensemble of regression trees, Proceedings of the Computer Vision and Pattern Recognition, 2014, pp. 1867–1874.

[45]

F.M. Sukno, J.L. Waddington, P.F. Whelan, 3-d facial landmark localization with asymmetry patterns and shape regression from incomplete local features., IEEE Trans. Cybern. 45 (9) (2014) 1717–1730.

[46]

S. Li, W. Deng, J. Du, Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2017, pp. 2584–2593.

[47]

S. Li, W. Deng, Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition, IEEE Trans. Image Process. 28 (1) (2019) 356–370.

Digital Library

Cited By

Luo BZhu JYang TZhao SHu CZhao XGao YWilliams BChen YNeville J(2023)Learning deep hierarchical features with spatial regularization for one-class facial expression recognitionProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v37i5.25749(6065-6073)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1609/aaai.v37i5.25749
Zhu JLuo BSun ATan JZhao XGao YEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Variance-Aware Bi-Attention Expression Transformer for Open-Set Facial Expression Recognition in the WildProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612546(862-870)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612546
Zhu JLuo BYang TWang ZZhao XGao Y(2023)Knowledge Conditioned Variational Learning for One-Class Facial Expression RecognitionIEEE Transactions on Image Processing10.1109/TIP.2023.329377532(4010-4023)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3293775
Show More Cited By

Index Terms

3D facial expression modeling based on facial landmarks in single image
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Shape modeling

Index terms have been assigned to the content through auto-classification.

Recommendations

Pose-Invariant Facial Expression Recognition Based on 3D Face Reconstruction and Synthesis from a Single 2D Image
ICPR '14: Proceedings of the 2014 22nd International Conference on Pattern Recognition

In this paper, a novel method is proposed for person-independent pose-invariant facial expression recognition based on 3D face reconstruction from only 2D frontal images in a training set. A 3D Facial Expression Generic Elastic Model (3D FE-GEM) is ...
Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Pose-Robust Facial Expression Recognition Using View-Based 2D + 3D AAM

This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Neurocomputing

Neurocomputing Volume 355, Issue C

Aug 2019

222 pages

ISSN:0925-2312

Issue’s Table of Contents

Copyright © 2019.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 25 August 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Luo BZhu JYang TZhao SHu CZhao XGao YWilliams BChen YNeville J(2023)Learning deep hierarchical features with spatial regularization for one-class facial expression recognitionProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v37i5.25749(6065-6073)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1609/aaai.v37i5.25749
Zhu JLuo BSun ATan JZhao XGao YEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Variance-Aware Bi-Attention Expression Transformer for Open-Set Facial Expression Recognition in the WildProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612546(862-870)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612546
Zhu JLuo BYang TWang ZZhao XGao Y(2023)Knowledge Conditioned Variational Learning for One-Class Facial Expression RecognitionIEEE Transactions on Image Processing10.1109/TIP.2023.329377532(4010-4023)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3293775
Tan S(2022)Animation Image Art Design Mode Using 3D Modeling TechnologyWireless Communications & Mobile Computing10.1155/2022/65774612022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/6577461
Allaert BWard IBilasco IDjeraba CBennamoun M(2022)A comparative study on optical flow for facial expression analysisNeurocomputing10.1016/j.neucom.2022.05.077500:C(434-448)Online publication date: 21-Aug-2022
https://dl.acm.org/doi/10.1016/j.neucom.2022.05.077

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents