Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

Facial expression recognition based on deep learning

Published: 01 March 2022 Publication History

Highlights

Autonomous driving, virtual reality and all kinds of robots integrated into our life rely on facial expression recognition technology.
Facial expression recognition and computer vision is based on deep learning technology and convolutional neural network.
Whether it is two-stage target detection or single-stage target detection, performance of algorithm is measured by detection speed and accuracy.
Large variety of training data with accurate expression tags can fundamentally improve expression recognition rate.

Abstract

Background and objective

Facial expression recognition technology will play an increasingly important role in our daily life. Autonomous driving, virtual reality and all kinds of robots integrated into our life depend on the development of facial expression recognition technology. Many tasks in the field of computer vision are based on deep learning technology and convolutional neural network. The paper proposes an occluded expression recognition model based on the generated countermeasure network. The model is divided into two modules, namely, occluded face image restoration and face recognition.

Methods

Firstly, this paper summarizes the research status of deep facial expression recognition methods in recent ten years and the development of related facial expression database. Then, the current facial expression recognition methods based on deep learning are divided into two categories: Static facial expression recognition and dynamic facial expression recognition. The two methodswill be introduced and summarized respectively. Aiming at the advanced deep expression recognition algorithms in the field, the performance of these algorithms on common expression databases is compared, and the strengths and weaknesses of these algorithms are analyzed in detail.

Discussion and results

As the task of facial expression recognition is gradually transferred from the controlled laboratory environment to the challenging real-world environment, with the rapid development of deep learning technology, deep neural network can learn discriminative features, and is gradually applied to automatic facial expression recognition task. The current deep facial expression recognition system is committed to solve the following two problems: (1) Overfitting due to lack of sufficient training data; (2) In the real world environment, other variables that have nothing to do with expression bring interference problems.

Conclusion

From the perspective of algorithm, combining other expression models, such as facial action unit model and pleasure arousal dimension model, as well as other multimodal models, such as audio mode, 3D face depth information and human physiological information, can make expression recognition more practical.

References

[1]
C. He, Overview of face recognition technology, Intell. Comput. Appl. 6 (5) (2016) 112–114.
[2]
M.H. Zhu, S.T. Li, Y. Hua, Facial expression recognition method based on sparse representation, Pattern Recognit. Artif. Intell. 27 (8) (2014) 708–712.
[3]
Y.L. Xue, X. Mao, C. Catalin-Daniel, Robust facial expression recognition method under occlusion conditions, J. Beijing Univ. Aeronaut. Astronaut. 36 (4) (2010) 429–433.
[4]
Y. Chenglin, P. Hailong, Face recognition framework based on effective computing and adversarial neural network and its implementation in machine vision for social robots, Comput. Electr. Eng. 92 (2021) 92.
[5]
Y. Sun, X. Wang, X. Tang, Deep learning face representation from predicting 10,000 classes, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2014, pp. 1891–1898.
[6]
Y. Sun, Y. Chen, X. Wang, et al., Deep learning face representation by joint identification verification, in: Proceedings of the Advances in Neural Information Processing Systems, Barcelona, SPAIN, 2014, pp. 1988–1996.
[7]
Y. Sun, X. Wang, X. Tang, Deeply learned face representations are sparse, selective, and robust, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2015, pp. 2892–2900.
[8]
F. Schroff, D. Kalenichenko, J. Philbin, Facenet: a unified embedding for face recognition and clustering, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2015, pp. 815–823.
[9]
Y. Taigman, M. Yang, M.A. Ranzato, et al., Deepface: closing the gap to human-level performance in face verification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2014, pp. 1701–1708.
[10]
Y. Wen, K. Zhang, Z. Li, et al., A discriminative feature learning approach for deep face recognition, European Conference on Computer Vision, Springer, Cham, Berlin, 2016, pp. 499–515.
[11]
W. Liu, Y. Wen, Z. Yu, et al., Large-margin softmax loss for convolutional neural networks, in: Proceedings of the International Conference on Machine Learning, 2, ICML, Stockholm., 2016, p. 7.
[12]
F. Wang, J. Cheng, W. Liu, et al., Additive margin softmax for face verification, IEEE Signal Processing Letters, 25, PISCATAWAY, USA, 2018, pp. 926–930.
[13]
H. Wang, Y. Wang, Z. Zhou, et al., CosFace: large margin cosine loss for deep face recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2018, pp. 5265–5274.
[14]
S. Chen, Y. Liu, X. Gao, et al., MobileFaceNets: efficient CNNs for accurate real-time face verification on mobile devices, Chinese Conference on Biometric Recognition, Springer, Cham, Berlin, 2018, pp. 428–438.
[15]
M. Sandler, A. Howard, M. Zhu, et al., MobileNetV2: inverted residuals and linear bottlenecks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2018, pp. 4510–4520.
[16]
R. Girshick, J. Donahue, T. Darrell, et al., R-CNN for object detection, in: Proceedings of the IEEE Conference, 2014.
[17]
Zhu G., Porikli F., Li H. Tracking randomly moving objects on edge box proposals. arXiv preprint arXiv: 1507.08085, 2015.
[18]
J.R.R. Uijlings, K.E.A. Van De Sande, T. Gevers, et al., Selective search for object recognition, Int. J. Comput. Vis. 104 (2) (2013) 154–171.
[19]
C. Lee, J.H. Kim, K.W. Oh, Comparison of faster R-CNN models for object detection[C]//2016 16th international conference on control, in: Automation and systems (iccas), IEEE, 2016, pp. 107–110.
[20]
S. Ren, K. He, R. Girshick, et al., Faster R-CNN: towards real-time object detection with region proposal networks, IEEE transactions on pattern analysis and machine intelligence 39 (6) (2016) 1137–1149.
[21]
X.X. Zhang, X. Zhu, Moving vehicle detection in aerial infrared image sequences via fast image registration and improved YOLOv3 network, Int. J. Remote Sens. 41 (11) (2020) 4312–4335.
[22]
Nguyen H.T. Contributions to facial feature extraction for face recognition. université de grenoble, 2014.
[23]
Z. Tang, G. Zhao, T. Ouyang, Two-phase deep learning model for short-term wind direction forecasting, Renew. Energy 173 (2021) 1005–1016,.
[24]
C. Lunerti, R.M. Guest, R. Blanco-Gonzalo, et al., Environmental effects on face recognition in smartphones, in: Proceedings of the 51st IEEE International Carnahan Conference on Security Technology, IEEE, 2017.
[25]
Z.H. Tang, Y.Y. Li, X.Y. Chai, H.Y. Zhang, S.X. Cao, Adaptive nonlinear model predictive control of NOx emissions under load constraints in power plant boilers, J. Chem. Eng. Jpn. 53 (1) (2020) 36–44,.
[26]
L.C. Yan, B. Yoshua, H. Geoffrey, Deep learning, Nature 521 (7553) (2015) 436–444.
[27]
Simonyan K., Zisserman A., Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[28]
Knyazev B., Shvetsov R., Efremova N., et al. Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video. arXiv preprint arXiv:1711.04598, 2017.
[29]
Yue-Hei Ng J, Hausknecht M, Vijayanarasimhan S, et al. Beyond short snippets: Deep networks for video classification[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 4694-4702.
[30]
Gabrielsson S. A moral endeavour in a demoralizing context: Psychiatric inpatient care from the perspective of professional caregivers[D]. Luleå tekniska universitet, 2015.
[31]
Nazir D, Hashmi KA, Pagani A, et al. Title. Preprints 2021, 1, 0[C]//Conference on Document Analysis and Recognition (ICDAR). IEEE. s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affil-iations. 1 Department of Computer Science, Technical University of Kaiserslautern, 67663 Kaiserslautern, Germany; dn8034@gmail. com (DN); khurram_azeem. hashmi@ dfki. de (KAH); muhammad_zeshan. afzal@ dfki. de (MZA); alain. pagani@ dfki. de (AP); didier. stricker@ dfki. de (DS); 2 Mindgarage, Technical University of Kaiserslautern, 67663 Kaiserslautern, Germany 3 German Research Institute for Artificial Intelligence (DFKI), 67663 Kaiserslautern, Germany, 4 Department of Computer Science, Luleå University of Technology, 971 87 Luleå, Sweden; marcus. liwicki@ ltu. se (ML);, 2017, 1: 1417-1422.
[32]
W. Chen, M. Fang, Y.H. Liu, et al., Monocular semantic SLAM in dynamic street scene based on multiple object tracking[C]//2017, IEEE, 2017, pp. 599–604.
[33]
Xu X., Dou P., Le H.A., et al. When 3D-Aided 2D Face Recognition Meets Deep Learning: An extended UR2D for Pose-Invariant Face Recognition[J]. arXiv preprint arXiv:1709.06532, 2017.
[34]
J. Wang, J. Ding, H. Guo, et al., Mask OBB: A semantic attention-based mask oriented bounding box representation for multi-category object detection in aerial images[J], Remote Sensing 11 (24) (2019) 2930.
[35]
Felsberg M., Five years after the Deep Learning revolution of computer vision: State of the art methods for online image and video analysis[J]. 2017.
[36]
Rezaei M, Azarmi M. Deepsocial: Social distancing monitoring and infection risk assessment in covid-19 pandemic[J]. Applied Sciences, 2020, 10(21): 7514.
[37]
S. Paul, L. Singh, A review on advances in deep learning[C]//2015 IEEE Workshop on Computational Intelligence: Theories, Applications and Future Directions (WCI), IEEE (2015) 1–6.
[38]
H Ge, Y Dai, Z Zhu, et al., A Robust Face Recognition Algorithm Based on an Improved Generative Confrontation Network[J], Applied Sciences 11 (24) (2021) 11588.

Cited By

View all
  • (2024)Facial Expression Recognition Using a Semantic-Based Bottleneck Attention ModuleInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.35241820:1(1-25)Online publication date: 16-May-2024
  • (2024)Bodily Sensation Map vs. Bodily Motion Map: Visualizing and Analyzing Emotional Body MotionsIEEE Transactions on Affective Computing10.1109/TAFFC.2024.336589515:3(1649-1658)Online publication date: 1-Jul-2024
  • (2024)Attention based hybrid deep learning model for wearable based stress recognitionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107391127:PBOnline publication date: 1-Jan-2024
  • Show More Cited By

Index Terms

  1. Facial expression recognition based on deep learning
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Please enable JavaScript to view thecomments powered by Disqus.

          Information & Contributors

          Information

          Published In

          cover image Computer Methods and Programs in Biomedicine
          Computer Methods and Programs in Biomedicine  Volume 215, Issue C
          Mar 2022
          522 pages

          Publisher

          Elsevier North-Holland, Inc.

          United States

          Publication History

          Published: 01 March 2022

          Author Tags

          1. 3D face depth information
          2. Deep learning
          3. Facial expression recognition
          4. Target detection
          5. Convolutional neural network

          Qualifiers

          • Research-article

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)0
          • Downloads (Last 6 weeks)0
          Reflects downloads up to 22 Sep 2024

          Other Metrics

          Citations

          Cited By

          View all
          • (2024)Facial Expression Recognition Using a Semantic-Based Bottleneck Attention ModuleInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.35241820:1(1-25)Online publication date: 16-May-2024
          • (2024)Bodily Sensation Map vs. Bodily Motion Map: Visualizing and Analyzing Emotional Body MotionsIEEE Transactions on Affective Computing10.1109/TAFFC.2024.336589515:3(1649-1658)Online publication date: 1-Jul-2024
          • (2024)Attention based hybrid deep learning model for wearable based stress recognitionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107391127:PBOnline publication date: 1-Jan-2024
          • (2024)Exploring the potential of deep learning and machine learning techniques for randomness analysis to enhance security on IoTInternational Journal of Information Security10.1007/s10207-023-00783-y23:2(1117-1130)Online publication date: 1-Apr-2024
          • (2023)Overcoming Occlusion for Robust Facial Expression Recognition using Adaptive Dual-Attention NetProceedings of the 6th International Conference on Information Technologies and Electrical Engineering10.1145/3640115.3640175(368-373)Online publication date: 3-Nov-2023
          • (2023)Deep Learning-based Facial Expression Recognition for Fatigue DrivingProceedings of the 2023 International Conference on Computer, Vision and Intelligent Technology10.1145/3627341.3627344(1-8)Online publication date: 25-Aug-2023
          • (2023)Deep neural network technique for automated detection of ADHD and CD using ECG signalComputer Methods and Programs in Biomedicine10.1016/j.cmpb.2023.107775241:COnline publication date: 1-Nov-2023
          • (2023)An efficient deep learning framework for occlusion face prediction systemKnowledge and Information Systems10.1007/s10115-023-01896-565:11(5043-5063)Online publication date: 1-Nov-2023
          • (2023)CSLSEP: an ensemble pruning algorithm based on clustering soft label and sorting for facial expression recognitionMultimedia Systems10.1007/s00530-023-01062-529:3(1463-1479)Online publication date: 3-Mar-2023
          • (2022)Facial expression recognition based on improved depthwise separable convolutional networkMultimedia Tools and Applications10.1007/s11042-022-14066-682:12(18635-18652)Online publication date: 23-Nov-2022

          View Options

          View options

          Get Access

          Login options

          Media

          Figures

          Other

          Tables

          Share

          Share

          Share this Publication link

          Share on social media