research-article

A Systematic Literature Review of Recognition of Compound Facial Expression of Emotions

Authors:

Wenhong TianAuthors Info & Claims

ICVIP '20: Proceedings of the 2020 4th International Conference on Video and Image Processing

Pages 116 - 121

https://doi.org/10.1145/3447450.3447469

Published: 09 April 2021 Publication History

Abstract

Recently, emotion recognition has gained increasing attention in various fields. Human facial expression plays key role in daily interaction among people. The understanding of different categories of facial expression is crucial in perceptual interfaces, computational models, and human cognition and even in every field of life. The basic facial expression of emotions is categorized into seven groups which are including neutral, disgust, fear, anger, surprise, sadness and happiness. The compound facial expression produced from the combination of some of the basic emotions. However, there has been no survey article on Compound Facial Expression of Emotions. In this article, we present survey on recognition, research methodologies and applications of Compound Facial Expression of Emotions. Diverse types of datasets/databases for the recognition of compound facial expressions of emotion with their pros and cons have been discussed on detail. Furthermore, different research methods along with advantages, disadvantages and possible future improvements are also deliberated. In the last, opportunities and challenges are enlisted.

References

[1]

Li, S., & Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing.

[2]

Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of personality and social psychology, 17(2), 124.

[3]

Pelachaud, C. (2009). Modelling multimodal expression of emotion in a virtual agent. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3539-3548.

[4]

Du, S., & Martinez, A. M. (2015). Compound facial expressions of emotion: from basic research to clinical applications. Dialogues in clinical neuroscience, 17(4), 443.

[5]

Swaminathan, A., Vadivel, A., & Arock, M. (2020). FERCE: Facial Expression Recognition for Combined Emotions Using FERCE Algorithm. IETE Journal of Research, 1-16.

[6]

Kaltwang, S., Rudovic, O., & Pantic, M. (2012, July). Continuous pain intensity estimation from facial expressions. In International Symposium on Visual Computing (pp. 368-377). Springer, Berlin, Heidelberg.

[7]

Egede, J., Valstar, M., & Martinez, B. (2017, May). Fusing deep learned and hand-crafted features of appearance, shape, and dynamics for automatic pain estimation. In 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017) (pp. 689-696). IEEE.

Digital Library

[8]

Yang, M. T., Cheng, Y. J., & Shih, Y. C. (2011, July). Facial expression recognition for learning status analysis. In International Conference on Human-Computer Interaction (pp. 131-138). Springer, Berlin, Heidelberg.

[9]

Slimani, K., Messoussi, R., Bourekkadi, S., & Khoulji, S. (2017, April). An intelligent system solution for improving the distance collaborative work. In 2017 Intelligent Systems and Computer Vision (ISCV) (pp. 1-4). IEEE.

[10]

Mourão, A., & Magalhães, J. (2013, October). Competitive affective gaming: winning with a smile. In Proceedings of the 21st ACM international conference on Multimedia (pp. 83-92).

Digital Library

[11]

Riek, L. D., & Robinson, P. (2011, June). Using robots to help people habituate to visible disabilities. In 2011 IEEE International Conference on Rehabilitation Robotics (pp. 1-8). IEEE.

[12]

Bourekkadi, S., Khoulji, S., Slimani, K., Messoussi, R., & Kerkeb, M. L. (2016). THE DESIGN OF A PSYCHOTHERAPY REMOTE INTELLIGENT SYSTEM. Journal of Theoretical & Applied Information Technology, 93(1).

[13]

Bartlett, M. S., Littlewort, G., Fasel, I., & Movellan, J. R. (2003, June). Real time face detection and facial expression recognition: development and applications to human computer interaction. In 2003 Conference on computer vision and pattern recognition workshop (Vol. 5, pp. 53-53). IEEE.

[14]

Zhao, X., Shi, X., & Zhang, S. (2015). Facial expression recognition via deep learning. IETE technical review, 32(5), 347-355.

[15]

Shan, K., Guo, J., You, W., Lu, D., & Bie, R. (2017, June). Automatic facial expression recognition based on a deep convolutional-neural-network structure. In 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA) (pp. 123-128). IEEE.

[16]

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).

Digital Library

[17]

LeCun, Y., Huang, F. J., & Bottou, L. (2004, June). Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. (Vol. 2, pp. II-104). IEEE.

[18]

LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010, May). Convolutional networks and applications in vision. In Proceedings of 2010 IEEE international symposium on circuits and systems (pp. 253-256). IEEE.

[19]

Ekman, P. (1992). An argument for basic emotions. Cognition & emotion, 6(3-4), 169-200.

[20]

Berrios, R., Totterdell, P., & Kellett, S. (2015). Eliciting mixed emotions: a meta-analysis comparing models, types, and measures. Frontiers in psychology, 6, 428.

[21]

Izard, C. E. (1992). Basic emotions, relations among emotions, and emotion-cognition relations.

[22]

S. Du, Y. Tao, and A. M. Martinez, “Compound facial expressions of emotion,” Proc. Natl. Acad. Sci., vol. 111, no. 15, pp. E1454–E1462, Apr. 2014.

[23]

Kastemaa, J. (2017). Recognizing compound facial expressions of virtual characters in augmented reality.

[24]

Du, S. (2014). A Computational Model of the Production and Perception of Facial Expressions of Basic and Compound Emotions (Doctoral dissertation, The Ohio State University).

[25]

Du, S., & Martinez, A. M. (2015). Compound facial expressions of emotion: from basic research to clinical applications. Dialogues in clinical neuroscience, 17(4), 443.

[26]

Pallett, P., & Martinez, A. (2014). Beyond the basics: Facial expressions of compound emotions. Journal of Vision, 14(10), 1401-1401.

[27]

Keltner, D., Sauter, D., Tracy, J., & Cowen, A. (2019). Emotional expression: Advances in basic emotion theory. Journal of nonverbal behavior, 1-28.

[28]

Wagner, W. (2019). Recognizing Compound Expressions and Emotion Intensity (Doctoral dissertation, The University of Alabama at Birmingham).

[29]

Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering.

[30]

Benda, M. S., & Scherf, K. S. (2020). The Complex Emotion Expression Database: A validated stimulus set of trained actors. PloS one, 15(2), e0228248. https://doi.org/10.1371/journal.pone.0228248

[31]

Slimani, K., Lekdioui, K., Messoussi, R., & Touahni, R. (2019, March). Compound Facial Expression Recognition Based on Highway CNN. In Proceedings of the New Challenges in Data Sciences: Acts of the Second Conference of the Moroccan Classification Society (pp. 1-7).

Digital Library

[32]

Li, S., Deng, W., & Du, J. (2017). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2852-2861).

[33]

Li, S., & Deng, W. (2018). Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Transactions on Image Processing, 28(1), 356-370.

Digital Library

[34]

Liu, Z., Li, S., & Deng, W. (2016, October). Recognizing compound emotional expression in real-world using metric learning method. In Chinese Conference on Biometric Recognition (pp. 528-536). Springer, Cham.

[35]

Zhao, Y., & Xu, J. (2019). A Convolutional Neural Network for Compound Micro-Expression Recognition. Sensors, 19(24), 5553.

[36]

Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010, June). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 ieee computer society conference on computer vision and pattern recognition-workshops (pp. 94-101). IEEE.

[37]

Li, Y., Lu, Y., Li, J., & Lu, G. (2019, October). Separate Loss for Basic and Compound Facial Expression Recognition in the Wild. In Asian Conference on Machine Learning (pp. 897-911).

[38]

C. F. Benitez-Quiroz, R. Srinivasan, Q. Feng, Y. Wang, and A. M. Martinez. (2017). ‘‘EmotioNet challenge: Recognition of facial expressions of emotion in the wild.’’ [Online]. Available: https://arxiv.org/abs/1703.01210

[39]

Kotsia, I., & Pitas, I. (2006). Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE transactions on image processing, 16(1), 172-187.

[40]

Luo, Y., Wu, C. M., & Zhang, Y. (2013). Facial expression recognition based on fusion feature of PCA and LBP with SVM. Optik-International Journal for Light and Electron Optics, 124(17), 2767-2770.

[41]

Deng, H. B., Jin, L. W., Zhen, L. X., & Huang, J. C. (2005). A new facial expression recognition method based on local Gabor filter bank and PCA plus LDA. International Journal of Information Technology, 11(11), 86-96.

[42]

George, T., Potty, S. P., & Jose, S. (2014, July). Smile detection from still images using KNN algorithm. In 2014 International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT) (pp. 461-465). IEEE.

[43]

Guo, J., Lei, Z., Wan, J., Avots, E., Hajarolasvadi, N., Knyazev, B., ... & Escalera, S. (2018). Dominant and complementary emotion recognition from still images of faces. IEEE Access, 6, 26391-26403.

[44]

Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15), E1454-E1462.

[45]

Liliana, D. Y., Basaruddin, C., & Widyanto, M. R. (2017, August). Mix emotion recognition from facial expression using SVM-CRF sequence classifier. In Proceedings of the International Conference on Algorithms, Computing and Systems (pp. 27-31).

Digital Library

[46]

Hamsici, O. C., & Martinez, A. M. (2009, October). Active appearance models with rotation invariant kernels. In 2009 IEEE 12th International Conference on Computer Vision (pp. 1003-1009). IEEE.

[47]

Jarraya, S. K., Masmoudi, M., & Hammami, M. (2020). Compound Emotion Recognition of Autistic Children During Meltdown Crisis Based on Deep Spatio-Temporal Analysis of Facial Geometric Features. IEEE Access, 8, 69311-69326.

[48]

Fabian Benitez-Quiroz, C., Srinivasan, R., & Martinez, A. M. (2016). Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5562-5570).

[49]

Liliana, D. Y., Basaruddin, T., & Oriza, I. I. D. (2018, November). The Indonesian Mixed Emotion Dataset (IMED) A Facial Expression Dataset for Mixed Emotion Recognition. In Proceedings of the 2018 International Conference on Artificial Intelligence and Virtual Reality (pp. 56-60).

Digital Library

[50]

Thuseethan, S., Rajasegarar, S., & Yearwood, J. (2020). Complex Emotion Profiling: An Incremental Active Learning Based Approach With Sparse Annotations. IEEE Access, 8, 147711-147727.

[51]

Guo, J., Zhou, S., Wu, J., Wan, J., Zhu, X., Lei, Z., & Li, S. Z. (2017, May). Multi-modality network with visual and geometrical information for micro emotion recognition. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) (pp. 814-819). IEEE.

Digital Library

[52]

Li, H., & Li, Q. (2020). End-to-End Training for Compound Expression Recognition. Sensors, 20(17), 4727.

Cited By

Ullah SOu JXie YTian W(2024)Compound facial expressions recognition approach using DCGAN and CNNMultimedia Tools and Applications10.1007/s11042-024-20138-683:38(85703-85723)Online publication date: 28-Aug-2024
https://doi.org/10.1007/s11042-024-20138-6
Chaudhari ABhatt CKrishna ATravieso-González C(2023)Facial Emotion Recognition with Inter-Modality-Attention-Transformer-Based Self-Supervised LearningElectronics10.3390/electronics1202028812:2(288)Online publication date: 5-Jan-2023
https://doi.org/10.3390/electronics12020288
Jiddah SYurtkan K(2023)Dominant and complementary emotion recognition using hybrid recurrent neural networkSignal, Image and Video Processing10.1007/s11760-023-02563-617:7(3415-3423)Online publication date: 8-May-2023
https://doi.org/10.1007/s11760-023-02563-6
Show More Cited By

Recommendations

Human-Computer Interaction Using Emotion Recognition from Facial Expression
EMS '11: Proceedings of the 2011 UKSim 5th European Symposium on Computer Modeling and Simulation

This paper describes emotion recognition system based on facial expression. A fully automatic facial expression recognition system is based on three steps: face detection, facial characteristic extraction and facial expression classification. We have ...
Human centric facial expression recognition
HCI '18: Proceedings of the 32nd International BCS Human Computer Interaction Conference

Facial expression recognition (FER) is an area of active research, both in computer science and in behavioural science. Across these domains there is evidence to suggest that humans and machines find it easier to recognise certain emotions, for example ...
Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICVIP '20: Proceedings of the 2020 4th International Conference on Video and Image Processing

December 2020

255 pages

ISBN:9781450389075

DOI:10.1145/3447450

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Key Research and Development Plan

Conference

ICVIP 2020

ICVIP 2020: 2020 The 4th International Conference on Video and Image Processing

December 25 - 27, 2020

Xi'an, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
279
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)8

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ullah SOu JXie YTian W(2024)Compound facial expressions recognition approach using DCGAN and CNNMultimedia Tools and Applications10.1007/s11042-024-20138-683:38(85703-85723)Online publication date: 28-Aug-2024
https://doi.org/10.1007/s11042-024-20138-6
Chaudhari ABhatt CKrishna ATravieso-González C(2023)Facial Emotion Recognition with Inter-Modality-Attention-Transformer-Based Self-Supervised LearningElectronics10.3390/electronics1202028812:2(288)Online publication date: 5-Jan-2023
https://doi.org/10.3390/electronics12020288
Jiddah SYurtkan K(2023)Dominant and complementary emotion recognition using hybrid recurrent neural networkSignal, Image and Video Processing10.1007/s11760-023-02563-617:7(3415-3423)Online publication date: 8-May-2023
https://doi.org/10.1007/s11760-023-02563-6
Pendhari HNagdeote SRathod SKhan LVishwakarma S(2022)Compound Emotions: A Mixed emotion detectionSSRN Electronic Journal10.2139/ssrn.4120265Online publication date: 2022
https://doi.org/10.2139/ssrn.4120265
Canal FMüller TMatias JScotton Gde Sa Junior APozzebon ESobieranski A(2022)A survey on facial emotion recognition techniquesInformation Sciences: an International Journal10.1016/j.ins.2021.10.005582:C(593-617)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1016/j.ins.2021.10.005

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents