Article

Facial Emotions Classification Supported in an Ensemble Strategy

Authors:

Pedro J. S. Cardoso,

João M. F. RodriguesAuthors Info & Claims

Universal Access in Human-Computer Interaction. Novel Design Approaches and Technologies: 16th International Conference, UAHCI 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part I

Pages 477 - 488

https://doi.org/10.1007/978-3-031-05028-2_32

Published: 26 June 2022 Publication History

Abstract

Humans are prepared to comprehend each other’s emotions from subtle body movements or facial expressions, and from those, they change the way they deliver messages when communicating between them. Machines, user interfaces, or robots need to empower this ability, in a way to change the interaction from the traditional “human-computer interaction” to a “human-machine cooperation”, where the machine provides the “right” information and functionality, at the “right” time, and in the “right” way. This paper presents a framework for facial expression prediction supported in an ensemble of facial expression methods, being the main contribution the integration of outputs from different methods in a single prediction consistent with the expression presented by the system’s user. Results show a classification accuracy above 73% in both FER2013 and RAF-DB datasets.

References

[1]

Deguchi, A., et al.: What is Society 5.0. Chapter 1 in Society 5.0 – A People-centric Super-smart Society. Hitachi-UTokyo Laboratory (eds.), pp. 1–23. Springer (2020).

[2]

Rothfuß, S., Wörner, M., Inga, J., Hohmann, S.: A study on human-machine cooperation on decision level. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 2291–2298. IEEE (2020)

[3]

Kumar P and Gupta A Active learning query strategies for classification, regression, and clustering: a survey J. Comput. Sci. Technol. 2020 35 4 913-945

[4]

Ardabili S, Mosavi A, and Várkonyi-Kóczy AR Várkonyi-Kóczy AR Advances in machine learning modeling reviewing hybrid and ensemble methods Engineering for Sustainable Future 2020 Cham Springer 215-227

[5]

Zhang, F., Zhang, T., Mao, Q., Xu, C.: Joint pose and expression modeling for facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3359–3368 (2018)

[6]

Noroozi F, Kaminska D, Corneanu C, Sapinski T, Escalera S, and Anbarjafari G Survey on emotional body gesture recognition IEEE Trans. Affect. Comput. 2018 12 2 505-523

[7]

Filntisis PP, Efthymiou N, Koutras P, Potamianos G, and Maragos P Fusing body posture with facial expressions for joint recognition of affect in child–robot interaction IEEE Robot. Autom. Lett. 2019 4 4 4011-4018

[8]

Leiva S, Margulis L, Micciulli A, and Ferreres A Dissociation between facial and bodily expressions in emotion recognition: a case study Clin. Neuropsychol. 2019 33 1 166-182

[9]

Canedo, D., Neves, A.J.: Mood estimation based on facial expressions and postures. In: Proceedings of the RECPAD 2020, pp. 49–50 (2020)

[10]

Bänziger T, Mortillaro M, and Scherer KR Introducing the geneva multimodal expression corpus for experimental research on emotion perception Emotion 2012 12 5 1161

[11]

Kleinsmith A and Bianchi-Berthouze N Affective body expression perception and recognition: a survey IEEE Trans. Affect. Comput. 2012 4 1 15-33

[12]

Senecal S, Cuel L, Aristidou A, and Magnenat-Thalmann N Continuous body emotion recognition system during theater performances Comput. Animat. Virtual Worlds 2016 27 3–4 311-320

[13]

Ahmed F, Bari AH, and Gavrilova ML Emotion recognition from body movement IEEE Access 2019 8 11761-11781

[14]

Liang, G., Wang, S., Wang, C.: Pose-aware adversarial domain adaptation for personalized facial expression recognition. arXiv preprint arXiv:2007.05932 (2020)

[15]

Ekman P and Friesen WV Constants across cultures in the face and emotion J. Pers. Soc. Psychol. 1971 17 2 124

[16]

Zavaschi, T.H., Koerich, A.L., Oliveira, L.E.S.: Facial expression recognition using ensemble of classifiers. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1489–1492 (2011).

[17]

Renda A, Barsacchi M, Bechini A, and Marcelloni F Comparing ensemble strategies for deep learning: an application to facial expression recognition Expert Syst. Appl. 2019 136 1-11

[18]

Ali G et al. Artificial neural network based ensemble approach for multicultural facial expressions analysis IEEE Access 2020 8 134950-134963

[19]

Wang, Z., Zeng, F., Liu, S., Zeng, B.: OAENet: oriented attention ensemble for accurate facial expression recognition. Pattern Recognit. 112, 107694 (2021)

[20]

Benamara, N.K., et al.: Real-time facial expression recognition using smoothed deep neural network ensemble. Integr. Comput.-Aid. Eng. (Preprint) 28, 1–15 (2021)

[21]

Pecoraro, R., Basile, V., Bono, V., Gallo, S.: Local multi-head channel self-attention for facial expression recognition. arXiv preprint arXiv:2111.07224 (2021)

[22]

Goodfellow I et al. Lee M, Hirose A, Hou Z-G, Kil RM, et al. Challenges in representation learning: a report on three machine learning contests Neural Information Processing 2013 Heidelberg Springer 117-124

[23]

Cheong, J.H., Xie, T., Byrne, S., Chang, L.J.: Py-Feat: Python facial expression analysis toolbox. arXiv preprint arXiv:2104.03509 (2021)

[24]

Banerjee, R., De, S., Dey, S.: A survey on various Deep Learning algorithms for an efficient facial expression recognition system. Int. J. Image Graph., 2240005 (2021)

[25]

Revina IM and Emmanuel WS A survey on human face expression recognition techniques J. King Saud Univ.-Comput. Inf. Sci. 2021 33 6 619-628

[26]

LHC-NET: Local multi-head channel self-attention (code) (2021). https://github.com/bodhis4ttva/lhc_net. Accessed 28 Dec 2021

[27]

Py-FEAT: Python facial expression analysis toolbox (code) (2021). https://pythonrepo.com/repo/cosanlab-py-feat-python-deep-learning. Accessed 28 Dec 2021

[28]

Shenk, J.: Facial expression recognition (code) (2021). https://github.com/justinshenk/fer. Accessed 28 Dec 2021

[29]

Breiman L Random forests Mach. Learn. 2001 45 1 5-32

[30]

Hastie T, Rosset S, Zhu J, and Zou H Multi-class Adaboost. Statistics and its Interface 2009 2 3 349-360

[31]

Ayyadevara VK Pro Machine Learning Algorithms 2018 Berkeley Apress

[32]

FER2013: Learn facial expressions from an image (2021). https://www.kaggle.com/msambare/fer2013. Accessed 28 Dec 2021

[33]

RAF-DB: Real-world affective faces database (2021). http://www.whdeng.cn/raf/model1.html. Accessed 28 Dec 2021

[34]

OpenCV: OpenCV: Cascade classifier – face detection (2021). https://docs.opencv.org/4.5.5/db/d28/tutorial_cascade_classifier.html. Accessed 28 Dec 2021

[35]

Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

[36]

Li S and Deng W Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition IEEE Trans. Image Process. 2019 28 1 356-370

[37]

Cheong, J.H., Xie, T., Byrne, S., Chang, L.J.: Py-Feat: Python facial expression analysis toolbox. arXiv preprint arXiv:2104.03509 (2021)

Cited By

Cardoso PRodrigues JNovais R(2023)Multimodal Emotion Classification Supported in the Aggregation of Pre-trained Classification ModelsComputational Science – ICCS 202310.1007/978-3-031-36030-5_35(433-447)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-36030-5_35
Novais RCardoso PRodrigues J(2022)Emotion Classification from Speech by an Ensemble StrategyProceedings of the 10th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion10.1145/3563137.3563170(85-90)Online publication date: 31-Aug-2022
https://dl.acm.org/doi/10.1145/3563137.3563170

Index Terms

Facial Emotions Classification Supported in an Ensemble Strategy
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Classification and regression trees

Index terms have been assigned to the content through auto-classification.

Recommendations

Facial expression recognition with Convolutional Neural Networks

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...
Ensemble Approaches to Facial Action Unit Classification
CIARP '08: Proceedings of the 13th Iberoamerican congress on Pattern Recognition: Progress in Pattern Recognition, Image Analysis and Applications

Facial action unit ( au ) classification is an approach to face expression recognition that decouples the recognition of expression from individual actions. In this paper, upper face aus are classified using an ensemble of MLP (Multi-layer perceptron)...
Generating Facial Emotions for Diagnosis and Training
CBMS '15: Proceedings of the 2015 IEEE 28th International Symposium on Computer-Based Medical Systems

The ability to process and identify facial emotions is an essential factor for an individuals social interaction. There are certain psychiatric disorders that can limit an individuals ability to recognize emotions in facial expressions. This problem ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Universal Access in Human-Computer Interaction. Novel Design Approaches and Technologies: 16th International Conference, UAHCI 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part I

Jun 2022

576 pages

ISBN:978-3-031-05027-5

DOI:10.1007/978-3-031-05028-2

Editors:
Margherita Antona
Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece
,
Constantine Stephanidis
University of Crete and Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 26 June 2022

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cardoso PRodrigues JNovais R(2023)Multimodal Emotion Classification Supported in the Aggregation of Pre-trained Classification ModelsComputational Science – ICCS 202310.1007/978-3-031-36030-5_35(433-447)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-36030-5_35
Novais RCardoso PRodrigues J(2022)Emotion Classification from Speech by an Ensemble StrategyProceedings of the 10th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion10.1145/3563137.3563170(85-90)Online publication date: 31-Aug-2022
https://dl.acm.org/doi/10.1145/3563137.3563170

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents