Explainable Facial Expression Recognition for People with Intellectual Disabilities
Authors: 
Silvia Ramis Guarinos, Cristina Manresa Yee, Jose Maria Buades Rubio, 
Francesc Xavier Gaya-MoreyAuthors Info & Claims
Article No.: 5, Pages 1 - 7
Abstract
Facial expression recognition plays an important role in human behaviour, communication, and interaction. Recent neural networks have demonstrated to perform well at its automatic recognition, with different explainability techniques available to make them more transparent. In this work, we propose a facial expression recognition study for people with intellectual disabilities that would be integrated into a social robot. We train two well-known neural networks with five databases of facial expressions and test them with two databases containing people with and without intellectual disabilities. Finally, we study in which regions the models focus to perceive a particular expression using two different explainability techniques: LIME and RISE, assessing the differences when used on images containing disabled and non-disabled people.
References
[1]
Ko, B. (2018). A brief review of facial emotion recognition based on visual information. sensors, 18(2), 401.
[2]
Carroll, J. M., & Kjeldskov, J. (2013). The encyclopedia of human-computer interaction. 2nd. Ed. Interaction Design Foundation.
[3]
Huang, W. (2015). When HCI Meets HRI: the intersection and distinction.
[4]
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003). A survey of socially interactive robots. Robotics and autonomous systems, 42(3-4), 143-166.
[5]
Mitchell, A., Sitbon, L., Balasuriya, S.S., Koplick, S., Beaumont, C. (2021). Social Robots in Learning Experiences of Adults with Intellectual Disability: An Exploratory Study. Human-Computer Interaction – INTERACT 2021. INTERACT 2021. Lecture Notes in Computer Science, vol 12932. Springer, Cham. https://doi.org/10.1007/978-3-030-85623-6_17]
[6]
D. Silvera-Tawil and C. R. Yates, "Socially-Assistive Robots to Enhance Learning for Secondary Students with Intellectual Disabilities and Autism," 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2018, pp. 838-843.
[7]
Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021). Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Computing and Applications, 1-18.
[8]
Generosi, A., Ceccacci, S., Faggiano, S., Giraldi, L., & Mengoni, M. (2020). A toolkit for the automatic analysis of human behavior in HCI applications in the wild. Adv. Sci. Technol. Eng. Syst. J, 5(6), 185-192.
[9]
Martínez A, Belmonte LM, García AS, Fernández-Caballero A, Morales R. Facial Emotion Recognition from an Unmanned Flying Social Robot for Home Care of Dependent People. Electronics. 2021; 10(7):868. https://doi.org/10.3390/electronics10070868
[10]
Ramis S, Buades JM, Perales FJ. Using a Social Robot to Evaluate Facial Expressions in the Wild. Sensors. 2020; 20(23):6716. https://doi.org/10.3390/s20236716
[11]
K. Weitz, T. Hassan, U. Schmid, and J.-U. Garbas, “Deep-learned faces of pain and emotions: Elucidating the differences of facial expressions with the help of explainable AI methods,” Technisches Messen, vol. 86, no. 7–8, pp. 404–412, 2019.
[12]
Ramis, S., Buades, J.M., Perales, F.J. et al. A Novel Approach to Cross dataset studies in Facial Expression Recognition. Multimed Tools Appl 81, 39507–39544 (2022). https://doi.org/10.1007/s11042-022-13117-2
[13]
Ribeiro MT, Singh S, Guestrin C. “Why should I trust you?”: Explaining the predictions of any classifier," CoRR, vol. abs/1602.04938, 2016. http://arxiv.org/abs/1602.04938
[14]
A. Heimerl, K. Weitz, T. Baur, and E. Andre, “Unraveling ML Models of Emotion with NOVA: Multi-Level Explainable AI for Non-Experts,” IEEE Transactions on Affective Computing, vol. 1, no. 1, pp. 1–13, 2020.
[15]
M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135–1144.
[16]
M. Alber, “iNNvestigate Neural Networks!,” Journal of Machine Learning Research, vol. 20, no. 93, pp. 1–8, 2019.
[17]
Lucey P, Cohn JF, Kanade T (2010) The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition -workshops, CVPRW 2010.
[18]
Yin L, Wei X, Sun Y (2006) A 3D facial expression database for facial behavior research. In: FGR2006: proceedings of the 7th international conference on automatic face and gesture recognition.
[19]
Lyons M, Kamachi M, Gyoba J (2017) Japanese female facial expression (JAFFE) database. Available:http://www.kasrl.org/jaffe.htm
[20]
Olszanowski M, Pochwatko G, Kuklinski K (2014) Warsaw set of emotional facial expression pictures:a validation study of facial display photographs. Front Psychol 5. https://doi.org/10.3389/fpsyg.2014.01516
[21]
Shukla, J., Barreda-Ángeles, M., Oliver, J., & Puig, D. (2016). MuDERI: Multimodal database for emotion recognition among intellectually disabled individuals. In Social Robotics: 8th International Conference, ICSR 2016, Kansas City, MO, USA, November 1-3, 2016 Proceedings 8 (pp. 264-273). Springer International Publishing.
[22]
Lisani JL, Ramis S, Perales FJ (2017) A contrario detection of faces: a case example. SIAM J Imaging Sci10:2091–2118. https://doi.org/10.1137/17M1118774
[23]
Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., & Pantic, M. (2013). 300 Faces in-the-wild challenge: The first facial landmark localization challenge. In Proceedings of the IEEE International Conference on Computer Vision Workshops (pp. 397-403).
[24]
Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neuralnetworks. In: Advances in Neural Information Processing Systems
[25]
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
[26]
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., ... & Herrera, F. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion, 58, 82-115.
[27]
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).
[28]
Petsiuk, V., Das, A., & Saenko, K. (2018). Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421.
[29]
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., & Süsstrunk, S. (2010). Slic superpixels (No. REP_WORK).
Recommendations
Expression-invariant face recognition by facial expression transformations
In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Facial expression recognition with Convolutional Neural Networks
Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...
Pose-Robust Facial Expression Recognition Using View-Based 2D + 3D AAM
This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2023
203 pages
ISBN:9798400707902
DOI:10.1145/3612783
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 January 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Ministerio de Ciencia e Innovación
Conference
Interaccion 2023
Interaccion 2023: XXIII International Conference on Human-Computer Interaction
September 4 - 6, 2023
Lleida, Spain
Acceptance Rates
Overall Acceptance Rate 109 of 163 submissions, 67%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 163Total Downloads
- Downloads (Last 12 months)163
- Downloads (Last 6 weeks)25
Reflects downloads up to 29 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in