research-article

Facial Expression Recognition: Impact of Gender on Fairness and Expressions∗

Authors:

Cristina Manresa-Yee,

Silvia Ramis Guarinos,

Jose Maria Buades RubioAuthors Info & Claims

Interacción '22: Proceedings of the XXII International Conference on Human Computer Interaction

Article No.: 13, Pages 1 - 8

https://doi.org/10.1145/3549865.3549904

Published: 09 September 2022 Publication History

Abstract

Multiple and varied domains can benefit from automated Facial Expression Recognition (FER) like human computer interfaces or health applications. New approaches using Machine learning (ML) are achieving successful results, but its use raises concerns related with biases, fairness or explainability, which can undermine the trust of the users. This work aims to study how gender biased training datasets alter fairness in FER. The main outcomes show which facial expressions recognition are more impacted by gender bias.

References

[1]

W. Mellouk and W. Handouzi, “Facial emotion recognition using deep learning: review and insights,” Procedia Comput. Sci., vol. 175, pp. 689–694, 2020.

[2]

C. Darwin, The origin of species by means of natural selection: or, the preservation of favoured races in the struggle for life and the descent of man and selection in relation to sex, 1st Editio. London: Modern Library., 1859.

[3]

P. Ekman, “Universals and cultural differences in facial expressions of emotion,” Nebraska Symp. Motiv., vol. 19, pp. 207–283, 1971.

[4]

K. Grabowski, “Emotional expression in psychiatric conditions: New technology for clinicians,” Psychiatry Clin. Neurosci., vol. 73, no. 2, pp. 50–62, 2019.

[5]

A. M. Barreto, “Application of facial expression studies on the field of marketing,” Emot. Expr. brain face, vol. 9, no. June, pp. 163–189, 2017.

[6]

S. Medjden, N. Ahmed, and M. Lataifeh, “Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor,” PLoS One, vol. 15, no. 7, p. e0235908, 2020.

[7]

B. Lepri, N. Oliver, E. Letouzé, A. Pentland, and P. Vinck, “Fair, Transparent, and Accountable Algorithmic Decision-making Processes,” Philos. Technol., vol. 31, no. 4, pp. 611–627, 2018.

[8]

D. Danks and A. J. London, “Algorithmic Bias in Autonomous Systems A Taxonomy of Algorithmic Bias,” 26th Int. Jt. Conf. Artif. Intell., no. Ijcai, pp. 1–7, 2017.

[9]

[9] X. Ferrer, T. van Nuenen, J. M. Such, M. Coté, and N. Criado, “Bias and discrimination in AI: A cross-disciplinary perspective,” IEEE Technol. Soc. Mag., vol. 40, no. 2, pp. 72–80, 2021.

[10]

S. Verma and J. Rubin, “Fairness Definitions Explained,” in Proceedings of the International Workshop on Software Fairness, 2018, pp. 1–7.

Digital Library

[11]

N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A Survey on Bias and Fairness in Machine Learning,” ACM Comput. Surv., vol. 54, no. 6, 2021.

Digital Library

[12]

A. Narayanan, “Translation tutorial: 21 fairness definitions and their politics.,” 2018.

[13]

A. Castelnovo, R. Crupi, G. Greco, D. Regoli, I. G. Penco, and A. C. Cosentini, “A clarification of the nuances in the fairness metrics landscape,” Sci. Rep., vol. 12, no. 1, p. 4209, 2022.

[14]

C. Szegedy, “Going deeper with convolutions,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9.

[15]

A. Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild,” IEEE Trans. Affect. Comput., vol. 10, no. 01, pp. 18–31, 2019.

Digital Library

[16]

M. Georgopoulos, Y. Panagakis, and M. Pantic, “Investigating bias in deep face analysis: The KANFace dataset and empirical study,” Image Vis. Comput., vol. 102, p. 103954, 2020.

[17]

J. Buolamwini and T. Gebru, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification,” 2018.

[18]

S. Gong, X. Liu, and A. K. Jain, “Mitigating Face Recognition Bias via Group Adaptive Classifier,” 2021.

[19]

A. Morales, J. Fierrez, R. Vera-Rodriguez, and R. Tolosana, “SensitiveNets: Learning Agnostic Representations with Application to Face Images,” IEEE Trans. Pattern Anal. Mach. Intell., vol. PP, Aug. 2020.

[20]

J. Robinson, G. Livitz, Y. Henon, C. Qin, Y. Fu, and S. Timoner, “Face Recognition: Too Bias, or Not Too Bias?,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun. 2020, pp. 1–10.

[21]

M. Shin, J.-H. Seo, and D.-S. Kwon, “Face image-based age and gender estimation with consideration of ethnic difference,” 2017 26th IEEE Int. Symp. Robot Hum. Interact. Commun., pp. 567–572, 2017.

Digital Library

[22]

A. Clapés, G. Anbarjafari, O. Bilici, D. Temirova, E. Avots, and S. Escalera, “From Apparent to Real Age: Gender, Age, Ethnic, Makeup, and Expression Bias Analysis in Real Age Estimation,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018, pp. 2436–243609.

[23]

A. Das, A. Dantcheva, and F. Brémond, “Mitigating Bias in Gender, Age and Ethnicity Classification: A Multi-task Convolution Neural Network Approach,” 2018.

[24]

W. Wu, P. Protopapas, Z. Yang, and P. Michalatos, “Gender Classification and Bias Mitigation in Facial Images,” in 12th ACM Conference on Web Science, 2020, pp. 106–114.

Digital Library

[25]

I. Serna, A. Pena, A. Morales, and J. Fierrez, “InsideBias: Measuring Bias in Deep Networks and Application to Face Gender Biometrics,” 2020 25th Int. Conf. Pattern Recognit., pp. 3720–3727, 2021.

[26]

Z. Wang, “Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation,” 2020.

[27]

T. Xu, J. White, S. Kalkan, and H. Gunes, “Investigating Bias and Fairness in Facial Expression Recognition,” 2020.

Digital Library

[28]

A. Domnich and G. Anbarjafari, “Responsible AI: Gender bias assessment in emotion recognition,” arXiv, pp. 1–19, 2021, [Online]. Available: http://arxiv.org/abs/2103.11436.

[29]

M. Lyons, M. Kamachi, and J. Gyoba, “The Japanese Female Facial Expression (JAFFE) Dataset.” Zenodo, 1998.

[30]

Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep Learning Face Attributes in the Wild,” in 2015 IEEE International Conference on Computer Vision (ICCV), Nov. 2015, pp. 3730–3738.

Digital Library

[31]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.

[32]

O. Russakovsky, “ImageNet Large Scale Visual Recognition Challenge,” Int. J. Comput. Vis., vol. 115, no. 3, pp. 211–252, 2015.

Digital Library

[33]

S. Li, W. Deng, and J. Du, “Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2584–2593.

[34]

P. Ekman, “An argument for basic emotions,” Cogn. Emot., vol. 6, no. 3–4, pp. 169–200, 1992.

[35]

J.-L. Lisani, S. Ramis, and F. Perales, “A Contrario Detection of Faces: A Case Example,” SIAM J. Imaging Sci., vol. 10, pp. 2091–2118, Jan. 2017.

[36]

C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge,” in 2013 IEEE International Conference on Computer Vision Workshops, 2013, pp. 397–403.

Digital Library

[37]

S. Ramis, J. . Buades, F. J. Perales, and C. Manresa-Yee, “A Novel Approach to Cross dataset studies in Facial Expression Recognition,” Multimed. Tools Appl.

[38]

M. J. Kusner, C. Russell, J. Loftus, and R. Silva, “Counterfactual Fairness,” Jan. 2017.

[39]

M. Hardt, E. Price, and N. Srebro, “Equality of Opportunity in Supervised Learning,” in Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016, pp. 3323–3331.

[40]

K. Gurumoorthy, A. Dhurandhar, G. Cecchi, and C. Aggarwal, “Efficient Data Representation by Selecting Prototypes with Importance Weights,” 2019, pp. 260–269.

[41]

K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 6896–6905, 2020.

[42]

Q. T. Ngo and S. Yoon, “Facial expression recognition based on weighted-cluster loss and deep transfer learning using a highly imbalanced dataset,” Sensors (Switzerland), vol. 20, no. 9, 2020.

[43]

G. E. Schwartz, S. ‐L Brown, and G. L. Ahern, “Facial Muscle Patterning and Subjective Experience During Affective Imagery: Sex Differences.,” Psychophysiology, vol. 17, pp. 75–82, 1980.

[44]

D. Schiller, T. Huber, M. Dietz, and E. André, “Relevance-Based Data Masking: A Model-Agnostic Transfer Learning Approach for Facial Expression Recognition,” Front. Comput. Sci., vol. 2, p. 6, 2020.

[45]

H. G. Wallbott, “Big girls don't frown, big boys don't cry—Gender differences of professional actors in communicating emotion via facial expression.,” J. Nonverbal Behav., vol. 12, no. 2, pp. 98–106, 1988.

[46]

C. Evers, A. H. Fischer, and A. S. R. Manstead, “Gender and emotion regulation: a social appraisal perspective on anger.,” in Emotion regulation and well-being., Evers, Catharine: Department of Clinical and Health Psychology, Utrecht University, P.O. Box 80140, Utrecht, Netherlands, 3508 TC, [email protected]: Springer Science + Business Media, 2011, pp. 211–222.

[47]

R. A. Fabes and C. L. Martin, “Gender and age stereotypes of emotionality.,” Personal. Soc. Psychol. Bull., vol. 17, no. 5, pp. 532–540, 1991.

[48]

N. Eisenberg and R. A. Fabes, “The relation of young children's vicarious emotional responding to social competence, regulation, and emotionality,” Cogn. Emot., vol. 9, no. 2–3, pp. 203–228, 1995.

[49]

U. Dimberg and L.-O. Lundquist, “Gender differences in facial reactions to facial expressions.,” Biological Psychology, vol. 30, no. 2. Elsevier Science, Netherlands, pp. 151–159, 1990.

Cited By

Gaya-Morey FRamis-Guarinos SManresa-Yee CBuades-Rubio J(2024)Unveiling the human-like similarities of automatic facial expression recognition: An empirical exploration through explainable aiMultimedia Tools and Applications10.1007/s11042-024-20090-583:38(85725-85753)Online publication date: 28-Aug-2024
https://doi.org/10.1007/s11042-024-20090-5
Testa BXiao YSharma HGump ASalekin A(2023)Privacy against Real-Time Speech Emotion Detection via Acoustic Adversarial Evasion of Machine LearningProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108877:3(1-30)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610887
Kolahdouzi MEtemad A(2023)Toward Fair Facial Expression Recognition with Improved Distribution AlignmentProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614141(574-583)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614141
Show More Cited By

Index Terms

Facial Expression Recognition: Impact of Gender on Fairness and Expressions∗
1. Software and its engineering
  1. Software creation and management
    1. Software development process management
      1. Software development methods
        Agile software development

Recommendations

Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Face Recognition Through Different Facial Expressions

Face recognition has become an accessible issue for experts as well as ordinary people as it is a focal non-interfering biometric modality. In this paper, we introduced a new approach to perform face recognition under varying facial expressions. The ...
Facial expression recognition with Convolutional Neural Networks

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

Interacción '22: Proceedings of the XXII International Conference on Human Computer Interaction

September 2022

104 pages

ISBN:9781450397025

DOI:10.1145/3549865

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Agencia Estatal de Investigación/MCIN

Conference

Interaccion 2022

Interaccion 2022: XXII International Conference on Human Computer Interaction

September 7 - 9, 2022

Teruel, Spain

Acceptance Rates

Overall Acceptance Rate 109 of 163 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
197
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)2

Reflects downloads up to 29 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gaya-Morey FRamis-Guarinos SManresa-Yee CBuades-Rubio J(2024)Unveiling the human-like similarities of automatic facial expression recognition: An empirical exploration through explainable aiMultimedia Tools and Applications10.1007/s11042-024-20090-583:38(85725-85753)Online publication date: 28-Aug-2024
https://doi.org/10.1007/s11042-024-20090-5
Testa BXiao YSharma HGump ASalekin A(2023)Privacy against Real-Time Speech Emotion Detection via Acoustic Adversarial Evasion of Machine LearningProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108877:3(1-30)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610887
Kolahdouzi MEtemad A(2023)Toward Fair Facial Expression Recognition with Improved Distribution AlignmentProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614141(574-583)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614141
Uddin MYin LCanavan S(2023)Spatio-Temporal Graph Analytics on Secondary Affect Data for Improving Trustworthy Emotional AIIEEE Transactions on Affective Computing10.1109/TAFFC.2023.329669515:1(30-49)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1109/TAFFC.2023.3296695
Bello HMarin LSuh SZhou BLukowicz P(2023)InMyFaceInformation Fusion10.1016/j.inffus.2023.10188699:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.inffus.2023.101886

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents