Abstract
Recognition of expressions of emotions and affect from facial images is a well-studied research problem in the fields of affective computing and computer vision with a large number of datasets available containing facial images and corresponding expression labels. However, virtually none of these datasets have been acquired with consideration of fair distribution across the human population. Therefore, in this work, we undertake a systematic investigation of bias and fairness in facial expression recognition by comparing three different approaches, namely a baseline, an attribute-aware and a disentangled approach, on two well-known datasets, RAF-DB and CelebA. Our results indicate that: (i) data augmentation improves the accuracy of the baseline model, but this alone is unable to mitigate the bias effect; (ii) both the attribute-aware and the disentangled approaches equipped with data augmentation perform better than the baseline approach in terms of accuracy and fairness; (iii) the disentangled approach is the best for mitigating demographic bias; and (iv) the bias mitigation strategies are more suitable in the existence of uneven attribute distribution or imbalanced number of subgroup data.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alvi, M., Zisserman, A., Nellåker, C.: Turning a blind eye: explicit removal of biases and variation from deep neural network embeddings. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Amini, A., Soleimany, A.P., Schwarting, W., Bhatia, S.N., Rus, D.: Uncovering and mitigating algorithmic bias through learned latent structure. In: AAAI/ACM Conference on AI, Ethics, and Society, AIES (2019)
Bellamy, R.K., et al.: Ai fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 (2018)
Breslin, S., Wadhwa, B.: Gender and Human-Computer Interaction, chap. 4, pp. 71–87. Wiley (2017). https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118976005.ch4
Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency, pp. 77–91 (2018)
Clapes, A., Bilici, O., Temirova, D., Avots, E., Anbarjafari, G., Escalera, S.: From apparent to real age: gender, age, ethnic, makeup, and expression bias analysis in real age estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2373–2382 (2018)
Commission, E.: White paper on artificial intelligence-a european approach to excellence and trust (2020)
Das, A., Dantcheva, A., Bremond, F.: Mitigating bias in gender, age and ethnicity classification: a multi-task convolution neural network approach. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Denton, E.L., Hutchinson, B., Mitchell, M., Gebru, T.: Detecting bias with generative counterfactual face attribute augmentation. ArXiv abs/1906.06439 (2019)
Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., Busch, C.: Demographic bias in biometrics: A survey on an emerging challenge. IEEE Transactions on Technology and Society (2020)
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226 (2012)
Georgopoulos, M., Panagakis, Y., Pantic, M.: Investigating bias in deep face analysis: the kanface dataset and empirical study. Image and Vision Computing, in press (2020)
Gong, S., Liu, X., Jain, A.K.: Mitigating face recognition bias via group adaptive classifier. arXiv preprint arXiv:2006.07576 (2020)
Grother, P., Ngan, M., Hanaoka, K.: Ongoing face recognition vendor test (frvt) part 3: demographic effects. National Institute of Standards and Technology, Tech. Rep. NISTIR 8280 (2019)
Gunes, H., Schuller, B.: Categorical and dimensional affect analysis in continuous input: current trends and future directions. Image Vis. Comput. 31, 120–136 (2013)
Guo, G., Mu, G.: Human age estimation: what is the influence across race and gender? In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 71–78. IEEE (2010)
Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems, pp. 3315–3323 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Howard, A., Zhang, C., Horvitz, E.: Addressing bias in machine learning algorithms: a pilot study on emotion recognition for intelligent systems. In: 2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), pp. 1–7 (2017)
Iosifidis, V., Ntoutsi, E.: Dealing with bias via data augmentation in supervised learning scenarios. Jo Bates Paul D. Clough Robert Jäschke 24 (2018)
Kamiran, F., Calders, T.: Data preprocessing techniques for classification without discrimination. Knowl. Inf. Syst. 33(1), 1–33 (2012)
Khiyari, H., Wechsler, H.: Face verification subject to varying (age, ethnicity, and gender) demographics using deep learning. Journal of Biometrics & Biostatistics 07 (2016). https://doi.org/10.4172/2155-6180.1000323
Kilbride, J.E., Yarczower, M.: Ethnic bias in the recognition of facial expressions. J. Nonverbal Behav. 8(1), 27–41 (1983)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Koenecke, A., et al.: Racial disparities in automated speech recognition. Proc. National Acad. Sci. 117(14), 7684–7689 (2020)
Kuo, C.M., Lai, S.H., Sarkis, M.: A compact deep learning model for robust facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2121–2129 (2018)
Li, S., Deng, W.: Deep facial expression recognition: a survey. In: IEEE Transactions on Affective Computing, pp. 1–1 (2020)
Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)
Liu, Y., Wei, F., Shao, J., Sheng, L., Yan, J., Wang, X.: Exploring disentangled feature representation beyond face identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2080–2089 (2018)
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV), December 2015
Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Schölkopf, B., Bachem, O.: On the fairness of disentangled representations. In: Advances in Neural Information Processing Systems, pp. 14611–14624 (2019)
Lu, K., Mardziel, P., Wu, F., Amancharla, P., Datta, A.: Gender bias in neural natural language processing. ArXiv abs/1807.11714 (2018)
Martinez, B., Valstar, M.F., et al.: Automatic analysis of facial actions: a survey. IEEE Trans. Affective Comput. 10(3), 325–347 (2019)
Mayson, S.G.: Bias in, bias out. YAle lJ 128, 2218 (2018)
Morales, A., Fierrez, J., Vera-Rodriguez, R.: Sensitivenets: Learning agnostic representations with application to face recognition. arXiv preprint arXiv:1902.00334 (2019)
Ngxande, M., Tapamo, J., Burke, M.: Bias remediation in driver drowsiness detection systems using generative adversarial networks. IEEE Access 8, 55592–55601 (2020)
Paszke, A., et al.: Automatic differentiation in pytorch (2017)
Phillips, P.J., Grother, P., Micheals, R., Blackburn, D.M., Tabassi, E., Bone, M.: Face recognition vendor test 2002. In: 2003 IEEE International SOI Conference. Proceedings (Cat. No. 03CH37443), p. 44. IEEE (2003)
du Pin Calmon, F., Wei, D., Vinzamuri, B., Ramamurthy, K.N., Varshney, K.R.: Optimized pre-processing for discrimination prevention. In: Advances in Neural Information Processing Systems (NIPS) (2017)
Rhue, L.: Racial influence on automated perceptions of emotions. Available at SSRN. https://doi.org/10.2139/ssrn.3281765 (2018)
Robinson, J.P., Livitz, G., Henon, Y., Qin, C., Fu, Y., Timoner, S.: Face recognition: too bias, or not too bias? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–1 (2020)
Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015). https://doi.org/10.1109/TPAMI.2014.2366127
Shin, M., Seo, J.H., Kwon, D.S.: Face image-based age and gender estimation with consideration of ethnic difference. In: 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 567–572. IEEE (2017)
Terhörst, P., Kolf, J.N., Damer, N., Kirchbuchner, F., Kuijper, A.: Face quality estimation and its correlation to demographic and non-demographic bias in face recognition. arXiv preprint arXiv:2004.01019 (2020)
Verma, S., Rubin, J.: Fairness definitions explained. In: 2018 IEEE/ACM International Workshop on Software Fairness (FairWare), pp. 1–7. IEEE (2018)
Wang, M., Deng, W.: Mitigate bias in face recognition using skewness-aware reinforcement learning. CoRR abs/1911.10692 (2019), http://arxiv.org/abs/1911.10692
Wang, M., Deng, W., Hu, J., Tao, X., Huang, Y.: Racial faces in the wild: reducing racial bias by information maximization adaptation network. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. pp. 692–702. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00078
Wang, T., Zhao, J., Yatskar, M., Chang, K., Ordonez, V.: Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5309–5318 (2019)
Wang, Z., et al.: Towards fairness in visual recognition: effective strategies for bias mitigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8919–8928 (2020)
Wu, W., Michalatos, P., Protopapaps, P., Yang, Z.: Gender classification and bias mitigation in facial images. In: 12th ACM Conference on Web Science, pp. 106–114 (2020)
Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335–340 (2018)
Acknowledgments
The work of T. Xu and H. Gunes is funded by the European Union’s Horizon 2020 research and innovation programme, under grant agreement No. 826232. S. Kalkan is supported by Scientific and Technological Research Council of Turkey (TÜBİTAK) through BIDEB 2219 Program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Xu, T., White, J., Kalkan, S., Gunes, H. (2020). Investigating Bias and Fairness in Facial Expression Recognition. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12540. Springer, Cham. https://doi.org/10.1007/978-3-030-65414-6_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-65414-6_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65413-9
Online ISBN: 978-3-030-65414-6
eBook Packages: Computer ScienceComputer Science (R0)