Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

Mitigating Demographic Bias in Facial Datasets with Style-Based Multi-attribute Transfer

Published: 01 July 2021 Publication History

Abstract

Deep learning has catalysed progress in tasks such as face recognition and analysis, leading to a quick integration of technological solutions in multiple layers of our society. While such systems have proven to be accurate by standard evaluation metrics and benchmarks, a surge of work has recently exposed the demographic bias that such algorithms exhibit–highlighting that accuracy does not entail fairness. Clearly, deploying biased systems under real-world settings can have grave consequences for affected populations. Indeed, learning methods are prone to inheriting, or even amplifying the bias present in a training set, manifested by uneven representation across demographic groups. In facial datasets, this particularly relates to attributes such as skin tone, gender, and age. In this work, we address the problem of mitigating bias in facial datasets by data augmentation. We propose a multi-attribute framework that can successfully transfer complex, multi-scale facial patterns even if these belong to underrepresented groups in the training set. This is achieved by relaxing the rigid dependence on a single attribute label, and further introducing a tensor-based mixing structure that captures multiplicative interactions between attributes in a multilinear fashion. We evaluate our method with an extensive set of qualitative and quantitative experiments on several datasets, with rigorous comparisons to state-of-the-art methods. We find that the proposed framework can successfully mitigate dataset bias, as evinced by extensive evaluations on established diversity metrics, while significantly improving fairness metrics such as equality of opportunity.

References

[1]
Alvi, M., Zisserman, A., & Nellåker, C. (2018). Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European conference on computer vision (ECCV) (p. 0)
[2]
Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein gan
[3]
Arora, S., Zhang, Y. (2017). Do GANs actually learn the distribution? An empirical study. arXiv preprint arXiv:1706.08224
[4]
Bothwell RK, Brigham JC, and Malpass RS Cross-racial identification Personality and Social Psychology Bulletin 1989 15 1 19-25
[5]
Brock, A., Donahue, J., & Simonyan, K. (2018). Large scale GAN training for high fidelity natural image synthesis.
[6]
Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency (pp. 77–91).
[7]
Carroll JD and Chang JJ Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart–Young decomposition Psychometrika 1970 35 3 283-319
[8]
Chen, B.C., Chen, C.S., & Hsu, W.H. (2014). Cross-age reference coding for age-invariant face recognition and retrieval. In Proceedings of the European conference on computer vision (ECCV).
[9]
Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., & Choo, J. (2018). Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In 2018 IEEE/CVF conference on computer vision and pattern recognition.
[10]
Dantcheva A, Elia P, and Ross A What else does your biometric data reveal? A survey on soft biometrics IEEE Transactions on Information Forensics and Security 2015 11 3 441-467
[11]
Dua, D., & Graff, C. (2017). UCI machine learning repository. http://archive.ics.uci.edu/ml.
[12]
Duong, C.N., Luu, K., Quach, K.G., Nguyen, N., Patterson, E., Bui, T.D., & Le, N. (2019). Automatic face aging in videos via deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 10013–10022).
[13]
Edwards, H., & Storkey, A. (2015). Censoring representations with an adversary. arXiv preprint arXiv:1511.05897.
[14]
Fu Y, Guo G, and Huang TS Age synthesis and estimation via faces: A survey IEEE Transactions on Pattern Analysis and Machine Intelligence 2010 32 11 1955-1976
[15]
Gatys, L.A., Ecker, A.S., & Bethge, M. (2015). A neural algorithm of artistic style. arXiv:1508.06576.
[16]
Georgopoulos, M., Chrysos, G., Pantic, M., & Panagakis, Y. (2020). Multilinear latent conditioning for generating unseen attribute combinations. In International conference on machine learning
[17]
Georgopoulos, M., Oldfield, J., Nicolaou, M.A., Panagakis, Y., & Pantic, M. (2020). Enhancing facial data diversity with style-based face aging. In 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), IEEE (pp. 66–74), Seattle, WA, USA. https://ieeexplore.ieee.org/document/9150573/.
[18]
Georgopoulos M, Panagakis Y, and Pantic M Modeling of facial aging and kinship: A survey Image and Vision Computing 2018 80 58-79
[19]
Georgopoulos, M., Panagakis, Y., & Pantic, M. (2020). Investigating bias in deep face analysis: The kanface dataset and empirical study. Image and Vision Computing.
[20]
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial networks
[21]
Grover, A., Choi, K., Shu, R., & Ermon, S. (2019a). Fair generative modeling via weak supervision. arXiv preprint arXiv:1910.12008.
[22]
Grover, A., Song, J., Kapoor, A., Tran, K., Agarwal, A., Horvitz, E.J., & Ermon, S. (2019b). Bias correction of learned generative models using likelihood-free importance weighting. In Advances in neural information processing systems (pp. 11058–11070).
[23]
Hardt, M., Price, E., Srebro, N., & et al. (2016). Equality of opportunity in supervised learning. In Advances in neural information processing systems (pp. 3315–3323).
[24]
Harshman, R. A., et al. (1970). Foundations of the parafac procedure: Models and conditions for an “ explanatory” multimodal factor analysis.
[25]
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE conference on computer vision and pattern recognition (CVPR).
[26]
He, Z., Zuo, W., Kan, M., Shan, S., & Chen, X. (2017). Arbitrary facial attribute editing: Only change what you want. arXiv:1711.10678.
[27]
He Z, Zuo W, Kan M, Shan S, and Chen X Attgan: Facial attribute editing by only changing what you want IEEE Transactions on Image Processing 2019 28 11 54645478
[28]
Hendricks, L.A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018). Women also snowboard: Overcoming bias in captioning models. In European conference on computer vision (pp. 793–811). Springer.
[29]
Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudik, M., & Wallach, H. (2019). Improving fairness in machine learning systems: What do industry practitioners need? In: Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1–16).
[30]
Huang, G.B., Mattar, M., Berg, T., & Learned-Miller, E. (2008). Labeled faces in the wild: A database forstudying face recognition in unconstrained environments.
[31]
Huang, X., & Belongie, S. (2017). Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. arXiv:1703.06868.
[32]
Huang, X., Liu, M.Y., Belongie, S., & Kautz, J. (2018). Multimodal unsupervised image-to-image translation. arXiv:1804.04732.
[33]
Inoue, H. (2018). Data augmentation by pairing samples for images classification. arXiv preprint arXiv:1801.02929
[34]
Isola, P., Zhu, J.Y., Zhou, T., & Efros, A.A. (2017). Image-to-image translation with conditional adversarial networks. In 2017 IEEE conference on computer vision and pattern recognition (CVPR).
[35]
Jackson, P. T., Abarghouei, A. A., Bonner, S., Breckon, T. P., & Obara, B. (2019). Style augmentation: data augmentation via style randomization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
[36]
Jayakumar, S.M., Menick, J., Czarnecki, W.M., Schwarz, J., Rae, J., Osindero, S., Teh, Y.W., Harley, T., & Pascanu, R. (2020). Multiplicative interactions and where to find them. In International conference on learning representations.
[37]
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017). Progressive growing of GANs for improved quality, stability, and variation.
[38]
Karras, T., Laine, S., & Aila, T. (2019). A style-based generator architecture for generative adversarial networks. arXiv:1812.04948.
[39]
Kim, B., Kim, H., Kim, K., Kim, S., & Kim, J. (2019). Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 9012–9020).
[40]
Kim, J., Kim, M., Kang, H., & Lee, K. (2020). U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation. arXiv:1907.10830.
[41]
Kingma, D., & Ba, J. (2014). Adam: A method for stochastic optimization. International Conference on Learning Representations.
[42]
Kolda, T. G. (2006). Multilinear operators for higher-order decompositions. Technical Reports, Sandia National Laboratories.
[43]
Kolda TG and Bader BW Tensor decompositions and applications SIAM Review 2009 51 3 455-500
[44]
Kuhlman, C., Jackson, L., & Chunara, R. (2020). No computation without representation: Avoiding data and algorithm biases through diversity. arXiv preprint arXiv:2002.11836.
[45]
Lanitis, A. (2002). FG-NET Aging Database.
[46]
Li, M., Zuo, W., & Zhang, D. (2016). Deep identity-aware transfer of facial attributes.
[47]
Li, S., & Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing.
[48]
Lim, J.H., & Ye, J.C. (2017). Geometric GAN.
[49]
Liu, M.Y., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., & Kautz, J. (2019). Few-shot unsupervised image-to-image translation. arXiv: 1905.01723v2.
[50]
Liu, Y., Li, Q., Sun, Z., & Tan, T. (2019). A3gan: An attribute-aware attentive generative adversarial network for face aging.
[51]
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of international conference on computer vision (ICCV).
[52]
Ma, L., Jia, X., Georgoulis, S., Tuytelaars, T., & Gool, L.V. (2018). Exemplar guided unsupervised image-to-image translation with semantic consistency.
[53]
Madras, D., Creager, E., Pitassi, T., & Zemel, R. (2018). Learning adversarially fair and transferable representations. arXiv preprint arXiv:1802.06309.
[54]
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., & Smolley, S.P. (2017). Least squares generative adversarial networks. In 2017 IEEE international conference on computer vision (ICCV).
[55]
Masi, I., Wu, Y., Hassner, T., & Natarajan, P. (2018) Deep face recognition: A survey. In 2018 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), IEEE (pp. 471–478).
[56]
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019). A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635.
[57]
Merler, M., Ratha, N., Feris, R. S., & Smith, J. R. (2019). Diversity in faces.
[58]
Mescheder, L., Geiger, A., & Nowozin, S. (2018). Which Training Methods for GANs do actually Converge?arXiv:1801.04406 [cs].
[59]
Nagpal, S., Singh, M., Singh, R., Vatsa, M., & Ratha, N. (2019). Deep learning for face recognition: Pride or prejudiced? arXiv preprint arXiv:1904.01219.
[60]
Ng, C.B., Tay, Y.H., & Goi, B.M. (2012). Vision-based human gender recognition: A survey. arXiv preprint arXiv:1204.1611.
[61]
Odena, A., Olah, C., & Shlens, J. (2017). Conditional image synthesis with auxiliary classifier GANs. In Proceedings of the 34th international conference on machine learning (Vol. 70, pp. 2642–2651). JMLR. org.
[62]
Park, T., Liu, M.Y., Wang, T.C., & Zhu, J.Y. (2019). Semantic image synthesis with spatially-adaptive normalization. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR).
[63]
Perarnau, G., van de Weijer, J., Raducanu, B., & Ivarez, J.M. (2016). Invertible conditional GANs for image editing.
[64]
Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621.
[65]
Quadrianto, N., Sharmanska, V., & Thomas, O. (2018). Discovering fair representations in the data domain.
[66]
Quadrianto, N., Sharmanska, V., & Thomas, O. (2019). Discovering fair representations in the data domain. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8227–8236).
[67]
Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks.
[68]
Raji, I.D., & Buolamwini, J. (2019). Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. In Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society (pp. 429–435).
[69]
Ramanathan, N., Chellappa, R., Biswas, S., et al. (2009). Age progression in human faces: A survey. Visual Languages and Computing, 15, 3349–3361.
[70]
Ricanek, K., & Tesafaye, T. (2006). Morph: A longitudinal image database of normal adult age-progression. In 7th international conference on automatic face and gesture recognition (FGR06) (pp. 341–345).
[71]
Rothe R, Timofte R, and Gool LV Deep expectation of real and apparent age from a single image without facial landmarks International Journal of Computer Vision 2018 126 2–4 144-157
[72]
Salimans, T., Goodfellow, I. J., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016). Improved techniques for training GANs. In CoRR. arXiv:1606.03498.
[73]
Sandfort V, Yan K, Pickhardt PJ, and Summers RM Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in CT segmentation tasks Scientific Reports 2019 9 1 1-9
[74]
Sattigeri, P., Hoffman, S.C., Chenthamarakshan, V., & Varshney, K.R. (2018). Fairness GAN.
[75]
Schaich A, Obermeyer S, Kolling T, and Knopf M An own-age bias in recognizing faces with horizontal information Frontiers in Aging Neuroscience 2016 8 264
[76]
Serna, I., Morales, A., Fierrez, J., Cebrian, M., Obradovich, N., & Rahwan, I. (2019). Algorithmic discrimination: Formulation and exploration in deep learning-based face biometrics. arXiv preprint arXiv:1912.01842.
[77]
Shorten C and Khoshgoftaar TM A survey on image data augmentation for deep learning Journal of Big Data 2019 6 1 60
[78]
Tang, H., Liu, H., Xu, D., Torr, P.H.S., & Sebe, N. (2019). Attentiongan: Unpaired image-to-image translation using attention-guided generative adversarial networks.
[79]
Verma, S., & Rubin, J. (2018). Fairness definitions explained. In 2018 IEEE/ACM International Workshop on Software Fairness (FairWare), IEEE (pp. 1–7).
[80]
Wang, M., Deng, W., Hu, J., Tao, X., & Huang, Y. (2019). Racial faces in the wild: Reducing racial bias by information maximization adaptation network. In Proceedings of the IEEE international conference on computer vision, pp. 692–702.
[81]
Wang, W., Cui, Z., Yan, Y., Feng, J., Yan, S., Shu, X., & Sebe, N. (2016). Recurrent face aging. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2378–2386).
[82]
Wang, Z., Qinami, K., Karakozis, I.C., Genova, K., Nair, P., Hata, K., & Russakovsky, O. (2020). Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8919–8928).
[83]
Wang, Z. X., Tang, W. L., & Gao, S. (2018). Face aging with identity-preserved conditional generative adversarial networks. In 2018 IEEE conference on computer vision and pattern recognition (CVPR).
[84]
Yang, H., Huang, D., Wang, Y., & Jain, A. K. (2018). Learning face age progression: A pyramid architecture of GANs. In 2018 IEEE/CVF conference on computer vision and pattern recognition.
[85]
Yang H, Huang D, Wang Y, Wang H, and Tang Y Face aging effect simulation using hidden factor analysis joint sparse representation IEEE Transactions on Image Processing 2016 25 6 2493-2507
[86]
Yucer, S., Akçay, S., Al-Moubayed, N., & Breckon, T. P. (2020). Exploring racial bias within face recognition via per-subject adversarially-enabled data augmentation. arXiv preprint arXiv:2004.08945.
[87]
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018). Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 335–340).
[88]
Zhang, H., Cisse, M., Dauphin, Y.N., & Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.
[89]
Zhang, Z., Song, Y., & Qi, H. (2017). Age progression/regression by conditional adversarial autoencoder. In IEEE conference on computer vision and pattern recognition (CVPR).
[90]
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K. W. (2017). Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457.
[91]
Zhao, S., Ren, H., Yuan, A., Song, J., Goodman, N., & Ermon, S. (2018). Bias and generalization in deep generative models: An empirical study. In Advances in Neural Information Processing Systems (pp. 10792–10801).
[92]
Zheng, X., Chalasani, T., Ghosal, K., Lutz, S., & Smolic, A. (2019). Stada: Style transfer as data augmentation. arXiv preprint arXiv:1909.01056.
[93]
Zhu, J.Y., Park, T., Isola, P., & Efros, A.A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In 2017 IEEE international conference on computer vision (ICCV).

Cited By

View all
  • (2024)Chameleon: Foundation Models for Fairness-Aware Multi-Modal Data Augmentation to Enhance Coverage of MinoritiesProceedings of the VLDB Endowment10.14778/3681954.368201417:11(3470-3483)Online publication date: 30-Aug-2024
  • (2024)Getting Emotional Enough: Analyzing Emotional Diversity in Deepfake AvatarsProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685398(1-12)Online publication date: 13-Oct-2024
  • (2024)Synthetic Data in Human Analysis: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336282146:7(4957-4976)Online publication date: 6-Feb-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image International Journal of Computer Vision
International Journal of Computer Vision  Volume 129, Issue 7
Jul 2021
279 pages

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 July 2021
Accepted: 20 February 2021
Received: 15 May 2020

Author Tags

  1. Data augmentation
  2. Style transfer
  3. Dataset bias
  4. Demographic bias
  5. Algorithmic fairness
  6. Diversity
  7. Age progression

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 27 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Chameleon: Foundation Models for Fairness-Aware Multi-Modal Data Augmentation to Enhance Coverage of MinoritiesProceedings of the VLDB Endowment10.14778/3681954.368201417:11(3470-3483)Online publication date: 30-Aug-2024
  • (2024)Getting Emotional Enough: Analyzing Emotional Diversity in Deepfake AvatarsProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685398(1-12)Online publication date: 13-Oct-2024
  • (2024)Synthetic Data in Human Analysis: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336282146:7(4957-4976)Online publication date: 6-Feb-2024
  • (2024)Learning Dynamic Prototypes for Visual Pattern DebiasingInternational Journal of Computer Vision10.1007/s11263-023-01956-x132:5(1777-1799)Online publication date: 1-May-2024
  • (2024)CCR: Facial Image Editing with Continuity, Consistency and ReversibilityInternational Journal of Computer Vision10.1007/s11263-023-01938-z132:4(1336-1349)Online publication date: 1-Apr-2024
  • (2024)A CNN-based multi-level face alignment approach for mitigating demographic bias in clinical populationsComputational Statistics10.1007/s00180-023-01395-939:5(2557-2579)Online publication date: 1-Jul-2024
  • (2023)Representation Bias in Data: A Survey on Identification and Resolution TechniquesACM Computing Surveys10.1145/358843355:13s(1-39)Online publication date: 13-Jul-2023
  • (2023)Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service”Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581463(1-17)Online publication date: 19-Apr-2023
  • (2023)AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image SynthesisInternational Journal of Computer Vision10.1007/s11263-023-01791-0131:8(1995-2014)Online publication date: 3-May-2023
  • (2022)Enhancing Fairness of Visual Attribute PredictorsComputer Vision – ACCV 202210.1007/978-3-031-26351-4_10(151-167)Online publication date: 4-Dec-2022
  • Show More Cited By

View Options

View options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media