research-article

Not So Fair: The Impact of Presumably Fair Machine Learning Models

Authors:

Mackenzie Jorgensen,

Hannah Richert,

Elizabeth Black,

Natalia Criado,

Jose SuchAuthors Info & Claims

AIES '23: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society

Pages 297 - 311

https://doi.org/10.1145/3600211.3604699

Published: 29 August 2023 Publication History

Abstract

When bias mitigation methods are applied to make fairer machine learning models in fairness-related classification settings, there is an assumption that the disadvantaged group should be better off than if no mitigation method was applied. However, this is a potentially dangerous assumption because a “fair” model outcome does not automatically imply a positive impact for a disadvantaged individual—they could still be negatively impacted. Modeling and accounting for those impacts is key to ensure that mitigated models are not unintentionally harming individuals; we investigate if mitigated models can still negatively impact disadvantaged individuals and what conditions affect those impacts in a loan repayment example. Our results show that most mitigated models negatively impact disadvantaged group members in comparison to the unmitigated models. The domain-dependent impacts of model outcomes should help drive future bias mitigation method development.

Supplemental Material

PDF File

Appendix

Download
610.45 KB

References

[1]

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. 2018. A Reductions Approach to Fair Classification. In Proceedings of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 60–69.

[2]

Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019. Fairness and Machine Learning. fairmlbook.org. http://www.fairmlbook.org.

[3]

R. K. E. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilović, S. Nagar, K. N. Ramamurthy, J. Richards, D. Saha, P. Sattigeri, M. Singh, K. R. Varshney, and Y. Zhang. 2019. AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development 63, 4/5 (2019), 4:1–4:15.

[4]

Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Polity, Medford, MA.

[5]

Sarah Bird, Miroslav Dudík, Hanna Wallach, and Kathleen Walker. 2020. Fairlearn: A toolkit for assessing and improving fairness in AI. Technical Report. Microsoft. 6 pages.

[6]

Miranda Bogen and Aaron Rieke. 2018. Help Wanted: An Examination of Hiring Algorithms, Equity, and Bias. Technical Report.

[7]

Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77–91.

[8]

Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. 2009. Building Classifiers with Independency Constraints. In 2009 IEEE International Conference on Data Mining Workshops. 13–18.

[9]

Alexandra Chouldechova. 2017. Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data 5, 2 (2017), 153–163.

[10]

Kate Crawford. 2017. The Trouble with Bias. NIPS Keynote.

[11]

Natalia Criado, Xavier Ferrer, and Jose Such. 2021. Attesting Digital Discrimination Using Norms. International Journal of Interactive Multimedia and Artificial Intelligence (IJIMAI) 6, 5 (2021), 16–23.

[12]

Natalia Criado and Jose Such. 2019. Digital Discrimination. In Algorithmic Regulation. Oxford University Press.

[13]

Alexander D’Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D. Sculley, and Yoni Halpern. 2020. Fairness is Not Static: Deeper Understanding of Long Term Fairness via Simulation Studies(FAT* ’20). Association for Computing Machinery, New York, NY, USA, 525–534.

[14]

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference. 214–226.

Digital Library

[15]

Xavier Ferrer, Tom van Nuenen, Jose Such, Mark Cote, and Natalia Criado. 2021. Bias and Discrimination in AI: a cross-disciplinary perspective. IEEE Technology and Society 20, 2 (2021), 72–80.

[16]

Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2021. The (Im)Possibility of Fairness: Different Value Systems Require Different Mechanisms for Fair Decision Making. Commun. ACM 64, 4 (2021), 136–143.

Digital Library

[17]

Sara Hajian and Josep Domingo-Ferrer. 2013. A Methodology for Direct and Indirect Discrimination Prevention in Data Mining. IEEE Transactions on Knowledge and Data Engineering 25, 7 (2013), 1445–1459.

Digital Library

[18]

Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. 2016. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.). Vol. 29. Curran Associates, Inc.

[19]

Deborah Hellman. 2020. Measuring Algorithmic Fairness. Virginia Law Review 106 (2020).

[20]

Lily Hu and Yiling Chen. 2018. A Short-Term Intervention for Long-Term Fairness in the Labor Market. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1389–1398.

Digital Library

[21]

Ben Hutchinson and Margaret Mitchell. 2019. 50 Years of Test (Un)fairness: Lessons for Machine Learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency. 49–58.

Digital Library

[22]

Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. 2010. Discrimination Aware Decision Tree Learning. In 2010 IEEE International Conference on Data Mining. IEEE, 869–874.

[23]

Faisal Kamiran, Indrė Žliobaitė, and Toon Calders. 2013. Quantifying Explainable Discrimination and Removing Illegal Discrimination in Automated Decision Making. Knowledge and information systems 35, 3 (2013), 613–644.

[24]

Matt Kusner, Chris Russell, Joshua Loftus, and Ricardo Silva. 2019. Making Decisions that Reduce Discriminatory Impacts. In Proceedings of the 36th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 3591–3600.

[25]

David Lindner, Hoda Heidari, and Andreas Krause. 2021. Addressing the Long-term Impact of ML Decisions via Policy Regret. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Zhi-Hua Zhou (Ed.). International Joint Conferences on Artificial Intelligence Organization, 537–544. Main Track.

[26]

Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed Impact of Fair Machine Learning. In Proceedings of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, Stockholmsmässan, Stockholm, Sweden, 3150–3158.

[27]

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A Survey on Bias and Fairness in Machine Learning. ACM Comput. Surv. 54, 6, Article 115 (2021), 35 pages.

[28]

Jacob Metcalf, Emanuel Moss, 2019. Owning ethics: Corporate logics, silicon valley, and the institutionalization of ethics. Social Research: An International Quarterly 86, 2 (2019), 449–476.

[29]

Arvind Narayanan. 2018. 21 Fairness Definitions and Their Politics. FAT* 2018 Tutorial.

[30]

Ziad Obermeyer and Sendhil Mullainathan. 2019. Dissecting Racial Bias in an Algorithm That Guides Health Decisions for 70 Million People. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 89.

Digital Library

[31]

Cathy O’Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, USA.

Digital Library

[32]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.

Digital Library

[33]

Charan Reddy, Deepak Sharma, Soroush Mehri, Adriana Romero Soriano, Samira Shabanian, and Sina Honari. 2021. Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.). Vol. 1.

[34]

Willy E. Rice. 1996. Race, Gender, "Redlining," and the Discriminatory Access to Loans, Credit, and Insurance: An Historical and Empirical Analysis of Consumers Who Sued Lenders and Insurers in Federal and State Courts, 1950-1995. San Diego Law Review 33 (1996).

[35]

Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and Abstraction in Sociotechnical Systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 59–68.

Digital Library

[36]

Jose Such. 2017. Privacy and Autonomous Systems. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 4761–4767.

[37]

Latanya Sweeney. 2013. Discrimination in Online Ad Delivery: Google Ads, Black Names and White Names, Racial Discrimination, and Click Advertising. Queue 11, 3 (Mar 2013), 10–29.

Digital Library

[38]

Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. 2021. DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks. In Conference on Neural Information Processing Systems(NeurIPS) 2021.

[39]

Tom van Nuenen, Xavier Ferrer, Jose Such, and Mark Cote. 2020. Transparency for Whom? Assessing Discriminatory Artificial Intelligence. IEEE Computer 53 (2020), 36–44.

[40]

Tom van Nuenen, Jose Such, and Mark Cote. 2022. Intersectional Experiences of Unfair Treatment Caused by Automated Computational Systems. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–30.

Digital Library

[41]

Sahil Verma and Julia Rubin. 2018. Fairness Definitions Explained. In 2018 IEEE/ACM International Workshop on Software Fairness (FairWare). IEEE, 1–7.

[42]

Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2021. Bias Preservation in Machine Learning: The Legality of Fairness Metrics Under EU Non-Discrimination Law. West Virginia Law Review 123, 3 (2021).

[43]

Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. 2019. Achieving Causal Fairness through Generative Adversarial Networks. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 1452–1458.

[44]

Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. 2018. FairGAN: Fairness-aware Generative Adversarial Networks. In 2018 IEEE International Conference on Big Data (Big Data). 570–575.

[45]

Muhammad Bilal Zafar. 2019. Discrimination in Algorithmic Decision Making: From Principles to Measures and Mechanisms. Ph. D. Dissertation.

Cited By

Zhang AHosoi A(2024)Structural Interventions and the Dynamics of InequalityProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658952(1014-1030)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658952
Hertweck CLoi MHeitz C(2024)Group Fairness Refocused: Assessing the Social Impact of ML Systems2024 11th IEEE Swiss Conference on Data Science (SDS)10.1109/SDS60720.2024.00034(189-196)Online publication date: 30-May-2024
https://doi.org/10.1109/SDS60720.2024.00034
Alvarez JColmenarejo AElobaid AFabbrizzi SFahimi MFerrara AGhodsi SMougan CPapageorgiou IReyero PRusso MScott KState LZhao XRuggieri S(2024)Policy advice and best practices on bias and fairness in AIEthics and Information Technology10.1007/s10676-024-09746-w26:2Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1007/s10676-024-09746-w
Show More Cited By

Index Terms

Not So Fair: The Impact of Presumably Fair Machine Learning Models
1. Computing methodologies
  1. Machine learning
2. Social and professional topics
  1. User characteristics

Recommendations

Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

In recent years, many incidents have been reported where machine learning models exhibited discrimination among people based on race, sex, age, etc. Research has been conducted to measure and mitigate unfairness in machine learning models. For a machine ...
Researching the democratic impact of open government data: A systematic literature review

This systematic literature review examines the impact of open government data on democratic societies. The paper uses a multidimensional operationalization of democracy, concentrating on monitorial, deliberative and participatory processes. It ...
Algorithmic Impact Assessments and Accountability: The Co-construction of Impacts
FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency

Algorithmic impact assessments (AIAs) are an emergent form of accountability for organizations that build and deploy automated decision-support systems. They are modeled after impact assessments in other domains. Our study of the history of impact ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

AIES '23: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society

August 2023

1026 pages

ISBN:9798400702310

DOI:10.1145/3600211

Editors:
Francesca Rossi
IBM
,
Sanmay Das
George Mason University
,
Jenny Davis
Australian National University
,
Kay Firth-Butterfield
Centre for Trustworthy Technology
,
Alex John
London, Carnegie Mellon University

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Engineering and Physical Sciences Research Council
German Academic Exchange Service

Conference

AIES '23

Sponsor:

SIGAI

AIES '23: AAAI/ACM Conference on AI, Ethics, and Society

August 8 - 10, 2023

QC, Montr\'{e}al, Canada

Acceptance Rates

Overall Acceptance Rate 61 of 162 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
217
Total Downloads

Downloads (Last 12 months)150
Downloads (Last 6 weeks)11

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang AHosoi A(2024)Structural Interventions and the Dynamics of InequalityProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658952(1014-1030)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658952
Hertweck CLoi MHeitz C(2024)Group Fairness Refocused: Assessing the Social Impact of ML Systems2024 11th IEEE Swiss Conference on Data Science (SDS)10.1109/SDS60720.2024.00034(189-196)Online publication date: 30-May-2024
https://doi.org/10.1109/SDS60720.2024.00034
Alvarez JColmenarejo AElobaid AFabbrizzi SFahimi MFerrara AGhodsi SMougan CPapageorgiou IReyero PRusso MScott KState LZhao XRuggieri S(2024)Policy advice and best practices on bias and fairness in AIEthics and Information Technology10.1007/s10676-024-09746-w26:2Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1007/s10676-024-09746-w
Jorgensen MWaller MCocarascu OCriado NRodrigues OSuch JBlack E(2023)Investigating the Legality of Bias Mitigation Methods in the United KingdomIEEE Technology and Society Magazine10.1109/MTS.2023.334146542:4(87-94)Online publication date: Dec-2023
https://doi.org/10.1109/MTS.2023.3341465

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents