Regulating Explainability in Machine Learning Applications -- Observations from a Policy Design Experiment
Authors: 
Nadia Nahar, Jenny Rowlett, Matthew Bray, Zahra Abba Omar, Xenophon Papademetris, Alka Menon, 
Christian KästnerAuthors Info & Claims
Pages 2101 - 2112
Abstract
With the rise of artificial intelligence (AI), concerns about AI applications causing unforeseen harms to safety, privacy, security, and fairness are intensifying. While attempts to create regulations are underway, with initiatives such as the EU AI Act and the 2023 White House executive order, skepticism abounds as to the efficacy of such regulations. This paper explores an interdisciplinary approach to designing policy for the explainability of AI applications, as the widely discussed "right to explanation" associated with the EU General Data Protection Regulation is ambiguous. To develop practical guidance for explainability, we conducted an experimental study that involved continuous collaboration among a team of researchers with AI and policy backgrounds over the course of ten weeks. The objective was to determine whether, through interdisciplinary effort, we can reach consensus on a policy for explainability in AI–one that is clearer, and more actionable and enforceable than current guidelines. We share nine observations, derived from an iterative policy design process, which included drafting the policy, attempting to comply with it (or circumvent it), and collectively evaluating its effectiveness on a weekly basis. Key observations include: iterative and continuous feedback was useful to improve policy drafts over time, discussing evidence of compliance was necessary during policy design, and human-subject studies were found to be an important form of evidence. We conclude with a note of optimism, arguing that meaningful policies can be achieved within a moderate time frame and with limited experience in policy design, as demonstrated by our student researchers on the team. This holds promising implications for policymakers, signaling that practical and effective regulation for AI applications is attainable.
References
[1]
[1] Alkhatib, A. and Bernstein, M. 2019. Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019), 1–13.
[2]
[2] Anderson, R. 2001. Why information security is hard - an economic perspective. Proceedings of the 17th Annual Computer Security Applications Conference (2001), 358–365.
[3]
[3] Arnold, M. et al. 2019. FactSheets: Increasing trust in AI services through supplier’s declarations of conformity. IBM Journal of Research and Development. 63, 4/5 (2019), 6:1–6:13.
[4]
[4] Ayres, I. and Braithwaite, J. 1995. Responsive Regulation: Transcending the Deregulation Debate. Oxford University Press.
[5]
[5] Baik, J.S. 2020. Data Privacy Against Innovation or Against Discrimination?: The Case of the California Consumer Privacy Act (CCPA). Telematics and Informatics. 52, (2020).
[6]
[6] Baum, K. et al. 2023. From fear to action: AI governance and opportunities for all. Frontiers in Computer Science. 5, (2023).
[7]
[7] Bender, E. and Hannah, A. 2023. AI causes real harm. Let’s focus on that over the end-of-humanity hype. Scientific American.
[8]
[8] Benjamin, R. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. John Wiley & Sons.
[9]
[9] Bhatt, U. et al. 2020. Explainable machine learning in deployment. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020), 648–657.
[10]
[10] Bietti, E. 2020. From ethics washing to ethics bashing: a view on tech ethics from within moral philosophy. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020), 210–219.
[11]
[11] Blueprint for an AI Bill of Rights: 2022. https://www.whitehouse.gov/ostp/ai-bill-of-rights/.
[12]
[12] Broughel, J. 2023. The Case For Artificial Intelligence Regulation Is Surprisingly Weak. Forbes Magazine.
[13]
[13] Broussard, M. 2023. More than a Glitch: Confronting Race, Gender, and Ability Bias in Tech. MIT Press.
[14]
[14] Carpenter, D. and Moss, D.A. 2013. Preventing Regulatory Capture: Special Interest Influence and How to Limit it. Cambridge University Press.
[15]
[15] Claypoole, T. and Dickinson, W.B. 2023. Why We Shouldn’t Talk About Regulating AI. Legaltech News.
[16]
[16] Colaner, N. 2022. Is explainable artificial intelligence intrinsically valuable? AI & society. 37, 1 (2022), 231–238.
[17]
[17] Commission, Federal Trade 2016. Using Consumer Reports for Credit Decisions: What to Know About Adverse Action and Risk-Based Pricing Notices. (2016).
[18]
[18] D’Amour, A. et al. 2022. Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research. 23, 226 (2022), 1–61.
[19]
[19] Dastin, J. 2018. Insight - Amazon scraps secret AI recruiting tool that showed bias against women. Reuters.
[20]
[20] De Paor, A. et al. 2017. Algorithmic governance: Developing a research agenda through the power of collective intelligence. Big Data and Society. 4, 2 (2017).
[21]
[21] Diakopoulos, N. 2016. Accountability in algorithmic decision making. Communications of the ACM. 59, 2 (2016), 56–62.
[22]
[22] Dobbin, F. and Sutton, J.R. 1998. The Strength of a Weak State: The Rights Revolution and the Rise of Human Resources Management Divisions. The American journal of sociology. 104, 2 (1998), 441–476.
[23]
[23] Edwards, L. and Veale, M. 2017. Slave to the algorithm? Why a’right to an explanation’is probably not the remedy you are looking for. Duke law and technology review. 16, (2017), 18.
[24]
[24] Ehsan, U. et al. 2021. The Who in Explainable AI: How AI Background Shapes Perceptions of AI Explanations. arXiv [cs.HC].
[25]
[25] Eslami, M. et al. 2015. “I always assumed that I wasn’t really that close to [her]”: Reasoning about Invisible Algorithms in News Feeds. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (2015), 153–162.
[26]
[26] EU: France, Germany and Italy risk unravelling landmark AI Act negotiations: 2023. https://www.amnesty.org/en/latest/news/2023/11/eu-france-germany-and-italy-risk-unravelling-landmark-ai-act-negotiations/.
[27]
[27] Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence: 2023. https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/.
[28]
[28] Eykholt, K. et al. 2018. Robust physical-world attacks on deep learning visual classification. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 1625–1634.
[29]
[29] Fafard, Patrick, Evidence and healthy public policy: Insights from health and political sciences: http://www.ncchpp.ca/docs/FafardEvidence08June.pdf.
[30]
[30] Feffer, M. et al. 2024. Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv [cs.CY].
[31]
[31] Ferreira, G. et al. 2019. Design Dimensions for Software Certification: A Grounded Analysis. arXiv [cs.SE].
[32]
[32] Fjeld, J. et al. 2020. Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI. Berkman Klein Center. 1, (2020).
[33]
[33] Flyvbjerg, B. 2006. Five Misunderstandings About Case-Study Research. Qualitative inquiry: QI. 12, 2 (2006), 219–245.
[34]
[34] Food and Drug Administration 2019. Proposed regulatory framework for modifications to Artificial Intelligence/Machine Learning (AI/ML)-based Software as a Medical Device (SaMD). Department of Health and Human Services (United States). (2019).
[35]
[35] Gebru, T. et al. 2021. Datasheets for datasets. Communications of the ACM. 64, 12 (2021), 86–92.
[36]
[36] Google, Responsible Development of AI: https://ai.google/static/documents/responsible-development-of-ai.pdf.
[37]
[37] Greene, D. et al. 2019. Better, Nicer, Clearer, Fairer: A Critical Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning. Proceedings of the Hawaii International Conference on System Sciences (HICSS-52) (2019).
[38]
[38] Guha, N. et al. 2023. Ai regulation has its own alignment problem: The technical and institutional feasibility of disclosure, registration, licensing, and auditing. George Washington Law Review, Forthcoming. (2023).
[39]
[39] Hagemann, Ryan and Leclerc, Jean-Marc, Precision regulation for artificial intelligence: https://www.ibm.com/policy/wp-content/uploads/2023/04/IBM-AI-POV_FINAL2.pdf.
[40]
[40] Hearn, J. 2004. Does the common criteria paradigm have a future? [security and privacy]. IEEE security & privacy. 2, 1 (2004), 64–65.
[41]
[41] Henshall, W. 2023. E.U.’s AI Regulation Could Be Softened After Pushback From Biggest Members. Time.
[42]
[42] Hohman, F. et al. 2019. Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019), 1–13.
[43]
[43] House, W. 2023. FACT SHEET: Biden-Harris administration secures voluntary commitments from leading artificial intelligence companies to manage the risks posed by AI. The White House. (2023).
[44]
[44] Howlett, M. and Ramesh, M. 2003. Studying Public Policy: Policy Cycles and Policy Subsystems. Oxford University Press.
[45]
[45] Huang, X. et al. 2020. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review. 37, (2020), 100270.
[46]
[46] Juelsen, E. and Thoresen, M.A. 2021. Shapley values in the context of GDPR: Can Shapley Values be used as a means of interpreting black-box machine learning models while also complying with the General Data Protection Regulation? (Master’s thesis).
[47]
[47] Junginger, S. 2013. Design and Innovation in the Public Sector: Matters of Design in Policy-Making and Policy Implementation. Annual Review of Policy Design. 1, 1 (2013), 1–11.
[48]
[48] Kallberg, J. 2012. The Common Criteria Meets Realpolitik: Trust, Alliances, and Potential Betrayal. IEEE Security Privacy. 10, 4 (2012), 50–53.
[49]
[49] Kang, C. 2023. OpenAI’s Sam Altman Urges AI Regulation in Senate Hearing’. The New York times.
[50]
[50] Kästner, C. 2022. Machine Learning in Production: From Models to Products.
[51]
[51] Keblawi, F. and Sullivan, D. 2006. Applying the common criteria in systems engineering. IEEE security & privacy. 4, 2 (2006), 50–55.
[52]
[52] Kiviat, B. 2019. The Moral Limits of Predictive Practices: The Case of Credit-Based Insurance Scores. American sociological review. 84, 6 (2019), 1134–1158.
[53]
[53] Krasadakis, George, To Regulate AI or Not? How should Governments React to the Artificial Intelligence Revolution? 2023. https://medium.com/60-leaders/to-regulate-or-not-how-should-governments-react-to-the-ai-revolution-c254d176304f.
[54]
[54] Kumar, I.E. et al. 2020. Problems with Shapley-value-based explanations as feature importance measures. Proceedings of the 37th International Conference on Machine Learning (2020), 5491–5500.
[55]
[55] Kurakin, A. et al. 2016. Adversarial Machine Learning at Scale. arXiv [cs.CV].
[56]
[56] Linardatos, P. et al. 2020. Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy. 23, 1 (2020), 18.
[57]
[57] Lipner, S.B. 1991. Criteria, evaluation, and the international environment: where have we been, where are we going. Proc. IFIP-SEC. 91, (1991).
[58]
[58] Lipton, Z.C. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queueing Systems. Theory and Applications. 16, 3 (Jun. 2018), 31–57.
[59]
[59] Liu, K.K. 2005. Fair and Accurate Credit Transactions Act Regulations: Disclosure, Opt-Out Rights, Medical Information Usage, and Consumer Information Disposal. ISJLP. 2, (2005), 715.
[60]
[60] Lundberg, S.M. and Lee, S.-I. 2017. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems (NIPS). 30, (2017).
[61]
[61] Luria, M. 2023. Co-Design Perspectives on Algorithm Transparency Reporting: Guidelines and Prototypes. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023), 1076–1087.
[62]
[62] Madaio, M.A. et al. 2020. Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020), 1–14.
[63]
[63] Maibaum, T. and Wassyng, A. 2008. A Product-Focused Approach to Software Certification. Computer. 41, 2 (2008), 91–93.
[64]
[64] Makkai, T. and Braithwaite, J. 1992. In and out of the revolving door: Making sense of regulatory capture. Journal of public policy. 12, 1 (1992), 61–78.
[65]
[65] McGraw, G. et al. 2020. An architectural risk analysis of machine learning systems: Toward more secure machine learning. Technical report, Berryville Institute of Machine Learning. (2020).
[66]
[66] McLennan, S. et al. 2020. An embedded ethics approach for AI development. Nature Machine Intelligence. 2, 9 (2020), 488–490.
[67]
[67] Metcalf, J. et al. 2019. Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics. Social Research: An International Quarterly. 86, 2 (2019), 449–476.
[68]
[68] Microsoft, Voluntary Commitments by Microsoft to Advance Responsible AI Innovation: 2023. https://blogs.microsoft.com/wp-content/uploads/prod/sites/5/2023/07/Microsoft-Voluntary-Commitments-July-21-2023.pdf.
[69]
[69] Mitchell, M. et al. 2019. Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (2019), 220–229.
[70]
[70] Mitchell, T.M. 1997. Machine Learning. McGraw-Hill.
[71]
[71] Molnar, C. 2020. Interpretable Machine Learning. Lulu.com.
[72]
[72] Nahar, Nadia, Supplementary Documents: Regulating Explainability in Machine Learning Applications: 2024. https://osf.io/4xzpr/.
[73]
[73] Panigutti, C. et al. 2023. The role of explainable AI in the context of the AI Act. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023), 1139–1150.
[74]
[74] Pant, A. et al. 2023. Ethics in the Age of AI: An Analysis of AI Practitioners’ Awareness and Challenges. ACM Transactions on Software Engineering and Methodology. (2023).
[75]
[75] Passi, S. and Jackson, S.J. 2018. Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proceedings of the ACM on Human-Computer Interaction. 2, CSCW (2018), 1–28.
[76]
[76] People + AI Guidebook: https://pair.withgoogle.com/guidebook/.
[77]
[77] Perrigo, B. 2023. Exclusive: OpenAI Lobbied the EU to Water Down AI Regulation. Time.
[78]
[78] Pierson, P. 2000. Increasing Returns, Path Dependence, and the Study of Politics. The American political science review. 94, 2 (2000), 251–267.
[79]
[79] Press Releases: Artificial Intelligence Act: MEPs adopt landmark law: 2024. https://www.europarl.europa.eu/news/en/press-room/20240308IPR19015/artificial-intelligence-act-meps-adopt-landmark-law.
[80]
[80] Rong, Y. et al. 2023. Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations. IEEE transactions on pattern analysis and machine intelligence. PP, (2023).
[81]
[81] Rudin, C. 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature machine intelligence. 1, 5 (2019), 206–215.
[82]
[82] Rushby, J. 2011. New challenges in certification for aircraft software. Proceedings of the 9th ACM international conference on Embedded software (2011), 211–218.
[83]
[83] Sagor, R. 2011. The Action Research Guidebook: A Four-Stage Process for Educators and School Teams. Corwin Press.
[84]
[84] Shaw, J.C. et al. 2003. To justify or excuse?: A meta-analytic review of the effects of explanations. The Journal of applied psychology. 88, 3 (2003), 444.
[85]
[85] Siebert, J. et al. 2020. Towards Guidelines for Assessing Qualities of Machine Learning Systems. Proceedings of the 3th International Conference on Quality of Information and Communications Technology (2020), 17–31.
[86]
[86] Silbey, S.S. 2013. Organizational Challenges to Regulatory Enforcement and Compliance. The Annals of the American Academy of Political and Social Science. 649, 1 (2013), 6–20.
[87]
[87] Slayton, R. and Clark-Ginsberg, A. 2018. Beyond regulatory capture: Coproducing expertise for critical infrastructure protection. Regulation & governance. 12, 1 (2018), 115–130.
[88]
[88] Smiley, L. 2023. The Legal Saga of Uber’s Fatal Self-Driving Car Crash Is Over. Wired.
[89]
[89] Sovrano, F. and Vitali, F. 2023. An objective metric for Explainable AI: How and why to estimate the degree of explainability. Knowledge-Based Systems. 278, (2023), 110866.
[90]
[90] Springer, A. et al. 2018. Dice in the black box: User experiences with an inscrutable algorithm. arXiv [cs.HC].
[91]
[91] Stringer, E.T. and Aragón, A.O. 2020. Action Research. SAGE Publications.
[92]
[92] Taherdoost, H. 2023. Enhancing Social Media Platforms with Machine Learning Algorithms and Neural Networks. Algorithms. 16, 6 (2023), 271.
[93]
[93] Teixeira, G.A. et al. 2019. The critical success factors of GDPR implementation: a systematic literature review. Digital Policy, Regulation and Governance. 21, 4 (2019), 402–418.
[94]
[94] Thomas, M. 2007. Unsafe Standardization. Computer. 40, 11 (2007), 109–111.
[95]
[95] Vale, D. et al. 2022. Explainable artificial intelligence (XAI) post-hoc explainability methods: risks and limitations in non-discrimination law. AI and Ethics. 2, 4 (2022), 815–826.
[96]
[96] Veale, M. et al. 2018. Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (2018), 1–14.
[97]
[97] Vera Liao, Q. and Varshney, K.R. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. arXiv [cs.AI].
[98]
[98] Volokh, Eugene, Journal of Free Speech Law: “The European Liability Regime For Large Language Models”: 2023. https://reason.com/volokh/2023/08/11/journal-of-free-speech-law-the-european-liability-regime-for-large-language-models/.
[99]
[99] Vranken, Bram, Big Tech lobbying is derailing the AI Act: https://www.socialeurope.eu/big-tech-lobbying-is-derailing-the-ai-act.
[100]
[100] Wachter, S. et al. 2017. Why a right to explanation of automated decision-making does not exist in the general data protection regulation. SSRN Electronic Journal. 7, 2 (2017), 76–99.
[101]
[101] Wegrich, K. and Jann, W. 2007. Theories of the Policy Cycle. Handbook of Public Policy Analysis. Routledge. 43–62.
[102]
[102] Wheeler, Tom, The three challenges of AI regulation: 2023. https://www.brookings.edu/articles/the-three-challenges-of-ai-regulation/.
Index Terms
- Regulating Explainability in Machine Learning Applications -- Observations from a Policy Design Experiment
Recommendations
The Policy Machine: A novel architecture and framework for access control policy specification and enforcement
The ability to control access to sensitive data in accordance with policy is perhaps the most fundamental security requirement. Despite over four decades of security research, the limited ability for existing access control mechanisms to generically ...
Directive Explanations for Actionable Explainability in Machine Learning Applications
In this article, we show that explanations of decisions made by machine learning systems can be improved by not only explaining why a decision was made but also explaining how an individual could obtain their desired outcome. We formally define the ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
2580 pages
ISBN:9798400704505
DOI:10.1145/3630106
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 June 2024
Check for updates
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
FAccT '24
FAccT '24: The 2024 ACM Conference on Fairness, Accountability, and Transparency
June 3 - 6, 2024
Rio de Janeiro, Brazil
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 241Total Downloads
- Downloads (Last 12 months)241
- Downloads (Last 6 weeks)91
Reflects downloads up to 18 Nov 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in