research-article

Public Access

Automated rationale generation: a technique for explainable AI and its effects on human perceptions

Authors:

Pradyumna Tambwekar,

Brent Harrison,

Mark O. RiedlAuthors Info & Claims

IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces

Pages 263 - 274

https://doi.org/10.1145/3301275.3302316

Published: 17 March 2019 Publication History

Abstract

Automated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human-like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike-ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.

Supplementary Material

MP4 File (p263-ehsan.mp4)

Download
186.48 MB

References

[1]

2017. streamproc/MediaStreamRecorder. (Aug 2017). https://github.com/streamproc/MediaStreamRecorder

[2]

Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankanhalli. 2018. Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 582.

Digital Library

[3]

Jacob Andreas, Anca Dragan, and Dan Klein. 2017. Translating neuralese. arXiv preprint arXiv:1704.06960 (2017).

[4]

J Aronson. 1994. A pragmatic view of thematic analysis: the qualitative report, 2,(1) Spring. (1994).

[5]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).

[6]

Jenay M Beer, Akanksha Prakash, Tracy L Mitzner, and Wendy A Rogers. 2011. Understanding robot acceptance. Technical Report. Georgia Institute of Technology.

[7]

Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018. 'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 377.

Digital Library

[8]

Ned Block. 2005. Two neural correlates of consciousness. Trends in cognitive sciences 9, 2 (2005), 46--52.

[9]

Ned Block. 2007. Consciousness, accessibility, and the mesh between psychology and neuroscience. Behavioral and brain sciences 30, 5-6 (2007), 481--499.

[10]

Joost Broekens, Maaike Harbers, Koen Hindriks, Karel Van Den Bosch, Catholijn Jonker, and John-Jules Meyer. 2010. Do you get it? User-evaluated explainable BDI agents. In German Conference on Multiagent System Technologies. Springer, 28--39.

Digital Library

[11]

John M Carroll. 2000. Making use: scenario-based design of human-computer interactions. MIT press.

Digital Library

[12]

Sonia Chernova and Manuela M Veloso. 2009. A Confidence-Based Approach to Multi-Robot Learning from Demonstration. In AAAI Spring Symposium: Agents that Learn from Human Teachers. 20--27.

[13]

Noel CF Codella, Michael Hind, Karthikeyan Natesan Ramamurthy, Murray Campbell, Amit Dhurandhar, Kush R Varshney, Dennis Wei, and Aleksandra Mojsilovic. 2018. Teaching Meaningful Explanations. arXiv preprint arXiv:1805.11648 (2018).

[14]

Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly (1989), 319--340.

Digital Library

[15]

Munjal Desai, Poornima Kaniarasu, Mikhail Medvedev, Aaron Steinfeld, and Holly Yanco. 2013. Impact of robot failures and feedback on real-time trust. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction. IEEE Press, 251--258.

Digital Library

[16]

Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2016. Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science 64, 3 (2016), 1155--1170.

Digital Library

[17]

Upol Ehsan, Brent Harrison, Larry Chan, and Mark O. Riedl. 2018. Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, Ethics, and Society.

Digital Library

[18]

Neta Ezer, Arthur D Fisk, and Wendy A Rogers. 2009. Attitudinal and intentional acceptance of domestic robots by younger and older adults. In International Conference on Universal Access in Human-Computer Interaction. Springer, 39--48.

Digital Library

[19]

Jerry A Fodor. 1994. The elm and the expert: Mentalese and its semantics. MIT press.

[20]

Marsha E Fonteyn, Benjamin Kuipers, and Susan J Grobe. 1993. A description of think aloud method and protocol analysis. Qualitative health research 3, 4 (1993), 430--441.

[21]

Matthew Guzdial, Joshua Reno, Jonathan Chen, Gillian Smith, and Mark Riedl. 2018. Explainable PCGML via Game Design Patterns. arXiv preprint arXiv:1809.09419 (2018).

[22]

Tad Hirsch, Kritzia Merced, Shrikanth Narayanan, Zac E Imel, and David C Atkins. 2017. Designing contestability: Interaction design, machine learning, and mental health. In Proceedings of the 2017 Conference on Designing Interactive Systems. ACM, 95--99.

Digital Library

[23]

Poornima Kaniarasu, Aaron Steinfeld, Munjal Desai, and Holly Yanco. 2013. Robot confidence and trust alignment. In Human-Robot Interaction (HRI), 2013 8th ACM/IEEE International Conference on. IEEE, 155--156.

Digital Library

[24]

Minae Kwon, Sandy H Huang, and Anca D Dragan. 2018. Expressing Robot Incapability. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 87--95.

Digital Library

[25]

Min Kyung Lee, Sara Kiesler, Jodi Forlizzi, Siddhartha Srinivasa, and Paul Rybski. 2010. Gracefully mitigating breakdowns in robotic services. In Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on. IEEE, 203--210.

Digital Library

[26]

Peter Lipton. 2001. What good is an explanation? In Explanation. Springer, 43--59.

[27]

Z. C. Lipton. 2016. The Mythos of Model Interpretability. ArXiv e-prints (June 2016).

[28]

Leib Litman, Jonathan Robinson, and Tzvi Abberbock. 2017. TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior research methods 49,2 (2017), 433--442.

[29]

Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015).

[30]

Shadd Maruna and Ruth E Mann. 2006. A fundamental attribution error? Rethinking cognitive distortions. Legal and Criminological Psychology 11, 2 (2006), 155--177.

[31]

Tim Miller. 2017. Explanation in artificial intelligence: insights from the social sciences. arXiv preprint arXiv:1706.07269 (2017).

[32]

Nicole Mirnig, Gerald Stollnberger, Markus Miksch, Susanne Stadler, Manuel Giuliani, and Manfred Tscheligi. 2017. To err is robot: How humans assess and act toward an erroneous social robot. Frontiers in Robotics and AI 4 (2017), 21.

[33]

Clifford Nass, BJ Fogg, and Youngme Moon. 1996. Can computers be teammates? International Journal of Human-Computer Studies 45, 6 (1996), 669--678.

Digital Library

[34]

Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of social issues 56, 1 (2000), 81--103.

[35]

Clifford Nass, Jonathan Steuer, Lisa Henriksen, and D Christopher Dryer. 1994. Machines, social attributions, and ethopoeia: Performance assessments of computers subsequent to" self-" or" other-" evaluations. International Journal of Human-Computer Studies 40,3 (1994),543--559.

Digital Library

[36]

Emilee Rader, Kelley Cotter, and Janghee Cho. 2018. Explanations as Mechanisms for Supporting Algorithmic Transparency. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 103.

Digital Library

[37]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 1135--1144.

Digital Library

[38]

Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. Handbook of qualitative research 17 (1994),273--85.

[39]

Viswanath Venkatesh, Michael G Morris, Gordon B Davis, and Fred D Davis. 2003. User acceptance of information technology: Toward a unified view. MIS quarterly (2003), 425--478.

Digital Library

[40]

Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. 2017. Residual Attention Network for Image Classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3156--3164.

[41]

Christopher Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3-4 (1992), 279--292.

Digital Library

[42]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning. 2048--2057.

Digital Library

[43]

Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015).

[44]

Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4651--4659.

Cited By

Cappuccio EFadda DLanzilotti RRinzivillo S(2025)FIPER: A Visual-Based Explanation Combining Rules and Feature ImportanceMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74633-8_11(171-184)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74633-8_11
Isakov APeregorodiev DTomilov IYe CGusarova NVatian ABoukhanovsky A(2024)Real-Time Scheduling with Independent Evaluators: Explainable Multi-Agent ApproachTechnologies10.3390/technologies1212025912:12(259)Online publication date: 14-Dec-2024
https://doi.org/10.3390/technologies12120259
Naveed SStevens GRobin-Kern D(2024)An Overview of the Empirical Evaluation of Explainable AI (XAI): A Comprehensive Guideline for User-Centered Evaluation in XAIApplied Sciences10.3390/app14231128814:23(11288)Online publication date: 3-Dec-2024
https://doi.org/10.3390/app142311288
Show More Cited By

Index Terms

Automated rationale generation: a technique for explainable AI and its effects on human perceptions
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Natural language generation
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach
HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence
Abstract
Explanations—a form of post-hoc interpretability—play an instrumental role in making systems accessible as AI continues to proliferate complex and sensitive sociotechnical systems. In this paper, we introduce Human-centered Explainable AI (HCXAI) ...
Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations
AIES '18: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society

We introduce \em AI rationalization, an approach for generating explanations of autonomous system behavior as if a human had performed the behavior. We describe a rationalization technique that uses neural machine translation to translate internal state-...
Explainable Argumentation for Wellness Consultation
Explainable, Transparent Autonomous Agents and Multi-Agent Systems
Abstract
There has been a recent resurgence in the area of explainable artificial intelligence as researchers and practitioners seek to provide more transparency to their algorithms. Much of this research is focused on explicitly explaining decisions or ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces

March 2019

713 pages

ISBN:9781450362726

DOI:10.1145/3301275

General Chairs:
Wai-Tat Fu,
Shimei Pan,
Program Chairs:
Oliver Brdiczka,
Polo Chau,
Gaelle Calvary

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

ONR

Conference

IUI '19

Sponsor:

IUI '19: 24th International Conference on Intelligent User Interfaces

March 17 - 20, 2019

California, Marina del Ray

Acceptance Rates

IUI '19 Paper Acceptance Rate 71 of 282 submissions, 25%;

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

133
Total Citations
View Citations
5,939
Total Downloads

Downloads (Last 12 months)1,014
Downloads (Last 6 weeks)112

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cappuccio EFadda DLanzilotti RRinzivillo S(2025)FIPER: A Visual-Based Explanation Combining Rules and Feature ImportanceMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74633-8_11(171-184)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74633-8_11
Isakov APeregorodiev DTomilov IYe CGusarova NVatian ABoukhanovsky A(2024)Real-Time Scheduling with Independent Evaluators: Explainable Multi-Agent ApproachTechnologies10.3390/technologies1212025912:12(259)Online publication date: 14-Dec-2024
https://doi.org/10.3390/technologies12120259
Naveed SStevens GRobin-Kern D(2024)An Overview of the Empirical Evaluation of Explainable AI (XAI): A Comprehensive Guideline for User-Centered Evaluation in XAIApplied Sciences10.3390/app14231128814:23(11288)Online publication date: 3-Dec-2024
https://doi.org/10.3390/app142311288
Tambwekar PGombolay M(2024)Towards reconciling usability and usefulness of policy explanations for sequential decision-making systemsFrontiers in Robotics and AI10.3389/frobt.2024.137549011Online publication date: 22-Jul-2024
https://doi.org/10.3389/frobt.2024.1375490
Mendez Guzman ESchlegel VBatista-Navarro R(2024)From outputs to insights: a survey of rationalization approaches for explainable text classificationFrontiers in Artificial Intelligence10.3389/frai.2024.13635317Online publication date: 23-Jul-2024
https://doi.org/10.3389/frai.2024.1363531
Herrewijnen ENguyen DBex Fvan Deemter K(2024)Human-annotated rationales and explainable text classification: a surveyFrontiers in Artificial Intelligence10.3389/frai.2024.12609527Online publication date: 24-May-2024
https://doi.org/10.3389/frai.2024.1260952
Al-Ansari NAl-Thani DAl-Mansoori R(2024)User‐Centered Evaluation of Explainable Artificial Intelligence (XAI): A Systematic Literature ReviewHuman Behavior and Emerging Technologies10.1155/2024/46288552024:1Online publication date: 15-Jul-2024
https://doi.org/10.1155/2024/4628855
Pramanick PRaggioli LRossi ARossi S(2024)Effects of Incoherence in Multimodal Explanations of Robot FailuresCompanion Proceedings of the 26th International Conference on Multimodal Interaction10.1145/3686215.3690155(6-10)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3686215.3690155
Ehsan URiedl M(2024)Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language ModelsProceedings of the Halfway to the Future Symposium10.1145/3686169.3686185(1-8)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3686169.3686185
Luo SIvison HHan SPoon J(2024)Local Interpretations for Explainable Natural Language Processing: A SurveyACM Computing Surveys10.1145/364945056:9(1-36)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3649450
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten