Evaluating the Role of Interactivity on Improving Transparency in Autonomous Agents
Pages 1083 - 1091
Abstract
Autonomous agents are increasingly being deployed amongst human end-users. Yet, human users often have little knowledge of how these agents work or what they will do next. This lack of transparency has already resulted in unintended consequences during AI use: a concerning trend which is projected to increase with the proliferation of autonomous agents. To curb this trend and ensure safe use of AI, assisting users in establishing an accurate understanding of agents that they work with is essential. In this work, we present AI teacher, a user-centered Explainable AI framework to address this need for autonomous agents that follow a Markovian policy. Our framework first computes salient instructions of agent behavior by estimating a user's mental model and utilizing algorithms for sequential decision-making. Next, in contrast to existing solutions, these instructions are presented interactively to the end-users, thereby enabling a personalized approach to improving AI transparency. We evaluate our framework, with emphasis on its interactive features, through experiments with human participants. The experiment results suggest that, relative to non-interactive approaches, interactive teaching can both reduce the amount of time it takes for humans to create accurate mental models of these agents and is subjectively preferred by human users.
Supplementary Material
# Supplementary Material
- Download
- 6.46 MB
References
[1]
Dan Amir and Ofra Amir. 2018. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 1168--1176.
[2]
Ofra Amir, Finale Doshi-Velez, and David Sarne. 2019. Summarizing agent strategies. Autonomous Agents and Multi-Agent Systems, Vol. 33, 5 (2019), 628--644.
[3]
Sule Anjomshoae, Amro Najjar, Davide Calvaresi, and Kary Fr"amling. 2019. Explainable agents and robots: Results from a systematic literature review. In 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, Canada, May 13--17, 2019. International Foundation for Autonomous Agents and Multiagent Systems, 1078--1088.
[4]
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009. A survey of robot learning from demonstration. Robotics and autonomous systems, Vol. 57, 5 (2009), 469--483.
[5]
Ross D. Arnold, Hiroyuki Yamaguchi, and Toshiyuki Tanaka. 2018. Search and rescue with autonomous flying robots through behavior-based cooperative intelligence. Journal of International Humanitarian Action, Vol. 3, 1 (12 2018), 18.
[6]
Chris Baker, Rebecca Saxe, and Joshua Tenenbaum. 2011. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science society, Vol. 33.
[7]
S. Baron-Cohen. 1991. Precursors to a theory of mind: Understanding attention in others. Whiten, Andrew (ed.), Natural theories of mind (1991), 233--251.
[8]
Paul Baxter, Grzegorz Cielniak, Marc Hanheide, and Pral From. 2018. Safe Human-Robot Interaction in Agriculture. In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (Chicago, IL, USA) (HRI '18). Association for Computing Machinery, New York, NY, USA, 59--60. https://doi.org/10.1145/3173386.3177072
[9]
G. S. Becker. 1976. The economic approach to human behavior .University of Chicago press.
[10]
Or Biran and Courtenay V. Cotton. 2017. Explanation and Justification in Machine Learning : A Survey Or.
[11]
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, Vol. 4, 1 (2012), 1--43.
[12]
Ruth Byrne and P.N. Johnson-Laird. 2009. "If' and the problems of conditional reasoning. Trends in cognitive sciences, Vol. 13 (07 2009), 282--7. https://doi.org/10.1016/j.tics.2009.04.003
[13]
Anthony R Cassandra, Leslie Pack Kaelbling, and Michael L Littman. 1994. Acting optimally in partially observable stochastic domains. In Aaai, Vol. 94. 1023--1028.
[14]
Tathagata Chakraborti, Sarath Sreedharan, Sachin Grover, and Subbarao Kambhampati. 2019. Plan Explanations as Model Reconciliation--An Empirical Study. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 258--266.
[15]
Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. 2020. The Emerging Landscape of Explainable Automated Planning & Decision Making. In IJCAI. 4803--4811.
[16]
Thomas Griffiths, Nick Chater, Charles Kemp, Amy Perfors, and Joshua Tenenbaum. 2010. Probabilistic models of cognition: Exploring representations and inductive biases. Trends in cognitive sciences, Vol. 14 (08 2010), 357--64.
[17]
Ahmed Hassanein, Mohanad Elhawary, Nour Jaber, and Mohammed El-Abd. 2015. An autonomous firefighting robot. In 2015 International Conference on Advanced Robotics (ICAR). 530--535. https://doi.org/10.1109/ICAR.2015.7251507
[18]
Bradley Hayes and Julie A Shah. 2017. Improving Robot Controller Transparency Through Autonomous Policy Explanation. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI. ieeexplore.ieee.org, 303--312.
[19]
Thomas Hellström and Suna Bensch. 2018. Understandable Robots. Paladyn, Journal of Behavioral Robotics, Vol. 9 (07 2018), 110--123. https://doi.org/10.1515/pjbr-2018-0009
[20]
Ayanna Howard and Jason Borenstein. 2018. The ugly truth about ourselves and our robot creations: the problem of bias and social inequity. Science and engineering ethics, Vol. 24, 5 (2018), 1521--1536.
[21]
Sandy H. Huang, David Held, Pieter Abbeel, and Anca D. Dragan. 2019. Enabling Robots to Communicate their Objectives. Autonomous Robots, Vol. 43, 2 (February 2019).
[22]
Tobias Huber, Katharina Weitz, Elisabeth André, and Ofra Amir. 2021. Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps. Artificial Intelligence, Vol. 301 (2021), 103571.
[23]
Philip Johnson-Laird. 2010. Mental models and human reasoning. Proceedings of the National Academy of Sciences of the United States of America, Vol. 107 (10 2010), 18243--50. https://doi.org/10.1073/pnas.1012933107
[24]
Lynn Julien-Schultz, Nancy Maynes, and Cilla Dunn. 2010. Managing Direct and Indirect Instruction: A Visual Model to Support Lesson Planning in Pre-Service Programs. The International Journal of Learning: Annual Review, Vol. 17 (2010), 125--140.
[25]
Omar Khan, Pascal Poupart, and James Black. 2009. Minimal sufficient explanations for factored markov decision processes. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 19.
[26]
Joseph Kim, Christian Muise, Ankit Shah, Shubham Agarwal, and Julie Shah. 2019. Bayesian Inference of Linear Temporal Logic Specifications for Contrastive Explanations. In IJCAI. 5591--5598.
[27]
Sau lai Lee, Ivy Yee man Lau, S. Kiesler, and Chi-Yue Chiu. 2005. Human Mental Models of Humanoid Robots. In 2005 IEEE International Conference on Robotics and Automation. 2762--2772.
[28]
Michael S. Lee, Henny Admoni, and Reid Simmons. 2021. Machine Teaching for Human Inverse Reinforcement Learning. Frontiers in Robotics and AI, Vol. 8 (2021), 188. https://doi.org/10.3389/frobt.2021.693050
[29]
Tim Miller, Piers D. L. Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences. ArXiv, Vol. abs/1712.00547 (2017).
[30]
Kevin A. Nguyen, Jenefer Husman, M. A. Trujillo Borrego, Prateek Shekhar, Michael J. Prince, Matt DeMonbrun, Cynthia J. Finelli, Charles Henderson, and Cindy K. Waters. 2017. Students' Expectations, Types of Instruction, and Instructor Strategies Predicting Student Response to Active Learning. International Journal of Engineering Education, Vol. 33 (2017), 2--18.
[31]
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. 2018. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, Vol. 7, 1--2 (2018), 1--179.
[32]
Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors, Vol. 39, 2 (1997), 230--253.
[33]
Martin L Puterman. 2014. Markov decision processes: discrete stochastic dynamic programming .John Wiley & Sons.
[34]
Tiia Ruutmann and Hants Kipper. 2011. Teaching Strategies for Direct and Indirect Instruction in Teaching Engineering. In 2011 14th International Conference on Interactive Collaborative Learning. 107--114.
[35]
David Silver. [n.d.]. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. ( [n.,d.]).
[36]
David Silver, Julian Schrittwieser, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Driessche, Thore Graepel, and Demis Hassabis. 2017. Mastering the game of Go without human knowledge. Nature, Vol. 550 (10 2017), 354--359.
[37]
Aaron Steinfeld, Odest Chadwicke Jenkins, and Brian Scassellati. 2009. The oz of wizard: simulating the human for interaction research. In Proceedings of the 4th ACM/IEEE international conference on Human robot interaction. 101--108.
[38]
Roykrong Sukkerd, Reid Simmons, and David Garlan. 2020. Tradeoff-Focused Contrastive Explanation for MDP Planning. arXiv preprint arXiv:2004.12960 (2020).
[39]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction .MIT press.
[40]
Aaquib Tabrez, Shivendra Agrawal, and Bradley Hayes. 2019. Explanation-based Reward Coaching to Improve Human Performance via Reinforcement Learning. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 249 -- 257.
[41]
Nicholay Topin and Manuela Veloso. 2019. Generation of policy-level explanations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2514--2521.
[42]
Giulia Vilone and Luca Longo. 2020. Explainable Artificial Intelligence: a Systematic Review. (05 2020).
[43]
Lihui Wang, Sichao Liu, Hongyi Liu, and Xi Vincent Wang. 2020. Overview of Human-Robot Collaboration in Manufacturing. In Proceedings of 5th International Conference on the Industry 4.0 Model for Advanced Manufacturing, Lihui Wang, Vidosav D. Majstorovic, Dimitris Mourtzis, Emanuele Carpanzano, Govanni Moroni, and Luigi Maria Galantucci (Eds.). Springer International Publishing, Cham, 15--58.
[44]
Yusen Zhan, Anestis Fachantidis, Ioannis Vlahavas, and Matthew E. Taylor. 2014. Agents Teaching Humans in Reinforcement Learning Tasks. In International Conference on Autonomous Agents and Multiagent Systems.
Index Terms
- Evaluating the Role of Interactivity on Improving Transparency in Autonomous Agents
Recommendations
A hybrid agent architecture for modeling autonomous agents in SAGE
IDEAL'05: Proceedings of the 6th international conference on Intelligent Data Engineering and Automated LearningThis paper highlights the Hybrid agent construction model being developed that allows the description and development of autonomous agents in SAGE (Scalable, fault Tolerant Agent Grooming Environment) – a second generation FIPA-Compliant Multi-Agent ...
Autonomous Learning Agents: Layered Learning and Ad Hoc Teamwork
AAMAS '16: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent SystemsIn order to achieve long-term autonomy in the real world, fully autonomous agents need to be able to learn, both to improve their behaviors in a complex, dynamically changing world, and to enable interaction with previously unfamiliar agents. This talk ...
Autonomy of autonomous agents
PRICAI'00: Proceedings of the 6th Pacific Rim international conference on Artificial intelligenceAutonomy is one of the most notable attributes of agency, and this paper presents a formal framework for modelhng it. By representing the mental states of agents, we provide an analysis of some attributes of autonomy. In particular, we define three ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2022
1990 pages
ISBN:9781450392136
- General Chairs:
- Catherine Pelachaud,
- Matthew E. Taylor,
- Program Chairs:
- Piotr Faliszewski,
- Viviana Mascardi
Sponsors
Publisher
International Foundation for Autonomous Agents and Multiagent Systems
Richland, SC
Publication History
Published: 09 May 2022
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Army Research Office
Conference
AAMAS ' 22
Sponsor:
AAMAS ' 22: International Conference on Autonomous Agents and Multi-Agent Systems
May 9 - 13, 2022
Virtual Event, New Zealand
Acceptance Rates
Overall Acceptance Rate 1,155 of 5,036 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 175Total Downloads
- Downloads (Last 12 months)111
- Downloads (Last 6 weeks)15
Reflects downloads up to 22 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in