Towards Explaining Sequences of Actions in Multi-Agent Deep Reinforcement Learning Models
Authors: 
Khaing Phyo Wai, 
Minghong Geng, Budhitama Subagdja, Shubham Pateria, Ah-Hwee TanAuthors Info & Claims
Pages 2325 - 2327
Abstract
Although Multi-agent Deep Reinforcement Learning (MADRL) has shown promising results in solving complex real-world problems, the applicability and reliability of MADRL models are often limited by a lack of understanding of their inner workings for explaining the decisions made. To address this issue, this paper proposes a novel method for explaining MADRL by generalizing the sequences of action events performed by agents into high-level abstract strategies using a spatio-temporal neural network model. Specifically, an interval-based memory retrieval procedure is developed to generalize the encoded sequences of action events over time into short sequential patterns. In addition, two abstraction algorithms are introduced, one for abstracting action events across multiple agents and the other for further abstracting the episodes over time into short sequential patterns, which can then be translated into symbolic form for interpretation. We evaluate the proposed method using the StarCraft Multi Agent Challenge (SMAC) benchmark task, which shows that the method is able to derive high-level explanations of MADRL models at various levels of granularity.
References
[1]
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion 58 (2020), 82--115.
[2]
Angel Ayala, Francisco Cruz, Bruno J. T. Fernandes, and Richard Dazeley. 2021. Explainable Deep Reinforcement Learning Using Introspection in a Non-episodic Task. CoRR abs/2108.08911 (2021). arXiv:2108.08911
[3]
Davide Calvaresi, Amro Najjar, Michael Winikoff, and Kary Främling. 2020. Explainable, Transparent Autonomous Agents and Multi-Agent Systems. Springer.
[4]
Poo-Hee Chang and Ah-Hwee Tan. 2017. Encoding and recall of spatio-temporal episodic memory in real time. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. 1490--1496.
[5]
Zichen Chen, Budhitama Subagdja, and Ah-Hwee Tan. 2019. End-to-end deep reinforcement learning for multi-agent collaborative exploration. In 2019 IEEE International Conference on Agents (ICA). IEEE, 99--102.
[6]
Wei Du and Shifei Ding. 2021. A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artificial Intelligence Review 54, 5 (2021), 3215--3238.
[7]
IU Ekanayake, DPP Meddage, and Upaka Rathnayake. 2022. A novel approach to explain the black-box nature of machine learning in compressive strength predictions of concrete using Shapley additive explanations (SHAP). Case Studies in Construction Materials 16 (2022), e01059.
[8]
Vilde B Gjærum, Inga Strümke, Jakob Løver, Timothy Miller, and Anastasios M Lekkas. 2023. Model tree methods for explaining deep reinforcement learning agents in real-time robotic applications. Neurocomputing 515 (2023), 133--144.
[9]
Sven Gronauer and Klaus Diepold. 2022. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review (2022), 1--49.
[10]
Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. 2021. Edge: Explaining deep reinforcement learning policies. Advances in Neural Information Processing Systems 34 (2021), 12222--12236.
[11]
Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. 2021. Explainability in deep reinforcement learning. Knowledge-Based Systems 214 (2021), 106685.
[12]
Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez. 2022. Collective explainable AI: Explaining cooperative strategies and agent contribution in multiagent reinforcement learning with shapley values. IEEE Computational Intelligence Magazine 17, 1 (2022), 59--71.
[13]
Yue Hu, Budhitama Subagdja, Ah-Hwee Tan, Chai Quek, and Quanjun Yin. 2022. Who are the ?silent spreaders'?: Contact tracing in spatio-temporal memory models. Neural Computing and Applications 34, 17 (2022), 14859--14879.
[14]
Stefan Kolek, Duc Anh Nguyen, Ron Levie, Joan Bruna, and Gitta Kutyniok. 2022. A rate-distortion framework for explaining black-box model decisions. In xxAI-Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers. Springer, 91--115.
[15]
Theocharis Kravaris, Konstantinos Lentzos, Georgios Santipantakis, George A Vouros, Gennady Andrienko, Natalia Andrienko, Ian Crook, Jose Manuel Cordero Garcia, and Enrique Iglesias Martinez. 2022. Explaining deep reinforcement learning decisions in complex multiagent settings: towards enabling automation in air traffic flow management. Applied Intelligence (2022), 1--36.
[16]
Deepthi Praveenlal Kuttichira, Sunil Gupta, Cheng Li, Santu Rana, and Svetha Venkatesh. 2019. Explaining Black-Box Models Using Interpretable Surrogates. In PRICAI 2019: Trends in Artificial Intelligence: 16th Pacific Rim International Conference on Artificial Intelligence, Cuvu, Yanuca Island, Fiji, August 26-30, 2019, Proceedings, Part I. 3--15.
[17]
Pascal Leroy, Jonathan Pisane, and Damien Ernst. 2022. Value-based CTDE Methods in Symmetric Two-team Markov Game: from Cooperation to Team Competition. In Deep Reinforcement Learning Workshop NeurIPS 2022. https: //openreview.net/forum?id=wDLQXjkzrS7
[18]
Q. Vera Liao and Kush R. Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. CoRR abs/2110.10790 (2021). arXiv:2110.10790
[19]
Tania Lombrozo. 2007. Simplicity and probability in causal explanation. Cognitive psychology 55, 3 (2007), 232--257.
[20]
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017).
[21]
Dang Minh, H Xiang Wang, Y Fen Li, and Tan N Nguyen. 2022. Explainable artificial intelligence: a comprehensive review. Artificial Intelligence Review (2022), 1--66.
[22]
Afshin Oroojlooy and Davood Hajinezhad. 2022. A review of cooperative multiagent deep reinforcement learning. Applied Intelligence (2022), 1--46.
[23]
Aske Plaat, Walter Kosters, and Mike Preuss. 2023. High-accuracy model-based reinforcement learning, a survey. Artificial Intelligence Review (2023), 1--33.
[24]
Erika Puiutta and Eric Veith. 2020. Explainable Reinforcement Learning: A Survey. In 4th International Cross-Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE). Springer International Publishing, 77--95.
[25]
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2018. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning. PMLR, 4295--4304.
[26]
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. 2019. The StarCraft Multi-Agent Challenge. International Foundation for Autonomous Agents and Multiagent Systems, 2186--2188. http://dl.acm.org/citation.cfm?id=3332052
[27]
Alexander Sieusahai and Matthew Guzdial. 2021. Explaining deep reinforcement learning agents in the atari domain through a surrogate model. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 17. 82--90.
[28]
Tom Silver, Ashay Athalye, Joshua B Tenenbaum, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. 2022. Learning Neuro-Symbolic Skills for Bilevel Planning. In 6th Annual Conference on Robot Learning. https://openreview.net/forum?id=OIaJRUo5UXy
[29]
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. 2019. Qtran: Learning to factorize with transformation for cooperative multiagent reinforcement learning. In International conference on machine learning. PMLR, 5887--5896.
[30]
Budhitama Subagdja and Ah-Hwee Tan. 2015. Neural modeling of sequential inferences and learning over episodic memory. Neurocomputing 161 (2015), 229--242.
[31]
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2018. Value-Decomposition Networks For Cooperative Multi- Agent Learning Based On Team Reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 2085--2087.
[32]
Ah-Hwee Tan, Budhitama Subagdja, Di Wang, and Lei Meng. 2019. Selforganizing neural networks for universal learning and multimodal memory encoding. Neural Networks 120 (2019), 58--73.
[33]
Philipp Theumer, Florian Edenhofner, Roland Zimmermann, and Alexander Zipfel. 2022. Explainable Deep Reinforcement Learning for Production Control. In Proceedings of the Conference on Production Systems and Logistics: CPSL 2022. Hannover: publish-Ing., 809--818.
[34]
Pulkit Verma, Shashank Rao Marpally, and Siddharth Srivastava. 2022. Discovering User-Interpretable Capabilities of Black-Box Planning Agents. In Proceedings of the 19th International Conference on Principles of Knowledge Representation and Reasoning. 362--372. https://doi.org/10.24963/kr.2022/36
[35]
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John P. Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy P. Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. 2017. StarCraft II: A New Challenge for Reinforcement Learning. CoRR abs/1708.04782 (2017). arXiv:1708.04782
[36]
George A Vouros. 2022. Explainable deep reinforcement learning: state of the art and challenges. Comput. Surveys 55, 5 (2022), 1--39.
[37]
Wenwen Wang, Budhitama Subagdja, Ah-Hwee Tan, and Janusz A Starzyk. 2012. Neural modeling of episodic memory: Encoding, retrieval, and forgetting. IEEE transactions on neural networks and learning systems 23, 10 (2012), 1574--1586.
[38]
XiangjunWang, Junxiao Song, Penghui Qi, Peng Peng, Zhenkun Tang,Wei Zhang, Weimin Li, Xiongjun Pi, Jujie He, Chao Gao, et al. 2021. SCC: An efficient deep reinforcement learning agent mastering the game of StarCraft II. In International conference on machine learning. PMLR, 10905--10915.
[39]
Chao Wen, Xinghu Yao, Yuhui Wang, and Xiaoyang Tan. 2020. Smix (??): Enhancing centralized value functions for cooperative multi-agent reinforcement learning. In Proceedings of the AAAI Conference on artificial intelligence, Vol. 34. 7301--7308.
[40]
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. 2021. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems 34 (2021), 3757--3769.
[41]
Weigui Jair Zhou, Budhitama Subagdja, Ah-Hwee Tan, and Darren Wee-Sze Ong. 2021. Hierarchical control of multi-agent reinforcement learning team in real-time strategy (RTS) games. Expert Systems with Applications 186 (2021), 115707.
Index Terms
- Towards Explaining Sequences of Actions in Multi-Agent Deep Reinforcement Learning Models
Recommendations
Explaining Sequences of Actions in Multi-agent Deep Reinforcement Learning Models
AAMAS '24: Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent SystemsThis paper introduces a method to explain MADRL agents' behaviors by abstracting their actions into high-level strategies. Particularly, a spatio-temporal neural network model is applied to encode the agents' sequences of actions as memory episodes ...
GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations
AAMAS '23: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent SystemsCounterfactual explanations are a common tool to explain artificial intelligence models. For Reinforcement Learning (RL) agents, they answer "Why not?" or "What if?" questions by illustrating what minimal change to a state is needed such that an agent ...
MADDPGViz: a visual analytics approach to understand multi-agent deep reinforcement learning
AbstractDeep reinforcement learning (DRL) has received widespread attention recently, where the control policies are trained through deep neural networks. Several visual analytics methods were proposed to reveal the internal mechanism of DRL. However, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2023
3131 pages
ISBN:9781450394321
- General Chairs:
- Noa Agmon,
- Bo An,
- Program Chairs:
- Alessandro Ricci,
- William Yeoh
Sponsors
Publisher
International Foundation for Autonomous Agents and Multiagent Systems
Richland, SC
Publication History
Published: 30 May 2023
Check for updates
Author Tags
Qualifiers
- Poster
Funding Sources
- DSO National Laboratories Singapore
- Singapore Management University
Conference
AAMAS '23
Sponsor:
AAMAS '23: International Conference on Autonomous Agents and Multiagent Systems
May 29 - June 2, 2023
London, United Kingdom
Acceptance Rates
Overall Acceptance Rate 1,155 of 5,036 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 64Total Downloads
- Downloads (Last 12 months)29
- Downloads (Last 6 weeks)1
Reflects downloads up to 14 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in