research-article

Free access

Integrating organizational control into multi-agent learning

Authors:

Chongjie Zhang,

Sherief Abdallah,

Victor LesserAuthors Info & Claims

AAMAS '09: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2

Pages 757 - 764

Published: 10 May 2009 Publication History

Abstract

Multi-Agent Reinforcement Learning (MARL) algorithms suffer from slow convergence and even divergence, especially in large-scale systems. In this work, we develop an organization-based control framework to speed up the convergence of MARL algorithms in a network of agents. Our framework defines a multi-level organizational structure for automated supervision and a communication protocol for exchanging information between lower-level agents and higher-level supervising agents. The abstracted states of lower-level agents travel upwards so that higher-level supervising agents generate a broader view of the state of the network. This broader view is used in creating supervisory information which is passed down the hierarchy. The supervisory policy adaptation then integrates supervisory information into existing MARL algorithms, guiding agents' exploration of their state-action space. The generality of our framework is verified by its applications on different domains (distributed task allocation and network routing) with different MARL algorithms. Experimental results show that our framework improves both the speed and likelihood of MARL convergence.

References

[1]

S. Abdallah and V. Lesser. Learning the task allocation game. In AAMAS'06, 2006.

Digital Library

[2]

S. Abdallah and V. Lesser. Multiagent reinforcement learning and self-organization in a network of agents. In AAMAS'07, 2007.

Digital Library

[3]

R. A. C. Bianchi, C. H. C. Ribeiro, and A. H. R. Costa. Heuristic selection of actions in multiagent reinforcement learning. In IJCAI'07, Hyderabad, India, 2007.

Digital Library

[4]

J. A. Boyan and M. L. Littman. Packet routing in dynamically changing networks: A reinforcement learning approach. In NIPS'94, volume 6, pages 671--678, 1994.

[5]

R. Makar, S. Mahadevan, and M. Ghavamzadeh. Hierarchical multi-agent reinforcement learning. In Autonomous Agents'01, pages 246--253, 2001.

Digital Library

[6]

A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: theory and application to reward shaping. In ICML'99, pages 278--287, 1999.

Digital Library

[7]

L. Peshkin and V. Savova. Reinforcement learning for adaptive routing. In International Joint Conference on Neural Networks (IJCNN), 2002.

[8]

M. T. Rosenstein and A. G. Barto. Supervised actor-critic reinforcement learning. In J. Si, A. Barto, W. Powell, and D. Wunsch, editors, Learning and Approximate Dynamic Programming: Scaling Up to the Real World, pages 359--380. John Wiley and Sons, 2004.

[9]

H. A. Simon. Nearly-decomposable systems. In The Sciences of the Artificial, pages 99--103, 1969.

[10]

S. P. Singh, T. Jaakkola, M. L. Littman, and C. Szepesvari. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning, 38(3):287--308, 2000.

Digital Library

[11]

P. Stone and M. Veloso. Team-partitioned, opaque-transition reinforcement learning. In Autonomous Agents'99, pages 206--212, 1999.

Digital Library

[12]

P. Tangamchit, J. Dolan, and P. Khosla. Learning-based task allocation in decentralized multirobot systems. In DARS'00, pages 381--390, 2000.

[13]

N. Tao, J. Baxter, and L. Weaver. A multi-agent policy-gradient approach to network routing. In ICML '01, pages 553--560, 2001.

Digital Library

[14]

C. Zhang, V. Lesser, and S. Abdallah. Self-organization for dynamically supervising distributed learning. In University of Massachusetts Amherst Computer Science Technical Report UM-CS-2009-007, 2009.

[15]

H. Zhang and V. Lesser. A reinforcement learning based distributed search algorithm for hierarchical content sharing systems. In AAMAS'07, 2007.

Digital Library

[16]

M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In ICML'03, pages 928--936, 2003.

Digital Library

Cited By

Suau MHe JÇelikok MSpaan MOliehoek FKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Distributed influence-augmented local simulators for parallel MARL in large networked systemsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3602322(28305-28318)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3602322
Yang TMeng ZHao JSen SYu C(2016)Accelerating norm emergence through hierarchical heuristic learningProceedings of the Twenty-second European Conference on Artificial Intelligence10.3233/978-1-61499-672-9-1344(1344-1352)Online publication date: 29-Aug-2016
https://dl.acm.org/doi/10.3233/978-1-61499-672-9-1344
Johnson CGonzalez A(2014)Learning collaborative team behavior from observationExpert Systems with Applications: An International Journal10.1016/j.eswa.2013.09.02941:5(2316-2328)Online publication date: 1-Apr-2014
https://dl.acm.org/doi/10.1016/j.eswa.2013.09.029
Show More Cited By

Index Terms

Integrating organizational control into multi-agent learning
1. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence
  2. Machine learning

Recommendations

Combining Dynamic Reward Shaping and Action Shaping for Coordinating Multi-agent Learning
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 02

Coordinating multi-agent reinforcement learning provides a promising approach to scaling learning in large cooperative multi-agent systems. It allows agents to learn local decision policies based on their local observations and rewards, and, meanwhile, ...
Cooperative Multi-Agent Learning: The State of the Art

Cooperative multi-agent systems (MAS) are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multi-agent problem complexity can rise rapidly with the ...
Local strategy learning in networked multi-agent team formation

Networked multi-agent systems are comprised of many autonomous yet interdependent agents situated in a virtual social network. Two examples of such systems are supply chain networks and sensor networks. A common challenge in many networked multi-agent ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

AAMAS '09: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2

May 2009

730 pages

ISBN:9780981738178

General Chairs:
Carles Sierra
Artificial Intelligence Research Institute of the Spanish Research Council (Spain)
,
Cristiano Castelfranchi
ISTC-CNR (Italy)
,
Program Chairs:
Keith S. Decker
University of Delaware
,
Jaime Simão Sichman
Politecnic School, University of São Paulo (Brazil)

Sponsors

Drexel University
Wiley-Blackwell
Microsoft Research: Microsoft Research
Whitestein Technologies
European Office of Aerospace Research and Development, Air Force Office of Scientific Research, United States Air Force Research Laboratory
The Foundation for Intelligent Physical Agents

Publisher

International Foundation for Autonomous Agents and Multiagent Systems

Richland, SC

Publication History

Published: 10 May 2009

Author Tags

Qualifiers

Research-article

Acceptance Rates

Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
332
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)7

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Suau MHe JÇelikok MSpaan MOliehoek FKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Distributed influence-augmented local simulators for parallel MARL in large networked systemsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3602322(28305-28318)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3602322
Yang TMeng ZHao JSen SYu C(2016)Accelerating norm emergence through hierarchical heuristic learningProceedings of the Twenty-second European Conference on Artificial Intelligence10.3233/978-1-61499-672-9-1344(1344-1352)Online publication date: 29-Aug-2016
https://dl.acm.org/doi/10.3233/978-1-61499-672-9-1344
Johnson CGonzalez A(2014)Learning collaborative team behavior from observationExpert Systems with Applications: An International Journal10.1016/j.eswa.2013.09.02941:5(2316-2328)Online publication date: 1-Apr-2014
https://dl.acm.org/doi/10.1016/j.eswa.2013.09.029
Zhang CLesser VGini MShehory OIto TJonker C(2013)Coordinating multi-agent reinforcement learning with limited communicationProceedings of the 2013 international conference on Autonomous agents and multi-agent systems10.5555/2484920.2485093(1101-1108)Online publication date: 6-May-2013
https://dl.acm.org/doi/10.5555/2484920.2485093
Campos JLopez-Sanchez MSalamó MAvila PRodríguez-Aguilar J(2013)Robust Regulation Adaptation in Multi-Agent SystemsACM Transactions on Autonomous and Adaptive Systems10.1145/25173288:3(1-27)Online publication date: 1-Sep-2013
https://dl.acm.org/doi/10.1145/2517328
Zhu XZhang CLesser V(2013)Combining Dynamic Reward Shaping and Action Shaping for Coordinating Multi-agent LearningProceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 0210.1109/WI-IAT.2013.127(321-328)Online publication date: 17-Nov-2013
https://dl.acm.org/doi/10.1109/WI-IAT.2013.127
Lau QLee MHsu Wvan der Hoek WPadgham LConitzer VWinikoff M(2012)Coordination guided reinforcement learningProceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 110.5555/2343576.2343607(215-222)Online publication date: 4-Jun-2012
https://dl.acm.org/doi/10.5555/2343576.2343607
Koeppen JLopez-Sanchez MMorales JEsteva M(2010)Learning from experience to generate new regulationsProceedings of the 6th international conference on Coordination, organizations, institutions, and norms in agent systems10.5555/2018118.2018140(337-356)Online publication date: 1-May-2010
https://dl.acm.org/doi/10.5555/2018118.2018140
Zhang CLesser VAbdallah SLuck MSen S(2010)Self-organization for coordinating decentralized reinforcement learningProceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 110.5555/1838206.1838304(739-746)Online publication date: 10-May-2010
https://dl.acm.org/doi/10.5555/1838206.1838304

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents