Cited By
View all- Hernandez-Leal PZhan YTaylor MSucar LMunoz De Cote E(2017)An exploration strategy for non-stationary opponentsAutonomous Agents and Multi-Agent Systems10.1007/s10458-016-9347-331:5(971-1002)Online publication date: 1-Sep-2017
P-MARL: Prediction-Based Multi-Agent Reinforcement Learning for Non-Stationary Environments
Authors: 
Andrei Marinescu, Ivana Dusparic, Adam Taylor, Vinny Cahill, Siobhán ClarkeAuthors Info & Claims
Pages 1897 - 1898
Abstract
Multi-Agent Reinforcement Learning (MARL) is a widely-used technique for optimization in decentralised control problems, addressing complex challenges when several agents change actions simultaneously and without collaboration. Such challenges are exacerbated when the environment in which the agents learn is inherently non-stationary, as agents' actions are then non-deterministic.
In this paper, we show that advance knowledge of environment behaviour through prediction significantly improves agents' performance in converging to near-optimal control solutions. We propose P-MARL, a MARL approach which employs a prediction mechanism to obtain such advance knowledge, which is then used to improve agents' learning. The underlying non-stationary behaviour of the environment is modelled as a time-series and prediction is based on historic data and key environment variables. This provides information regarding potential upcoming changes in the environment, which is a key influencer in agents' decision-making.
We evaluate P-MARL in a smart grid scenario and show that a 92% Pareto efficient solution can be achieved in an electric vehicle charging problem, where energy demand across a community of households is inherently non-stationary. Finally, we analyse the effects of environment prediction accuracy on the performance of our approach.
References
[1]
L. Busoniu, R. Babuska, and B. De Schutter. A comprehensive survey of multiagent reinforcement learning. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 38(2):156--172, 2008.
[2]
B. C. Da Silva, E. W. Basso, A. L. Bazzan, and P. M. Engel. Dealing with non-stationary environments using context detection. In ICML, pages 217--224. ACM, 2006.
[3]
K. Doya, K. Samejima, K.-i. Katagiri, and M. Kawato. Multiple model-based reinforcement learning. Neural computation, 14(6):1347--1369, 2002.
[4]
M. Humphrys. W-learning: Competition among selfish q-learners. Computer Laboratory Technical Report, 362, 1995.
[5]
A. Marinescu, I. Dusparic, C. Harris, V. Cahill, and S. Clarke. A dynamic forecasting method for small scale residential electrical demand. In IJCNN, pages 3767--3774, July 2014.
[6]
A. Marinescu, C. Harris, I. Dusparic, V. Cahill, and S. Clarke. A hybrid approach to very small scale electrical demand forecasting. In ISGT, 2014 IEEE PES, pages 1--5, Feb 2014.
Index Terms
- P-MARL: Prediction-Based Multi-Agent Reinforcement Learning for Non-Stationary Environments
Recommendations
Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments
Multi-agent reinforcement learning (MARL) is a widely researched technique for decentralised control in complex large-scale autonomous systems. Such systems often operate in environments that are continuously evolving and where agents’ actions are non-...
Learning intelligent behavior in a non-stationary and partially observable environment
Individual learning in an environment where more than one agent exist is a challenging task. In this paper, a single learning agent situated in an environment where multiple agents exist is modeled based on reinforcement learning. The environment is non-...
An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games
In this paper, we investigate Reinforcement learning (RL) in multi-agent systems (MAS) from an evolutionary dynamical perspective. Typical for a MAS is that the environment is not stationary and the Markov property is not valid. This requires agents to ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2015
2072 pages
ISBN:9781450334136
- General Chairs:
- Gerhard Weiss,
- Pınar Yolum,
- Program Chairs:
- Rafael H. Bordini,
- Edith Elkind
Sponsors
- IFAAMAS
In-Cooperation
Publisher
International Foundation for Autonomous Agents and Multiagent Systems
Richland, SC
Publication History
Published: 04 May 2015
Check for updates
Author Tags
Qualifiers
- Demonstration
Funding Sources
- Science Foundation Ireland to Lero - the Irish Software Engineering Research Centre
Conference
AAMAS'15
Sponsor:
AAMAS'15: International Conference on Autonomous Agents and Multiagent Systems
May 4 - 8, 2015
Istanbul, Turkey
Acceptance Rates
AAMAS '15 Paper Acceptance Rate 108 of 670 submissions, 16%;
Overall Acceptance Rate 1,155 of 5,036 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 150Total Downloads
- Downloads (Last 12 months)1
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Nov 2024
Other Metrics
Citations
Cited By
View all- Hernandez-Leal PZhan YTaylor MSucar LMunoz De Cote E(2017)An exploration strategy for non-stationary opponentsAutonomous Agents and Multi-Agent Systems10.1007/s10458-016-9347-331:5(971-1002)Online publication date: 1-Sep-2017
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in