Abstract
In this paper, we propose an alternative to model-free reinforcement learning approaches that recently have demonstrated Theory-of-Mind like behaviors. We propose a game theoretic approach to the problem in which pure RL has demonstrated to perform below the standards of human-human interaction. In this context, we propose alternative learning architectures that complement basic RL models with the ability to predict the other’s actions. This architecture is tested in different scenarios where agents equipped with similar or varying capabilities compete in a social game. Our different interaction scenarios suggest that our model-based approaches are especially effective when competing against models of equivalent complexity, in contrast to our previous results with more basic predictive architectures. We conclude that the evolution of mechanisms that allow for the control of other agents provide different kinds of advantages that can become significant when interacting with different kinds of agents. We argue that no single proposed addition to the learning architecture is sufficient to optimize performance in these scenarios, but a combination of the different mechanisms suggested is required to achieve near-optimal performance in any case.
ITF and JP have contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Rabinowitz, N.C., Perbet, F., Song, H.F., Zhang, C., Eslami, S.M., Botvinick, M.: Machine Theory of Mind (2018). arXiv preprint arXiv:1802.07740
Mordatch, I., Abbeel, P.: Emergence of grounded compositional language in multi-agent populations (2017). arXiv preprint arXiv:1703.04908
Hawkins, R.X.D., Goldstone, R.L.: The formation of social conventions in real-time environments. PLoS One 11, e0151670 (2016)
Freire, I.T., Moulin-Frier, C., Sanchez-Fibla, M., Arsiwalla, X.D., Verschure, P.: Modeling the Formation of Social Conventions in Multi-Agent Populations (2018). arXiv preprint arXiv:1802.06108
Verschure, P.F.M.J., Voegtlin, T., Douglas, R.J.: Environmentally mediated synergy between perception and behaviour in mobile robots. Nature 425, 620–624 (2003)
Moulin-Frier, C., Arsiwalla, X.D., Puigbo, J.Y., Sanchez-Fibla, M., Duff, A., Verschure, P.F.: Top-down and bottom-up interactions between low-level reactive control and symbolic rule learning in embodied agents. In: CoCo@ NIPS (2016)
Braitenberg, V.: Vehicles: Experiments in Synthetic Psychology. MIT Press, Cambridge (1986)
Sutton, R.S.: Learning to predict by the methods of temporal differences. Mach. Learn. 3, 9–44 (1988)
Moulin-Frier, C., Puigbo, J.Y., Arsiwalla, X.D., Sanchez-Fibla, M., Verschure, P.F.: Embodied artificial intelligence through distributed adaptive control: An integrated framework (2017). arXiv preprint arXiv:1704.01407
Arsiwalla, X.D., Herreros, I., Moulin-Frier, C., Sanchez, M., Verschure, P.F.: Is consciousness a control process? Artificial Intelligence Research and Development, pp. 233–238. IOS Press, Amsterdam (2016)
Arsiwalla, X.D., Herreros, I., Verschure, P.: On three categories of conscious machines. In: Conference on Biomimetic and Biohybrid Systems, pp. 389–392 (2016)
Arsiwalla, X.D., Herreros, I., Moulin-Frier, C., Verschure, P.: Consciousness as an Evolutionary Game-Theoretic, Strategy, pp. 509–514 (2017)
Arsiwalla, X.D., Moulin-Frier, C., Herreros, I., Sanchez-Fibla, M., Verschure, P.: The Morphospace of Consciousness (2017). ArXiv preprint arXiv:1705.11190
Acknowledgments
The research leading to these results has received funding from the European Commission’s Horizon 2020 socSMC project (socSMC-641321H2020-FETPROACT-2014) and by the European Research Council’s CDAC project (ERC-2013-ADG341196).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Freire, I.T., Puigbò, JY., Arsiwalla, X.D., Verschure, P.F.M.J. (2018). Modeling the Opponent’s Action Using Control-Based Reinforcement Learning. In: Vouloutsi , V., et al. Biomimetic and Biohybrid Systems. Living Machines 2018. Lecture Notes in Computer Science(), vol 10928. Springer, Cham. https://doi.org/10.1007/978-3-319-95972-6_19
Download citation
DOI: https://doi.org/10.1007/978-3-319-95972-6_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-95971-9
Online ISBN: 978-3-319-95972-6
eBook Packages: Computer ScienceComputer Science (R0)