Abstract
Cooperative transportation is one of the essential tasks for multi-robot systems to imitate the decentralized systems of social insects. However, in a situation involving an obstacle on the pathway, multiple robots need to realize transportation and obstacle removal simultaneously. To address this multitasking problem, we first introduce a learning scenario and train robots’ decentralized policies via multi-agent reinforcement learning. Next, we propose two virtual experiments with blindfold teams and homogeneous teams to analyze the individual behaviors of the trained robots. The results showed that three robots with different policies performed two tasks simultaneously as a team. One robot’s policy tended to perform obstacle removal, and the other robots’ policies tended to perform cooperative transportation. Further, the first robot’s policy had the potential to perform two tasks simultaneously depending on the situation. Finally, we demonstrated the trained policies with three ground robots to show the feasibility of the system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Beshers, S.N., Fewell, J.H.: Models of division of labor in social insects. Ann. Rev. Entomol. 46(1), 413–440 (2001)
Robinson, G. E., Page, R. E.: Genetic basis for division of labor in an insect society. Genet. Soc. Evol. 61–80 (1989)
Nouyan, S., Gross, R., Bonani, M., Mondada, F., Dorigo, M.: Teamwork in self-organized robot colonies. IEEE Trans. Evol. Comput. 13, 695–711 (2009)
Brutschy, A., Pini, G., Pinciroli, C., Birattari, M., Dorigo, M.: Self-organized task allocation to sequentially interdependent tasks in swarm robotics. Auton. Agent. Multi-Agent Syst. 28(1), 101–125 (2012). https://doi.org/10.1007/s10458-012-9212-y
Garattoni, L., Birattari, M.: Autonomous task sequencing in a robot swarm. Sci. Robot. 3 (2018)
Bonabeau, E., Sobkowski, A., Theraulaz, G., Deneubourg, J.L.: Adaptive task allocation inspired by a model of division of labor in social insects. In: BCEC, pp. 36–45 (1997)
Ferrante, E., Turgut, A.E., Duéñez-Guzmán, E., Dorigo, M., Wenseleers, T.: Evolution of self-organized task specialization in robot swarms. PLoS Comput. Biol. 11(8), e1004273 (2015)
Tuci, E., Alkilabi, M.H., Akanyeti, O.: Cooperative object transport in multi-robot systems: a review of the state-of-the-art. Front. Robot. AI 5, 59 (2018)
Kosuge, K., Oosumi, T.: Decentralized control of multiple robots handling an object. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 1996, vol. 1, pp. 318–323. IEEE (1996)
Wang, Z., Schwager, M.: Kinematic multi-robot manipulation with no communication using force feedback. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 427-432. IEEE, May 2016
Wang, Z., Yang, G., Su, X., Schwager, M.: OuijaBots: omnidirectional robots for cooperative object transport with rotation control using no communication. In: Groß, R., et al. (eds.) Distributed Autonomous Robotic Systems. SPAR, vol. 6, pp. 117–131. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73008-0_9
Culbertson, P., Schwager, M.: Decentralized adaptive control for collaborative manipulation. In: 2018 IEEE International Conference on Robotics and Automation, pp. 278–285. IEEE (2018)
Farivarnejad, H., Wilson, S., Berman, S.: Decentralized sliding mode control for autonomous collective transport by multi-robot systems. In: 2016 IEEE 55th Conference on Decision and Control, pp. 1826–1833. IEEE (2016)
Chen, J., Gauci, M., Li, W., Kolling, A., Gross, R.: Occlusion-based cooperative transport with a swarm of miniature mobile robots. IEEE Trans. Rob. 31(2), 307–321 (2015)
Gross, R., Dorigo, M.: Towards group transport by swarms of robots. Int. J. Bio-Inspir. Comput. 1(1–2), 1–13 (2009)
Alkilabi, M.H.M., Narayan, A., Tuci, E.: Cooperative object transport with a swarm of e-puck robots: robustness and scalability of evolved collective strategies. Swarm Intell. 11(3–4), 185–209 (2017)
Rahimi, M., Gibb, S., Shen, Y., La, H.M.: A comparison of various approaches to reinforcement learning algorithms for multi-robot box pushing. In: Fujita, H., Nguyen, D.C., Vu, N.P., Banh, T.L., Puta, H.H. (eds.) ICERA 2018. LNNS, vol. 63, pp. 16–30. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04792-4_6
Zhang, L., Sun, Y., Barth, A., Ma, O.: Decentralized control of multi-robot system in cooperative object transportation using deep reinforcement learning. IEEE Access 8, 184109–184119 (2020)
Wang, Y., de Silva, C.W.: Cooperative transportation by multiple robots with machine learning. In: 2006 IEEE International Conference on Evolutionary Computation, pp. 3050–3056. IEEE (2006)
Gupta, J.K., Egorov, M., Kochenderfer, M.: Cooperative multi-agent control using deep reinforcement learning. In: Sukthankar, G., Rodriguez-Aguilar, J.A. (eds.) AAMAS 2017. LNCS (LNAI), vol. 10642, pp. 66–83. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71682-4_5
Hernandez-Leal, P., Kartal, B., Taylor, M.E.: Agent modeling as auxiliary task for deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 15, no. 1, pp. 31–37 (2019)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. Adv. Neural. Inf. Process. Syst. 30, 6379–6390 (2017)
Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Process. Syst. 1057–1063 (2000)
Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48 (2009)
Cobbe, K., Klimov, O., Hesse, C., Kim, T., Schulman, J.: Quantifying generalization in reinforcement learning. In: International Conference on Machine Learning, pp. 1282–1289. PMLR (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Niwa, T., Shibata, K., Jimbo, T. (2022). Multi-agent Reinforcement Learning and Individuality Analysis for Cooperative Transportation with Obstacle Removal. In: Matsuno, F., Azuma, Si., Yamamoto, M. (eds) Distributed Autonomous Robotic Systems. DARS 2021. Springer Proceedings in Advanced Robotics, vol 22. Springer, Cham. https://doi.org/10.1007/978-3-030-92790-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-92790-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92789-9
Online ISBN: 978-3-030-92790-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)