Abstract
We consider multi-robot applications, where a team of robots can ask for the intervention of a human operator to handle difficult situations. As the number of requests grows, team members will have to wait for the operator attention, hence the operator becomes a bottleneck for the system. Our aim in this context is to make the robots learn cooperative strategies to decrease the idle time of the system by modeling the operator as a shared resource. In particular, we consider a balking queuing model where robots decide whether or not to join the queue and use multi-robot learning to estimate the best cooperative policy. In more detail, we formalize the problem as Decentralized Markov Decision Process and provide a suitable state representation, so to apply an independent learners approach. We evaluate the proposed method in a robotic water monitoring simulation and empirically show that our approach can significantly improve the team performance, while being computationally tractable.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Some part of this work appears in Raeissi and Farinelli (2018). That work describes basic ideas and preliminary results, here we provide a more detailed description of the methodologies, and more extensive empirical analysis.
While this may be a significant challenge in some domains, this is not the focus of our work.
References
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002). The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 27(4), 819–840.
Bevacqua, G., Cacace, J., Finzi, A., & Lippiello, V. (2015). Mixed-initiative planning and execution for multiple drones in search and rescue missions. In Proceedings of the twenty-fifth international conference on automated planning and scheduling, ICAPS’15 (pp. 315–323). AAAI Press
Cacace, J., Caccavale, R., Finzi, A., & Lippiello, V. (2016). Attentional multimodal interface for multidrone search in the alps. In 2016 IEEE international conference on systems, man, and cybernetics (SMC) (pp. 001178–001183). https://doi.org/10.1109/SMC.2016.7844401.
Chernova, S., & Veloso, M. (2009). Interactive policy learning through confidence-based autonomy. J. Artif. Int. Res., 34(1), 1–25. http://dl.acm.org/citation.cfm?id=1622716.1622717.
Chien, S. Y., Lewis, M., Mehrotra, S., Brooks, N., & Sycara, K. P. (2012). Scheduling operator attention for multi-robot control. In IEEE/RSJ international conference on intelligent robots and systems, IROS, Vilamoura, Algarve, Portugal, October 7–12 (pp. 473–479)
Collins, J., Bilot, C., Gini, M., & Mobasher, B. (2000). Mixed-initiative decision support in agent-based automated contracting. In Proceedings of the fourth international conference on autonomous agents, AGENTS ’00 (pp. 247–254). ACM, New York, NY, USA
Dorais, G. A., Bonasso, R. P., Kortenkamp, D., Pell, B., & Schreckenghost, D. (1998). Adjustable autonomy for human-centered autonomous systems. In: on Mars in First International Conference of the Mars Society.
Farinelli, A., Raeissi, M. M., Marchi, N., Brooks, N., & Scerri, P. (2017). Interacting with team oriented plans in multi-robot systems. Autonomous Agents and Multi-Agent Systems, 31(2), 332–361.
Goldman, C. V., & Zilberstein, S. (2003). Optimizing information exchange in cooperative multi-agent systems. In Proceedings of the second international joint conference on autonomous agents and multiagent systems, AAMAS ’03 (pp. 137–144). ACM, New York, NY, USA
Gromov, B., M. Gambardella, L., & Di Caro, G. (2016). Wearable multi-modal interface for human multi–robot interaction. In IEEE international symposium on safety, security, and rescue robotics, SSRR2016 (pp. 240–245).
Gunderson, J. P., & Martin, W. N. (1999). Effects of uncertainty on variable autonomy in maintenance robots. In Workshop on autonomy control software (pp. 26–34).
Horvitz, E., Jacobs, A., & Hovel, D. (1999). Attention-sensitive alerting. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence, UAI’99 (pp. 305–313). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA . http://dl.acm.org/citation.cfm?id=2073796.2073831.
Hsieh, M. A., Cowley, A., Keller, J. F., Chaimowicz, L., Grocholsky, B., Kumar, V., et al. (2007). Adaptive teams of autonomous aerial and ground robots for situational awareness. Journal of Field Robotics, 24(11–12), 991–1014.
Kaminka, G. A., & Frenkel, I. (2005). Flexible teamwork in behavior-based robots. In Proceedings of the 20th national conference on artificial intelligence (Vol. 1, pp. 108–113), AAAI’05. AAAI Press
Lewis, M., Chien, S. Y., Mehortra, S., Chakraborty, N., & Sycara, K. (2014). Task switching and single vs. multiple alarms for supervisory control of multiple robots. In International conference on engineering psychology and cognitive ergonomics (pp. 499–510). Springer.
Naor, P. (1969). The regulation of queue size by levying tolls. Econometrica, 37(1), 15–24. http://www.jstor.org/stable/1909200.
Panait, L., & Luke, S. (2005). Cooperative multi-agent learning: The state of the art. Autonomous Agents and Multi-agent Systems, 11(3), 387–434.
Raeissi, M. M., & Farinelli, A. (2018). Learning queuing strategies in human–multi-robot interaction. In Proceedings of the 17th international conference on autonomous agents and multiagent systems, AAMAS ’18 (pp. 2207–2209). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC.
Rosenfeld, A., Agmon, N., Maksimov, O., Azaria, A. & Kraus, S. (2015). Intelligent agent supporting human–multi-robot team collaboration. In Proceedings of the 24th international conference on artificial intelligence, IJCAI’15 (pp. 1902–1908).
Rosenthal, S., & Veloso, M. (2010). Using symbiotic relationships with humans to help robots overcome limitations. In Workshop for collaborative human/ai control for interactive experiences.
Scheutz, M. & Kramer, J. (2007). Reflection and reasoning mechanisms for failure detection and recovery in a distributed robotic architecture for complex robots. In 2007 IEEE international conference on robotics and automation (pp. 3699–3704). IEEE.
Stoica, A., Theodoridis, T., Hu, H., McDonald-Maier, K., & Barrero, D. (2013). Towards human-friendly efficient control of multi-robot teams. In Proceedings of the 2013 international conference on collaboration technologies and systems, CTS 2013 (pp. 226–231).
Sutton, R. S., McAllester, D. A., Singh, S. P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems (pp. 1057–1063).
Suzanne Barber, K., Goel, A., & Martin, C. E. (2000). Dynamic adaptive autonomy in multi-agent systems. Journal of Experimental & Theoretical Artificial Intelligence, 12(2), 129–147.
Tambe, M. (1997). Towards flexible teamwork. J. Artif. Int. Res., 7(1), 83–124.
Valada, A., Velagapudi, P., Kannan, B., Tomaszewski, C., Kantor, G., & Scerri, P. (2014). Development of a Low Cost Multi-robot Autonomous Marine Surface Platform. Berlin Heidelberg: Springer.
Wang, J., & Lewis, M. (2007). Human control for cooperating robot teams. In Proceedings of the ACM/IEEE international conference on human–robot interaction, HRI ’07 (pp. 9–16). ACM.
Xuan, P., & Lesser, V. (2002). Multi-agent policies: From centralized ones to decentralized ones. In Proceedings of the first international joint conference on autonomous agents and multiagent systems: Part 3, AAMAS ’02 (pp. 1098–1105). ACM.
Zhang, Y., Narayanan, V., Chakraborti, T. & Kambhampati, S. (2015). A human factors analysis of proactive support in human–robot teaming. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3586–3593). https://doi.org/10.1109/IROS.2015.7353878.
Acknowledgements
This work is partially funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 689341. This work reflects only the authors’ view and the EASME is not responsible for any use that may be made of the information it contains.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This is one of the several papers published in Autonomous Robots comprising the Special Issue on Multi-Robot and Multi-Agent Systems.
Rights and permissions
About this article
Cite this article
Raeissi, M.M., Farinelli, A. Cooperative Queuing Policies for Effective Scheduling of Operator Intervention. Auton Robot 44, 617–626 (2020). https://doi.org/10.1007/s10514-019-09877-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10514-019-09877-w