A System for Human-Robot Teaming through End-User Programming and Shared Autonomy
Authors: 
Michael Hagenow, Emmanuel Senft, Robert Radwin, Michael Gleicher, Michael Zinn, 
Bilge MutluAuthors Info & Claims
Pages 231 - 239
Abstract
Many industrial tasks-such as sanding, installing fasteners, and wire harnessing-are difficult to automate due to task complexity and variability. We instead investigate deploying robots in an assistive role for these tasks, where the robot assumes the physical task burden and the skilled worker provides both the high-level task planning and low-level feedback necessary to effectively complete the task. In this article, we describe the development of a system for flexible human-robot teaming that combines state-of-the-art methods in end-user programming and shared autonomy and its implementation in sanding applications. We demonstrate the use of the system in two types of sanding tasks, situated in aircraft manufacturing, that highlight two potential workflows within the human-robot teaming setup. We conclude by discussing challenges and opportunities in human-robot teaming identified during the development, application, and demonstration of our system.
Supplemental Material
MP4 File
Supplemental video
- Download
- 41.99 MB
References
[1]
David A Abbink, Mark Mulder, and Erwin R Boer. 2012. Haptic shared control: smoothly shifting control authority? Cognition, Technology & Work, Vol. 14 (2012).
[2]
021)]% ajaykumar2021survey, Gopika Ajaykumar, Maureen Steele, and Chien-Ming Huang. 2021. A survey on end-user robot programming. ACM Computing Surveys (CSUR), Vol. 54, 8 (2021).
[3]
Batu Akan, Afshin Ameri, Baran Cürüklü, and Lars Asplund. 2011. Intuitive industrial robot programming through incremental multimodal language and augmented reality. In 2011 IEEE International Conference on Robotics and Automation. IEEE, 3934--3939.
[4]
Andrea Bajcsy, Dylan P Losey, Marcia K O'Malley, and Anca D Dragan. 2017. Learning robot objectives from physical human interaction. In Conference on Robot Learning. PMLR, 217--226.
[5]
Daniel Bambu^sek, Zdenve k Materna, Michal Kapinus, V'itve zslav Beran, and Pavel Smrvz. 2019. Combining interactive spatial augmented reality with head-mounted display for end-user collaborative robot programming. In 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 1--8.
[6]
Yuanzhi Cao, Zhuangying Xu, Fan Li, Wentao Zhong, Ke Huo, and Karthik Ramani. 2019. V. ra: An in-situ visual authoring system for robot-iot task planning with augmented reality. In Proceedings of the 2019 on Designing Interactive Systems Conference. 1059--1070.
[7]
Marc G Carmichael, Stefano Aldini, Richardo Khonasty, Antony Tran, Christian Reeks, Dikai Liu, Kenneth J Waldron, and Gamini Dissanayake. 2019. The ANBOT: An intelligent robotic co-worker for industrial abrasive blasting. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 8026--8033.
[8]
Marco Cognetti, Marco Aggravi, Claudio Pacchierotti, Paolo Salaris, and Paolo Robuffo Giordano. 2020. Perception-aware human-assisted navigation of mobile robots on persistent trajectories. IEEE Robotics and Automation Letters, Vol. 5, 3 (2020), 4711--4718.
[9]
Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly (1989), 319--340.
[10]
Megh Vipul Doshi, Michael Hagenow, Robert Radwin, Michael Gleicher, Bilge Mutlu, and Michael Zinn. 2023. Handheld Haptic Device with Coupled Bidirectional Input. In 2023 IEEE World Haptics Conference (WHC). IEEE, 453--459.
[11]
Anca D Dragan and Siddhartha S Srinivasa. 2013. A policy-blending formalism for shared control. The International Journal of Robotics Research, Vol. 32, 7 (2013).
[12]
Mica R Endsley and Esin O Kiris. 1995. The out-of-the-loop performance problem and level of control in automation. Human factors, Vol. 37, 2 (1995), 381--394.
[13]
P. Evrard and A. Kheddar. 2009. Homotopy-based controller for physical human-robot interaction. In RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication. 1--6.
[14]
Jared A Frank, Matthew Moorhead, and Vikram Kapila. 2016. Realizing mixed-reality environments with tablets for intuitive human-robot collaboration for object manipulation tasks. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 302--307.
[15]
Masakazu Fujii, Hiroki Murakami, and Mitsuharu Sonehara. 2016. Study on application of a human-robot collaborative system using hand-guiding in a production line. IHI Eng. Rev, Vol. 49, 1 (2016), 24--29.
[16]
Yuxiang Gao and Chien-Ming Huang. 2019. PATI: a projection-based augmented table-top interface for robot programming. In Proceedings of the 24th international conference on intelligent user interfaces. 345--355.
[17]
Martin H"agele, Klas Nilsson, J Norberto Pires, and Rainer Bischoff. 2016. Industrial robotics. Springer handbook of robotics (2016), 1385--1422.
[18]
Michael Hagenow, Emmanuel Senft, Evan Laske, Kimberly Hambuchen, Terrence Fong, Robert Radwin, Michael Gleicher, Bilge Mutlu, and Michael Zinn. 2022. Registering Articulated Objects With Human-in-the-loop Corrections. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2343--2350.
[19]
Michael Hagenow, Emmanuel Senft, Nitzan Orr, Robert Radwin, Michael Gleicher, Bilge Mutlu, Dylan P. Losey, and Michael Zinn. 2023. Coordinated Multi-Robot Shared Autonomy Based on Scheduling and Demonstrations. IEEE Robotics and Automation Letters, Vol. 8, 12 (2023), 8335--8342.
[20]
Michael Hagenow, Emmanuel Senft, Robert Radwin, Michael Gleicher, Bilge Mutlu, and Michael Zinn. 2021a. Corrective shared autonomy for addressing task variability. IEEE robotics and automation letters, Vol. 6, 2 (2021), 3720--3727.
[21]
Michael Hagenow, Emmanuel Senft, Robert Radwin, Michael Gleicher, Bilge Mutlu, and Michael Zinn. 2021b. Informing real-time corrections in corrective shared autonomy through expert demonstrations. IEEE Robotics and Automation Letters, Vol. 6, 4 (2021), 6442--6449.
[22]
Shervin Javdani, Henny Admoni, Stefania Pellegrinelli, Siddhartha S. Srinivasa, and J. Andrew Bagnell. 2018. Shared autonomy via hindsight optimization for teleoperation and teaming. The International Journal of Robotics Research, Vol. 37, 7 (2018), 717--742. https://doi.org/10.1177/0278364918776060
[23]
Ariyan M Kabir, Joshua D Langsfeld, Krishnanand N Kaipa, and Satyandra K Gupta. 2018. Identifying optimal trajectory parameters in robotic finishing operations using minimum number of physical experiments. Integrated Computer-Aided Engineering, Vol. 25, 2 (2018), 111--135.
[24]
Wansoo Kim, Pietro Balatti, Edoardo Lamon, and Arash Ajoudani. 2020. MOCA-MAN: A MObile and reconfigurable Collaborative Robot Assistant for conjoined huMAN-robot actions. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 10191--10197.
[25]
Mathieu Labbé and Francc ois Michaud. 2019. RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. Journal of Field Robotics, Vol. 36, 2 (2019), 416--446.
[26]
Jens Lambrecht and Jörg Krüger. 2012. Spatial programming for industrial robots based on gestures and augmented reality. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 466--472.
[27]
Y. Li, K. P. Tee, W. L. Chan, R. Yan, Y. Chua, and D. K. Limbu. 2015. Continuous Role Adaptation for Human--Robot Shared Control. IEEE Transactions on Robotics, Vol. 31, 3 (2015), 672--681.
[28]
Dylan P Losey and Marcia K O'Malley. 2017. Trajectory deformations from physical human--robot interaction. IEEE Transactions on Robotics, Vol. 34, 1 (2017).
[29]
Matthew B Luebbers, Connor Brooks, Carl L Mueller, Daniel Szafir, and Bradley Hayes. 2021. Arc-lfd: Using augmented reality for interactive long-term robot skill maintenance via constrained learning from demonstration. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3794--3800.
[30]
Bruno Maric, Alan Mutka, and Matko Orsag. 2020. Collaborative human-robot framework for delicate sanding of complex shape surfaces. IEEE Robotics and Automation Letters, Vol. 5, 2 (2020), 2848--2855.
[31]
Carlo Masone, Paolo Robuffo Giordano, Heinrich H Bülthoff, and Antonio Franchi. 2014. Semi-autonomous trajectory generation for mobile robots with integral haptic shared control. In 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 6468--6475.
[32]
Carlos Mateo, Alberto Brunete, Ernesto Gambao, and Miguel Hernando. 2014. Hammer: An Android based application for end-user industrial robot programming. In 2014 IEEE/ASME 10th International Conference on Mechatronic and Embedded Systems and Applications (MESA). IEEE, 1--6.
[33]
J. R. Medina, M. Lawitzky, A. Molin, and S. Hirche. 2013. Dynamic strategy selection for physical robotic assistance in partially known tasks. In 2013 IEEE International Conference on Robotics and Automation. 1180--1186.
[34]
Joseph E Michaelis, Amanda Siebert-Evenstone, David Williamson Shaffer, and Bilge Mutlu. 2020. Collaborative or simply uncaged? understanding human-cobot interactions in automation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--12.
[35]
Lauren Milliken and Geoffrey A Hollinger. 2017. Modeling user expertise for choosing levels of shared autonomy. In 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2285--2291.
[36]
Bojan Nemec, Leon vZ lajpah, Sebastjan vS lajpa, Jovz ica Pivs kur, and Alevs Ude. 2018. An efficient pbd framework for fast deployment of bi-manual assembly tasks. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids). IEEE, 166--173.
[37]
Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. 2000. A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, Vol. 30, 3 (2000).
[38]
Chris Paxton, Andrew Hundt, Felix Jonathan, Kelleher Guerin, and Gregory D Hager. 2017. CoSTAR: Instructing collaborative robots with behavior trees and vision. In 2017 IEEE international conference on robotics and automation (ICRA). IEEE, 564--571.
[39]
Margaret Pearce, Bilge Mutlu, Julie Shah, and Robert Radwin. 2018. Optimizing makespan and ergonomics in integrating collaborative robots into manufacturing processes. IEEE transactions on automation science and engineering, Vol. 15, 4 (2018), 1772--1784.
[40]
Siddharth Reddy, Anca D Dragan, and Sergey Levine. 2018. Shared autonomy via deep reinforcement learning. arXiv preprint arXiv:1802.01744 (2018).
[41]
Branko Sarh, James Buttrick, Clayton Munk, and Richard Bossi. 2009. Aircraft manufacturing and assembly. Springer handbook of automation (2009), 893--910.
[42]
Mario Selvaggio, Marco Cognetti, Stefanos Nikolaidis, Serena Ivaldi, and Bruno Siciliano. 2021. Autonomy in physical human-robot interaction: A brief survey. IEEE Robotics and Automation Letters, Vol. 6, 4 (2021), 7989--7996.
[43]
Emmanuel Senft, Michael Hagenow, Robert Radwin, Michael Zinn, Michael Gleicher, and Bilge Mutlu. 2021a. Situated live programming for human-robot collaboration. In The 34th Annual ACM Symposium on User Interface Software and Technology. 613--625.
[44]
Emmanuel Senft, Michael Hagenow, Kevin Welsh, Robert Radwin, Michael Zinn, Michael Gleicher, and Bilge Mutlu. 2021b. Task-level authoring for remote robot teleoperation. Frontiers in Robotics and AI, Vol. 8 (2021), 707149.
[45]
Markku Suomalainen, Yiannis Karayiannidis, and Ville Kyrki. 2022. A survey of robot manipulation in contact. Robotics and Autonomous Systems, Vol. 156 (2022).
[46]
Ronald Wilcox, Stefanos Nikolaidis, and Julie Shah. 2013. Optimization of temporal dynamics for adaptive human-robot interaction in assembly manufacturing. Robotics, Vol. 8, 441 (2013), 10--15. io
Index Terms
- A System for Human-Robot Teaming through End-User Programming and Shared Autonomy
Recommendations
A Theatrical Mobile-Dexterous Robot Directed through Shared Autonomy
HRI '17: Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot InteractionWe present the deployment of a 16-DoF dual-arm mobile manipulator as an on-stage actor in the MIT2016 Pageant, a 60 minute live play performed for the centennial celebration of the Massachusetts Institute of Technology campus move from Boston to ...
Balanced information gathering and goal-oriented actions in shared autonomy
HRI '19: Proceedings of the 14th ACM/IEEE International Conference on Human-Robot InteractionRobotic teleoperation can be a complex task due to factors such as high degree-of-freedom manipulators, operator inexperience, and limited operator situational awareness. To reduce teleoperation complexity, researchers have developed the shared autonomy ...
Human-Robot Mutual Adaptation in Shared Autonomy
HRI '17: Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot InteractionShared autonomy integrates user input with robot autonomy in order to control a robot and help the user to complete a task. Our work aims to improve the performance of such a human-robot team: the robot tries to guide the human towards an effective ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2024
982 pages
ISBN:9798400703225
DOI:10.1145/3610977
- General Chairs:
- Dan Grollman,
- Elizabeth Broadbent,
- Program Chairs:
- Wendy Ju,
- Harold Soh,
- Tom Williams
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 March 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
HRI '24: ACM/IEEE International Conference on Human-Robot Interaction
March 11 - 15, 2024
CO, Boulder, USA
Acceptance Rates
Overall Acceptance Rate 268 of 1,124 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 434Total Downloads
- Downloads (Last 12 months)434
- Downloads (Last 6 weeks)64
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in