Abstract
We present a robot-to-human object handover algorithm and implement it on a 7-DOF arm equipped with a 3-finger mechanical hand. The system performs a fully autonomous and robust object handover to a human receiver in real-time. Our algorithm relies on two complementary sensor modalities: joint torque sensors on the arm and an eye-in-hand RGB-D camera for sensor feedback. Our approach is entirely implicit, i.e., there is no explicit communication between the robot and the human receiver. Information obtained via the aforementioned sensor modalities are used as inputs to their related deep neural networks. While the torque sensor network detects the human receiver’s “intention” such as: pull, hold, or bump, the vision sensor network detects if the receiver’s fingers have wrapped around the object. Networks’ outputs are then fused, based on which a decision is made to either release the object or not. Despite substantive challenges in sensor feedback synchronization, object and human hand detection, our system achieves robust robot-to-human handover with 98% accuracy in our preliminary real experiments using human receivers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
In fact, that work used a specialized simple optical sensor precisely because they mention the unacceptable amount of computation time that would be taken for processing RGB images, a problem that we solve via the use of SSD network.
References
Ortenzi, V., Cosgun, A., Pardi, T., Chan, W.P., Croft, E., Kulić, D.: Object handovers: a review for robotics. IEEE Trans. Robot. 37, 1855–1873 (2021)
Chan, W.P., Parker, C.A., Van der Loos, H.M., Croft, E.A.: Grip forces and load forces in handovers: implications for designing human-robot handover controllers. In: Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction, pp. 9–16 (2012)
Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2
Sileo, M., Nigro, M., Bloisi, D.D., Pierri, F.: Vision based robot-to-robot object handover. In: 2021 20th International Conference on Advanced Robotics (ICAR), pp. 664–669. IEEE (2021)
Eguiluz, A.G., Rañó, I., Coleman, S.A., McGinnity, T.M.: Reliable object handover through tactile force sensing and effort control in the shadow robot hand. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 372–377. IEEE (2017)
Parastegari, S., Noohi, E., Abbasi, B., Žefran, M.: A fail-safe object handover controller. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2003–2008. IEEE (2016)
Parastegari, S., Noohi, E., Abbasi, B., Žefran, M.: Failure recovery in robot-human object handover. IEEE Trans. Rob. 34(3), 660–673 (2018)
Shi, C., Shiomi, M., Smith, C., Kanda, T., Ishiguro, H.: A model of distributional handing interaction for a mobile robot. In: Robotics: Science and Systems, pp. 24–28 (2013)
Choi, Y.S., Chen, T., Jain, A., Anderson, C., Glass, J.D., Kemp, C.C.: Hand it over or set it down: a user study of object delivery with an assistive mobile manipulator. In: RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interactive Communication, pp. 736–743. IEEE (2009)
Grigore, E.C., Eder, K., Pipe, A.G., Melhuish, C., Leonards, U.: Joint action understanding improves robot-to-human object handover. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4622–4629. IEEE (2013)
Koene, A., Endo, S., Remazeilles, A., Prada, M., Wing, A.M.: Experimental testing of the coglaboration prototype system for fluent human-robot object handover interactions. In: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 249–254. IEEE (2014)
Prada, M., Remazeilles, A., Koene, A., Endo, S.: Implementation and experimental validation of dynamic movement primitives for object handover. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2146–2153. IEEE (2014)
Chan, W.P., Parker, C.A., Van der Loos, H.M., Croft, E.A.: A human-inspired object handover controller. Int. J. Robot. Res. 32(8), 971–983 (2013)
Hasson, Y., et al.: Learning joint reconstruction of hands and manipulated objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11 807–11 816 (2019)
Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M., Brox, T.: FreiHAND: a dataset for markerless capture of hand pose and shape from single RGB images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 813–822 (2019)
Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: HOnnotate: a method for 3D annotation of hand and object poses. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3196–3206 (2020)
Ijspeert, A.J., Nakanishi, J., Schaal, S.: Movement imitation with nonlinear dynamical systems in humanoid robots. In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 2, pp. 1398–1403. IEEE (2002)
Schaal, S., Peters, J., Nakanishi, J., Ijspeert, A.: Learning movement primitives. In: Dario, P., Chatila, R. (eds.) Robotics Research. The Eleventh International Symposium. STAR, vol. 15, pp. 561–572. Springer, Heidelberg (2005). https://doi.org/10.1007/11008941_60
Yang, W., Paxton, C., Cakmak, M., Fox, D.: Human grasp classification for reactive human-to-robot handovers. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11 123–11 130. IEEE (2020)
Li, Z., Hauser, K.: Predicting object transfer position and timing in human-robot handover tasks. Sci. Syst. 38 (2015)
Luo, R., Hayne, R., Berenson, D.: Unsupervised early prediction of human reaching for human-robot collaboration in shared workspaces. Auton. Robot. 42(3), 631–648 (2018)
Controzzi, M., Singh, H., Cini, F., Cecchini, T., Wing, A., Cipriani, C.: Humans adjust their grip force when passing an object according to the observed speed of the partner’s reaching out movement. Exp. Brain Res. 236(12), 3363–3377 (2018)
Strabala, K., et al.: Toward seamless human-robot handovers. J. Hum.-Rob. Interact. 2(1), 112–132 (2013)
Davari, M.-J., Hegedus, M., Gupta, K., Mehrandezh, M.: Identifying multiple interaction events from tactile data during robot-human object transfer. In: 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 1–6. IEEE (2019)
Moradi, B.: (2021). https://github.com/Bmoradi93/SSD-Object-Detection
Moradi, B.: (2021). https://github.com/Bmoradi93/SSD-Object-Detection-ROS2
Szegedy, C., Reed, S., Erhan, D., Anguelov, D., Ioffe, D.: Scalable, high-quality object detection. arXiv preprint arXiv:1412.1441 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mohandes, M., Moradi, B., Gupta, K., Mehrandezh, M. (2023). Robot to Human Object Handover Using Vision and Joint Torque Sensor Modalities. In: Jo, J., et al. Robot Intelligence Technology and Applications 7. RiTA 2022. Lecture Notes in Networks and Systems, vol 642. Springer, Cham. https://doi.org/10.1007/978-3-031-26889-2_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-26889-2_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-26888-5
Online ISBN: 978-3-031-26889-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)