article

Hand gesture guided robot-assisted surgery based on a direct augmented reality interface

Authors:

Binh P. Nguyen,

Chin-Boon Chng,

Chee-Kong ChuiAuthors Info & Claims

Computer Methods and Programs in Biomedicine, Volume 116, Issue 2

Pages 68 - 80

https://doi.org/10.1016/j.cmpb.2013.12.018

Published: 01 September 2014 Publication History

Abstract

Radiofrequency (RF) ablation is a good alternative to hepatic resection for treatment of liver tumors. However, accurate needle insertion requires precise hand-eye coordination and is also affected by the difficulty of RF needle navigation. This paper proposes a cooperative surgical robot system, guided by hand gestures and supported by an augmented reality (AR)-based surgical field, for robot-assisted percutaneous treatment. It establishes a robot-assisted natural AR guidance mechanism that incorporates the advantages of the following three aspects: AR visual guidance information, surgeon's experiences and accuracy of robotic surgery. A projector-based AR environment is directly overlaid on a patient to display preoperative and intraoperative information, while a mobile surgical robot system implements specified RF needle insertion plans. Natural hand gestures are used as an intuitive and robust method to interact with both the AR system and surgical robot. The proposed system was evaluated on a mannequin model. Experimental results demonstrated that hand gesture guidance was able to effectively guide the surgical robot, and the robot-assisted implementation was found to improve the accuracy of needle insertion. This human-robot cooperative mechanism is a promising approach for precise transcutaneous ablation therapy.

References

[1]

Aggarwal, R., Crochet, P., Dias, A., Misra, A., Ziprin, P. and Darzi, A., Development of a virtual reality training curriculum for laparoscopic cholecystectomy. British Journal of Surgery. v96. 1086-1093.

[2]

Cai, Y.Y., Chui, C.K., Ye, X.Z., Fan, Z. and Anderson, J.H., Tactile VR for handeye coordination in simulated PTCA. Computers in Biology and Medicine. v36. 167-180.

[3]

Du, Q. and Zhang, X., Research on robotic technologies for ultrasound guided radio frequency ablation surgery. In: IEEE Proceedings of the International Conference on Industrial Technology (ICIT 2008), pp. 1-6.

[4]

Ferrari, V., Megali, G., Troia, E., Pietrabissa, A. and Mosca, F., A 3-D mixed-reality system for stereoscopic visualization of medical dataset. IEEE Transactions on Biomedical Engineering. v56. 2627-2633.

[5]

Fichtinger, G., Deguet, A., Masamune, K., Balogh, E., Fischer, G.S., Mathieu, H., Taylor, R.H., Zinreich, S.J. and Fayad, L.M., Image overlay guidance for needle insertion in CT scanner. IEEE Transactions on Biomedical Engineering. v52. 1415-1424.

[6]

Gallo, L., Placitelli, A.P. and Ciampi, M., Controller-free exploration of medical image data: experiencing the Kinect. In: IEEE Proceedings of the 24th International Symposium on Computer-Based Medical Systems (CBMS), pp. 1-6.

Digital Library

[7]

Hartley, R. and Zisserman, A., Multiple View Geometry in Computer Vision. 2004. second ed. Cambridge University Press, Cambridge, UK.

[8]

Hostettler, A., Nicolau, S., Rémond, Y., Marescaux, J. and Soler, L., A real-time predictive simulation of abdominal viscera positions during quiet free breathing. Progress in Biophysics and Molecular Biology. v103. 169-184.

[9]

Gestonurse: a robotic surgical nurse for handling surgical instruments in the operating room. Journal of Robotic Surgery. v6. 53-63.

[10]

Kipshagen, T., Graw, M., Tronnier, V., Bonsanto, M. and Hofmann, U., Touch- and marker-free interaction with medical software. In: Dössel, O., Schlegel, W.C. (Eds.), World Congress on Medical Physics and Biomedical Engineering, volume 25/6 of IFMBE Proceedings, Springer. pp. 75-78.

[11]

Nguyen-Duc-Thanh, N., Lee, S. and Kim, D., Two-stage hidden markov model in gesture recognition for human robot interaction. International Journal of Advanced Robotic Systems. v9. 1-10.

[12]

Sainani, N.I., Shyn, P.B., Tatli, S., Morrison, P.R., Tuncali, K. and Silverman, S.G., PET/CT-guided radiofrequency and cryoablation: is tumor fluorine-18 fluorodeoxyglucose activity dissipated by thermal ablation?. Journal of Vascular and Interventional Radiology. v22. 354-360.

[13]

Tay, W.L., Chui, C.K. and Ong, S.H., Single camera-based remote pointing and recognition for monitor-based augmented reality surgical systems. In: ACM Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, pp. 35-38.

[14]

The Johns Hopkins University, Gesture Based Surgical Manipulation of a da Vinci Robot Using a Kinect. 2013.

[15]

Emerging robotic platforms for minimally invasive surgery. IEEE Reviews in Biomedical Engineering. v6. 111-126.

[16]

Wachs, J., Stern, H., Edan, Y., Gillam, M., Feied, C., Smith, M. and Handler, J., Gestix: a doctor-computer sterile gesture interface for dynamic environments. In: Saad, A., Dahal, K., Sarfraz, M., Roy, R. (Eds.), Soft Computing in Industrial Applications, volume 39 of Advances in Soft Computing, Springer. pp. 30-39.

[17]

Wachs, J.P., Jacob, M., Li, Y.T. and Akingba, G., Does a robotic scrub nurse improve economy of movements?. In: Holmes, D.R., Wong, K.H. (Eds.), Medical Imaging 2012: Image-Guided Procedures, Robotic Interventions, and Modeling, volume 8316 of Proceedings of SPIE,

[18]

Weinberger, K.Q. and Saul, L.K., Distance metric learning for large margin nearest neighbor classification. The Journal of Machine Learning Research. v10. 207-244.

[19]

Wen, R., Chng, C.B., Chui, C.K., Lim, K.B., Ong, S.H. and Chang, S.K.Y., Robot-assisted RF ablation with interactive planning and mixed reality guidance. In: Proceedings of the IEEE/SICE International Symposium on System Integration (SII 2012), pp. 31-36.

[20]

Wen, R., Chui, C.K., Ong, S.H., Lim, K.B. and Chang, S.K.Y., Projection-based visual guidance for robot-aided RF needle insertion. International Journal Computer Assisted Radiology Surgery. v8. 1015-1025.

[21]

Primer: cognitive motor learning for teaching surgical skill-how are surgical skills taught and assessed?. Nature Reviews Urology. v5. 47-54.

[22]

Yang, L., Wen, R., Qin, J., Chui, C.K., Lim, K.B. and Chang, S.K.Y., A robotic system for overlapping radiofrequency ablation in large tumor treatment. IEEE/ASME Transactions on Mechatronics. v15. 887-897.

[23]

Yang, X., Yu, H., Choi, Y., Lee, W., Wang, B., Yang, J., Hwang, H., Kim, J., Song, J., Cho, B. and You, H., A hybrid semi-automatic method for liver segmentation based on level-set methods using multiple seed points. Computer Methods and Programs in Biomedicine. v13. 69-76.

[24]

Zheng, X. and Vaezy, S., A targeting method based on acoustic backscatter for treatment planning in tissue ablation using focused ultrasound. IEEE Transactions on Biomedical Engineering. v57. 71-79.

Cited By

Li RHan BLi HMa LZhang XZhao ZLiao H(2024)A Comparative Evaluation of Optical See-Through Augmented Reality in Surgical GuidanceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.326000130:7(4362-4374)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3260001
Gavgiotaki DNtoa SMargetis GApostolakis KStephanidis C(2023)Gesture-based Interaction for AR Systems: A Short ReviewProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594815(284-292)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594815
Chen HFeng DHao ZDang XNiu JQiao Z(2022)Air-CSLWireless Communications & Mobile Computing10.1155/2022/58854752022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/5885475
Show More Cited By

Recommendations

A Convenient Method of Video See-Through Augmented Reality Based on Image-Guided Surgery System
ICICSE '13: Proceedings of the 2013 Seventh International Conference on Internet Computing for Engineering and Science

Augmented Reality (AR) is a technology which is widely used in medical area, especially in surgery room. In many cases, cumbersome equipments or complicated algorithm are introduced to establish the whole system, which would make it inefficient and ...
Depth cues in augmented reality for training of robot-assisted minimally invasive surgery
AcademicMindtrek '17: Proceedings of the 21st International Academic Mindtrek Conference

Training of robot-assisted minimally invasive surgery often includes supervised practice with a robotic surgical system. In this case, augmented reality can improve the communication between instructor and trainee, for example, by allowing the ...
Augmented Reality System for Dental Implant Surgery
VMR '09: Proceedings of the 3rd International Conference on Virtual and Mixed Reality: Held as Part of HCI International 2009

Recently, computer-assisted navigation systems have been developed to realize safe and precise surgery. In conventional systems, surgeons feel anxious intra-operatively because they have to watch a surgical monitor while operating instruments in the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computer Methods and Programs in Biomedicine

Computer Methods and Programs in Biomedicine Volume 116, Issue 2

September, 2014

129 pages

ISSN:0169-2607

Issue’s Table of Contents

Copyright © Elsevier Ireland Ltd © 2013.

Publisher

Elsevier North-Holland, Inc.

United States

Publication History

Published: 01 September 2014

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li RHan BLi HMa LZhang XZhao ZLiao H(2024)A Comparative Evaluation of Optical See-Through Augmented Reality in Surgical GuidanceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.326000130:7(4362-4374)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3260001
Gavgiotaki DNtoa SMargetis GApostolakis KStephanidis C(2023)Gesture-based Interaction for AR Systems: A Short ReviewProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594815(284-292)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594815
Chen HFeng DHao ZDang XNiu JQiao Z(2022)Air-CSLWireless Communications & Mobile Computing10.1155/2022/58854752022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/5885475
Suzuki RKarim AXia THedayati HMarquardt N(2022)Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic InterfacesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517719(1-33)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517719
de Camargo VBalancieri RTeixeira HGuerino GMonteiro Ida Hora Rodrigues Kde Gois Ribeiro Darin TFreire AMota M(2021)Touchless Modalities of Human-Computer Interaction in HospitalsProceedings of the XX Brazilian Symposium on Human Factors in Computing Systems10.1145/3472301.3484328(1-11)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3472301.3484328
Arevalo Arboleda SRücker FDierks TGerken JKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Assisting Manipulation and Grasping in Robot Teleoperation with Augmented Reality Visual CuesProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445398(1-14)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445398
Lim KSin SLee CChin WLin JNguyen KNguyen QNguyen BChua M(2020)Video-based Skeletal Feature Extraction for Hand Gesture RecognitionProceedings of the 4th International Conference on Machine Learning and Soft Computing10.1145/3380688.3380711(108-112)Online publication date: 17-Jan-2020
https://dl.acm.org/doi/10.1145/3380688.3380711
Arévalo Arboleda SDierks TRücker FGerken JBelpaeme TYoung JGunes HRiek L(2020)There's More than Meets the EyeCompanion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3371382.3378240(104-106)Online publication date: 23-Mar-2020
https://dl.acm.org/doi/10.1145/3371382.3378240
Chng CWong PHo NTan XChui C(2019)Towards a Cyber-Physical Systems Based Operating Room of the FutureOR 2.0 Context-Aware Operating Theaters and Machine Learning in Clinical Neuroimaging10.1007/978-3-030-32695-1_6(47-55)Online publication date: 13-Oct-2019
https://dl.acm.org/doi/10.1007/978-3-030-32695-1_6
Peña-Rios AHagras HGardner MOwusu G(2017)A fuzzy logic based system for geolocated augmented reality field service support2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ-IEEE.2017.8015477(1-6)Online publication date: 9-Jul-2017
https://dl.acm.org/doi/10.1109/FUZZ-IEEE.2017.8015477
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents