ARAS: LLM-Supported Augmented Reality Assistance System for Pancreatic Surgery
Authors: 
Hamraz Javaheri, Omid Ghamarnejad, Paul Lukowicz, Gregor Alexander Stavrou, Jakob KarolusAuthors Info & Claims
Pages 176 - 180
Abstract
The integration of Augmented Reality (AR) technology into surgical procedures offers significant potential to enhance clinical outcomes. Despite numerous lab-proven prototypes, deploying these systems in actual clinical settings demands specialized design and rigorous clinical evaluation to meet the high standards of complex medical fields. Our research highlights the intricate requirements emerging from clinical environments, particularly operating theaters. To address these challenges, we introduce ARAS, an operational AR assistance system for open pancreatic surgery. Through a user-centric design methodology, ARAS was iteratively developed and refined, ensuring its practical applicability and effectiveness in real-world surgical settings during clinical trials. ARAS encompasses two different modes designed for preoperative and intraoperative sessions. Preoperative mode enables surgeons to visualize patient data and perform resection on the patient's 3D reconstructed vessels and tumors, helping surgeons during surgical planning. Intraoperative mode allows in-situ and precise visualization of these reconstructed 3D models during the surgical procedure and supports surgeons in decision-making. ARAS also includes a novel interaction design for surgical AR systems. It uses LLMs to enable context-aware, intuitive, and natural communication between surgeons and the AR system. This demo showcases the capabilities of ARAS, emphasizing its potential to transform surgical practices through advanced AR and LLM-supported interactions.
References
[1]
[n. d.]. Microsoft HoloLens 2. https://www.microsoft.com/en-us/hololens. Ac- cessed on September 14, 2023.
[2]
[n. d.]. Unity 3D Engine. https://unity.com/. Accessed on September 13, 2023.
[3]
Mareen Allgaier, Vuthea Chheang, Patrick Saalfeld, Vikram Apilla, Tobias Huber, Florentine Huettl, Belal Neyazi, I Erol Sandalcioglu, Christian Hansen, Bernhard Preim, et al . 2022. A comparison of input devices for precise interaction tasks in VR-based surgical planning and training. Computers in Biology and Medicine 145 (2022), 105429.
[4]
Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. 2023. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. Journal of medical systems 47, 1 (2023), 33.
[5]
Enylton Machado Coelho, Blair MacIntyre, and Simon J Julier. 2005. Supporting interaction in augmented reality in the presence of uncertain spatial knowledge. In Proceedings of the 18th annual ACM symposium on User interface software and technology. 111--114.
[6]
Fabrizio Cutolo, Benish Fida, Nadia Cattari, and Vincenzo Ferrari. 2019. Software framework for customized augmented reality headsets in medicine. IEEE Access 8 (2019), 706--720.
[7]
Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. 2023. Towards next-generation intelligent assistants leveraging llm techniques. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5792--5793.
[8]
PJ ? Eddie? Edwards, Manish Chand, Manuel Birlo, and Danail Stoyanov. 2021. The challenge of augmented reality in surgery. Digital Surgery (2021), 121--135.
[9]
Benish Fida, Fabrizio Cutolo, Gregorio di Franco, Mauro Ferrari, and Vincenzo Ferrari. 2018. Augmented reality in open surgery. Updates in surgery 70, 3 (2018), 389--400.
[10]
Yahya Ghazwani and Shamus Smith. 2020. Interaction in augmented reality: Challenges to enhance user experience. In Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations. 39--44.
[11]
Benjamin Hatscher and Christian Hansen. 2018. Hand, foot or voice: Alternative input modalities for touchless interaction in the medical domain. In Proceedings of the 20th ACM international conference on multimodal interaction. 145--153.
[12]
Julia Hertel, Sukran Karaosmanoglu, Susanne Schmidt, Julia Bräker, Martin Sem- mann, and Frank Steinicke. 2021. A taxonomy of interaction techniques for immersive augmented reality based on an iterative literature review. In 2021 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, 431--440.
[13]
Matthew B Hoy. 2018. Alexa, Siri, Cortana, and more: an introduction to voice assistants. Medical reference services quarterly 37, 1 (2018), 81--88.
[14]
Ajune Wanis Ismail and Mohd Shahrizal Sunar. 2015. Multimodal fusion: gesture and speech input in augmented reality environment. In Computational Intel- ligence in Information Systems: Proceedings of the Fourth INNS Symposia Series on Computational Intelligence in Information Systems (INNS-CIIS 2014). Springer, 245--254.
[15]
Vladimir M Ivanov, Anton M Krivtsov, Sergey V Strelkov, Anton Yu Smirnov, Ro- man Yu Shipov, Vladimir G Grebenkov, Valery N Rumyantsev, Igor S Gheleznyak, Dmitry A Surov, Michail S Korzhuk, et al. 2022. Practical application of aug- mented/Mixed reality technologies in surgery of abdominal cancer patients. Journal of Imaging 8, 7 (2022), 183.
[16]
Hamraz Javaheri, Omid Ghamarnejad, Ragnar Bade, Paul Lukowicz, Jakob Karo- lus, and Gregor Alexander Stavrou. 2024. Beyond the visible: preliminary evalu- ation of the first wearable augmented reality assistance system for pancreatic surgery. International Journal of Computer Assisted Radiology and Surgery (2024), 1--13.
[17]
Chaowanan Khundam, Varunyu Vorachart, Patibut Preeyawongsakul, Witthaya Hosap, and Frédéric Noël. 2021. A comparative study of interaction time and usability of using controllers and hand tracking in virtual reality training. In Informatics, Vol. 8. MDPI, 60.
[18]
Yubin Kim, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. 2024. Health-llm: Large language models for health prediction via wearable sensor data. arXiv preprint arXiv:2401.06866 (2024).
[19]
Minkyung Lee, Mark Billinghurst, Woonhyuk Baek, Richard Green, and Woon- tack Woo. 2013. A usability study of multimodal input in an augmented reality environment. Virtual Reality 17 (2013), 293--305.
[20]
Brenna Li, Amy Wang, Patricia Strachan, Julie Anne Séguin, Sami Lachgar, Karyn C Schroeder, Mathias S Fleck, Renee Wong, Alan Karthikesalingam, Vivek Natarajan, Yossi Matias, Greg S Corrado, Dale Webster, Yun Liu, Naama Hammel, Rory Sayres, Christopher Semturs, and Mike Schaekermann. 2024. Conver- sational AI in health: Design considerations from a Wizard-of-Oz dermatol- ogy case study with users, clinicians and a medical LLM. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI EA '24). Association for Computing Machinery, New York, NY, USA, Article 88, 10 pages. https://doi.org/10.1145/3613905.3651891
[21]
Kai Liu, Yuan Gao, Ahmed Abdelrehem, Lei Zhang, Xi Chen, Le Xie, and Xudong Wang. 2021. Augmented reality navigation method for recontouring surgery of craniofacial fibrous dysplasia. Scientific Reports 11, 1 (2021), 10043.
[22]
Abel J Lungu, Wout Swinkels, Luc Claesen, Puxun Tu, Jan Egger, and Xiaojun Chen. 2021. A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery. Expert review of medical devices 18, 1 (2021), 47--62.
[23]
Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien- Ming Huang. 2023. LLM-Powered Conversational Voice Assistants: Interac- tion Patterns, Opportunities, Challenges, and Design Guidelines. arXiv preprint arXiv:2309.13879 (2023).
[24]
Helena M Mentis, Kenton O'Hara, Gerardo Gonzalez, Abigail Sellen, Robert Corish, Antonio Criminisi, Rikin Trivedi, and Pierre Theodore. 2015. Voice or gesture in the operating room. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. 773--780.
[25]
SS Muhammad Nizam, Rimaniza Zainal Abidin, Nurhazarifah Che Hashim, Meng Chun Lam, Haslina Arshad, and NAA Majid. 2018. A review of mul- timodal interaction technique in augmented reality environment. Int. J. Adv. Sci. Eng. Inf. Technol 8, 4--2 (2018), 1460.
[26]
Philip Pratt, Matthew Ives, Graham Lawton, Jonathan Simmons, Nasko Radev, Liana Spyropoulou, and Dimitri Amiras. 2018. Through the HoloLens? looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels. European radiology experimental 2 (2018), 1--7.
[27]
Long Qian, Jie Ying Wu, Simon P DiMaio, Nassir Navab, and Peter Kazanzides. 2019. A review of augmented reality in robotic-assisted surgery. IEEE Transactions on Medical Robotics and Bionics 2, 1 (2019), 1--16.
[28]
Yu Saito, Maki Sugimoto, Satoru Imura, Yuji Morine, Tetsuya Ikemoto, Shuichi Iwahashi, Shinichiro Yamada, and Mitsuo Shimada. 2020. Intraoperative 3D hologram support with mixed reality techniques in liver surgery. Annals of surgery 271, 1 (2020), e4--e7.
[29]
Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, Vol. 11. MDPI, 887.
[30]
Claudia Scherl, Johanna Stratemeier, Nicole Rotter, Jürgen Hesser, Stefan O Schönberg, Jérôme J Servais, David Männle, and Anne Lammert. 2021. Augmented reality with HoloLens® in parotid tumor surgery: a prospective feasibility study. ORL 83, 6 (2021), 439--448.
[31]
George Terzopoulos and Maya Satratzemi. 2020. Voice assistants and smart speakers in everyday life and in education. Informatics in Education 19, 3 (2020), 473--490.
[32]
Petr Vávra, Jan Roman, Pavel Zon?a, Peter Ihnát, Martin N?mec, Jayant Kumar, Nagy Habib, Ahmed El-Gendi, et al. 2017. Recent development of augmented reality in surgery: a review. Journal of healthcare engineering 2017 (2017).
[33]
Juan Pablo Wachs, Mathias Kölsch, Helman Stern, and Yael Edan. 2011. Vision- based hand-gesture applications. Commun. ACM 54, 2 (2011), 60--71.
[34]
Deutsche Welle. 2024. Artificial Intelligence Saving Lives in the Operat- ing Room. https://www.dw.com/en/artificial-intelligence-saving-lives-in-the- operating-room/video-68125878. Accessed: 2024-07--18.
[35]
Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K Dey, and Dakuo Wang. 2024. Mental-llm: Leveraging large language models for mental health prediction via online text data. Proceed- ings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 1 (2024), 1--32.
[36]
Zhen-yu Zhang, Wen-chao Duan, Ruo-kun Chen, Feng-jiang Zhang, Bin Yu, Yun-bo Zhan, Ke Li, Hai-biao Zhao, Tao Sun, Yu-chen Ji, et al . 2019. Preliminary application of mixed reality in neurosurgery: Development and evaluation of a new intraoperative procedure. Journal of Clinical Neuroscience 67 (2019), 234--238.
Index Terms
- ARAS: LLM-Supported Augmented Reality Assistance System for Pancreatic Surgery
Recommendations
Augmented Reality System for Dental Implant Surgery
VMR '09: Proceedings of the 3rd International Conference on Virtual and Mixed Reality: Held as Part of HCI International 2009Recently, computer-assisted navigation systems have been developed to realize safe and precise surgery. In conventional systems, surgeons feel anxious intra-operatively because they have to watch a surgical monitor while operating instruments in the ...
Augmented reality navigation for minimally invasive knee surgery using enhanced arthroscopy
Highlights- Α novel in-situ AR navigation based on the enhanced arthroscopic information is developed to solve the existed problems of in-situ AR navigation.
Abstract PurposeDuring the minimally invasive knee surgery, surgeons insert surgical instruments and arthroscopy through small incisions, and implement treatment assisted by 2D arthroscopic images. However, this 2D arthroscopic ...
An augmented reality system for surgical pre-operative planning
BEBI'08: Proceedings of the 1st WSEAS international conference on Biomedical electronics and biomedical informaticsAugmented Reality technology can be used in minimally invasive surgery to increase the physician's view by showing information gathered from patient medical images. The developed AR system can improve the performances of a surgeon involved in a ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
1032 pages
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 October 2024
Check for updates
Author Tags
Qualifiers
- Demonstration
Funding Sources
Conference
UbiComp '24: The 2024 ACM International Joint Conference on Pervasive and Ubiquitous Computing
October 5 - 9, 2024
Melbourne VIC, Australia
Acceptance Rates
Overall Acceptance Rate 764 of 2,912 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 101Total Downloads
- Downloads (Last 12 months)101
- Downloads (Last 6 weeks)41
Reflects downloads up to 14 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in