Abstract
Intelligent systems face an increasingly complex array of challenges in the rapidly evolving landscape of human-device interactions. To address these challenges, there arises a pressing need for the development of refined methods of information transfer, capable of capturing subtle nuances such as body language and tonal subtleties. The system presented in this paper is designed to highlight a novel method of refined human-device interactions and its potential impact on the medical domain. Leveraging the Oculus SDK in the Unity Game Engine alongside the Oculus VR headset, our system allows users to interact with virtual objects naturally, mirroring real-world hand movements while simultaneously promoting upper limb rehabilitation through engaging gameplay scenarios. The integration of haptic feedback enriches the immersive experience, enabling users to not only visualize but also feel their virtual interactions. The system accommodates varying levels of mobility, adapting its complexity to individual progress. Furthermore, our VR rehabilitation system has garnered positive outcomes from user assessments, demonstrating the system’s effectiveness and user satisfaction. While limited in addressing severe upper limb impairments, the system’s flexibility allows for modular improvements and broader clinical integration.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ernst, E., Merola, R., Samaan, D.: Economics of artificial intelligence: implications for the future of work. IZA J. Labor Policy 9(1) (2019)
Damiano, L., Dumouchel, P., Lehmann, H.: Towards human-robot affective co-evolution overcoming oppositions in constructing emotions and empathy. Int. J. Soc. Robot. 7, 7–18 (2015)
Amin, M.M., Cambria, E., Schuller, B.W.: Will affective computing emerge from foundation models and general AI? a first evaluation on ChatGPT. arXiv preprint arXiv:2303.03186 (2023)
Chenna, S.: Augmented reality and AI: enhancing human-computer interaction in the metaverse. Available at SSRN 4324629 (2023)
Bonarini, A.: Communication in human-robot interaction. Curr. Robot. Rep. 1, 279–285 (2020)
Xu, W., Dainoff, M.J., Ge, L., Gao, Z.: Transitioning to human interaction with AI systems: new challenges and opportunities for HCI professionals to enable human-centered AI. Int. J. Hum.-Comput. Interact. 39(3), 494–518 (2023)
Urakami, J., Seaborn, K.: Nonverbal cues in human-robot interaction: a communication studies perspective. ACM Trans. Hum.-Robot Interact. 12(2), 1–21 (2023)
Ham, J., Bokhorst, R., Cuijpers, R., van der Pol, D., Cabibihan, J.-J.: Making robots persuasive: the influence of combining persuasive strategies (gazing and gestures) by a storytelling robot on its persuasive power. In: Mutlu, B., Bartneck, C., Ham, J., Evers, V., Kanda, T. (eds.) ICSR 2011. LNCS (LNAI), vol. 7072, pp. 71–83. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25504-5_8
He, J., Baxter, S.L., Xu, J., Xu, J., Zhou, X., Zhang, K.: The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25(1), 30–36 (2019)
Sahija, D.: Critical review of mixed reality integration with medical devices for patientcare. 2022, 10 (2022)
Snoswell, A.J., Snoswell, C.L.: Immersive virtual reality in health care: systematic review of technology and disease states. JMIR Biomed. Eng. 4(1), e15025 (2019)
Davenport, T., Kalakota, R.: The potential for artificial intelligence in healthcare. Future Healthc. J. 6(2), 94 (2019)
Kyrarini, M., et al.: A survey of robots in healthcare. Technologies 9(1), 8 (2021)
Cabibihan, J.J., Javed, H., Ang, M., Aljunied, S.M.: Why robots? a survey on the roles and benefits of social robots in the therapy of children with autism. Int. J. Soc. Robot. 5, 593–618 (2013)
Mantovani, F., Castelnuovo, G., Gaggioli, A., Riva, G.: Virtual reality training for health-care professionals. Cyberpsychol. Behav. 6(4), 389–395 (2003)
Wang, M., Pan, C., Ray, P.K.: Technology entrepreneurship in developing countries: role of telepresence robots in healthcare. IEEE Eng. Manage. Rev. 49(1), 20–26 (2021)
Dang, B.K., O’Leary-Kelley, C., Palicte, J.S., Badheka, S., Vuppalapati, C.: Comparing virtual reality telepresence and traditional simulation methods: a pilot study. Nurs. Educ. Perspect. 41(2), 119–121 (2020)
Juan, M.C., Elexpuru, J., Dias, P., Santos, B.S., Amorim, P.: Immersive virtual reality for upper limb rehabilitation: comparing hand and controller interaction. Virtual Reality, 1–15 (2022)
McGloin, R., Farrar, K., Krcmar, M.: Video games, immersion, and cognitive aggression: does the controller matter? Media Psychol. 16(1), 65–87 (2013)
Xie, B., et al.: A review on virtual reality skill training applications. Front. Virtual Reality 2, 645153 (2021)
Cabibihan, J.J., et al.: Suitability of the openly accessible 3D printed prosthetic hands for war-wounded children. Front. Robot. AI 7, 594196 (2021)
Zhang, X.Y., Nakamura, Y., Goda, K., Yoshimoto, K.: Robustness of power grasp. In: Proceedings of the 1994 IEEE International Conference on Robotics and Automation, vol. 4, pp. 2828–2835 (1994)
Bullock, I.M., Zheng, J.Z., De La Rosa, S., Guertler, C., Dollar, A.M.: Grasp frequency and usage in daily household and machine shop tasks. IEEE Trans. Haptics 6(3), 296–308 (2013)
Demartino, A.M., Rodrigues, L.C., Gomes, R.P., Michaelsen, S.M.: Hand function and type of grasp used by chronic stroke individuals in actual environment. Top. Stroke Rehabil. 26(4), 247–254 (2019)
Goršič, M., Cikajlo, I., Goljar, N., Novak, D.: A multisession evaluation of an adaptive competitive arm rehabilitation game. J. Neuroeng. Rehabil. 14(1), 1–15 (2017)
Brooke, J.: SUS-a quick and dirty usability scale. Usability Evaluation in Industry, 189 (1996)
Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)
Witmer, B.G., Jerome, C.J., Singer, M.J.: The factor structure of the presence questionnaire. Presence: Teleoperators Virtual Environ. 14(3), 298–312 (2005). https://doi.org/10.1162/105474605323384654
Zhou, Z., Li, J., Wang, H., Luan, Z., Li, Y., Peng, X.: Upper limb rehabilitation system based on virtual reality for breast cancer patients: development and usability study. PLOS ONE 16(12), 1–16 (2021). https://doi.org/10.1371/journal.pone.0261220
Morizio, C., Compagnat, M., Boujut, A., Labbani-Igbida, O., Billot, M., Perrochon, A.: Immersive virtual reality during robot-assisted gait training: validation of a new device in stroke rehabilitation. Medicina 58(12) (2022). https://www.mdpi.com/1648-9144/58/12/1805
Lee, N., Choi, W., Lee, S.: Development of a 360-degree virtual reality video-based immersive cycle training system for physical enhancement in older adults: a feasibility study. BMC Geriatrics 21, 325 (2021). https://doi.org/10.1186/s12877-021-02263-1
Gentry, S.V., et al.: Serious gaming and gamification education in health professions: systematic review. J. Med. Internet Res. 21(3), e12994 (2019)
Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum.-Comput. Interact. 2013, 1–1 (2013)
Acknowledgements
This work was supported by a research grant from Qatar University under grant no. IRCC-2022-541. The statements made herein are solely the responsibility of the authors. The authors declare that they have no conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Mazhar, N., Gaballa, A., Pandey, A.K., Cabibihan, JJ. (2024). Virtual Reality Hand Tracking for Immersive Telepresence in Rehabilitative Serious Gaming. In: Ali, A.A., et al. Social Robotics. ICSR 2023. Lecture Notes in Computer Science(), vol 14453 . Springer, Singapore. https://doi.org/10.1007/978-981-99-8715-3_1
Download citation
DOI: https://doi.org/10.1007/978-981-99-8715-3_1
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8714-6
Online ISBN: 978-981-99-8715-3
eBook Packages: Computer ScienceComputer Science (R0)