Abstract
Among other aspects like attention and focus, the effectiveness of actions, such as training in percussion, also relies on the correct gesture and strength of the individual carrying out the activity repetitively to develop proper muscle memory and train motor skills. To make this process economical and efficient, we propose a system that models the feature of direction and strength of a tap and receives the corresponding active haptic feedback in an immersive virtual environment. We propose a novel tapping gesture model that takes electromyogram and vibration feedback into consideration. We also propose an approach of designing vibration modes that follow the mechanical stimulation principles and generate distinguishable vibration feedback with low-fidelity actuators. We developed a prototype using myoelectric and haptic apparatus (Myo armband and HTC VIVE controllers) and evaluated our system by conducting a user study with 50 participants. The evaluation shows that the users can distinguish three common materials (wood, rubber, and aluminum), and our system allows them to actively adjust their direction and strength to grasp the correct point while tapping. We demonstrate the efficacy of our tapping model on a virtual Chinese chimes application and observe positive feedback from both novices and expert users.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Akhtar A, Hargrove LJ, Bretl T (2012) Prediction of distal arm joint angles from EMG and shoulder orientation for prosthesis control. In: 2012 annual international conference of the IEEE engineering in medicine and biology society. IEEE, pp 4160–4163
Asamura N, Yokoyama N, Shinoda H (1999) A method of selective stimulation to epidermal skin receptors for realistic touch feedback. In: IEEE virtual reality, pp 274–281
Avanzini F, Crosato P (2006) Integrating physically based sound models in a multimodal rendering architecture. Comput Animat Virtual Worlds 17(3–4):411–419
Camacho Navarro J, Leon-Vargas F, Barrero Pérez J (2012) EMG-based system for basic hand movement recognition. Dyna 79(171):41–49
Chang GC, Kang WJ, Luh JJ, Cheng CK, Lai JS, Chen JJJ, Kuo TS (1996) Real-time implementation of electromyogram pattern recognition as a control command of man-machine interface. Med Eng Phys 18(7):529–537
Choi S, Kuchenbecker KJ (2013) Vibrotactile display: perception, technology, and applications. Proc IEEE 101(9):2093–2104
Choi K, Hirose H, Iijima T, Koike Y (2006) Prediction of four degrees of freedom arm movement using EMG signal. In: 2005 IEEE engineering in medicine and biology 27th annual conference. IEEE, pp 5820–5823
Cosman PH, Cregan PC, Martin CJ, Cartmill JA (2002) Virtual reality simulators: current status in acquisition and assessment of surgical skills. ANZ J Surg 72(1):30–34
Gescheider GA (1976) Evidence in support of the duplex theory of mechanoreception. Sensory Processes
Gescheider GA, Berryhill ME, Verrillo RT, Bolanowski SJ (1999) Vibrotactile temporal summation: probability summation or neural integration? Somatosens Motor Res 16(3):229–242
Gescheider GA, Bolanowski SJ, Pope JV, Verrillo RT (2002) A four-channel analysis of the tactile sensitivity of the fingertip: frequency selectivity, spatial summation, and temporal summation. Somatosens Motor Res 19(2):114–124
Giordano B, Avanzini F, Wanderley M, Mcadams S (2010) Multisensory integration in percussion performance. In: 10ème Congrès Français d’Acoustique
Hachisu T, Sato M, Fukushima S, Kajimoto H (2012) Augmentation of material property by modulating vibration resulting from tapping. In: International conference on human haptic sensing and touch enabled computer applications, pp 173–180
Hudgins B, Parker P, Scott RN (1993) A new strategy for multifunction myoelectric control. IEEE Trans Biomed Eng 40(1):82–94
Jones LA, Lederman SJ (2006) Human hand function. Oxford University Press, Oxford
Kaliki RR, Davoodi R, Loeb GE (2008) Prediction of distal arm posture in 3-d space from shoulder movements for control of upper limb prostheses. Proc IEEE 96(7):1217–1225
Kim D, Hilliges O, Izadi S, Butler AD, Chen J, Oikonomidis I, Olivier P (2012) Digits: freehand 3d interactions anywhere using a wrist-worn gloveless sensor. In: Proceedings of the 25th annual ACM symposium on User interface software and technology, pp 167–176
Kim K, Kang J, Lee Y, Moon C, Choi H, Mun C (2011) The development of muscle training system using the electromyogram and interactive game for physical rehabilitation. In: 5th Kuala Lumpur international conference on biomedical engineering 2011. Springer, pp 801–804
Konyo M, Tadokoro S, Yoshida A, Saiwaki N (2005) A tactile synthesis method using multiple frequency vibrations for representing virtual touch. In: IEEE international conference on intelligent robots and systems, pp 3965–3971
Kuchenbecker KJ, Fiene J, Niemeyer G (2006) Improving contact realism through event-based haptic feedback. IEEE Trans Vis Comput Gr 12(2):219–230
Kwon C (2019) Verification of the possibility and effectiveness of experiential learning using HMD-based immersive VR technologies. Virtual Real 23(1):101–118
Lamata P, Gómez E, Sánchez-Margallo F, Lamata F, Del Pozo F, Usón J (2006) Tissue consistency perception in laparoscopy to define the level of fidelity in virtual reality simulation. Surg Endosc Interv Tech 20(9):1368–1375
Lenzi T, De Rossi SM, Vitiello N, Carrozza MC (2012) Intention-based EMG control for powered exoskeletons. IEEE Trans Biomed Eng 59(8):2180–2190
Lyons G, Sharma P, Baker M, O’Malley S, Shanahan A (2003) A computer game-based EMG biofeedback system for muscle rehabilitation. In: IEEE engineering in medicine and biology society, pp 1625–1628
Nymoen K, Haugen MR, Jensenius AR (2015) Mumyo—evaluating and exploring the myo armband for musical interaction. In: International conference on new interfaces for musical expression, Louisiana State University
Okamura AM, Cutkosky MR, Dennerlein JT (2001) Reality-based models for vibration feedback in virtual environments. IEEE/ASME Trans Mechatron 6(3):245–252
Pacchierotti C (2015) Cutaneous haptic feedback in robotic teleoperation. Springer, Berlin
Pai YS, Dingler T, Kunze K (2019) Assessing hands-free interactions for VR using eye gaze and electromyography. Virtual Reality 23(2):119–131
Papetti S, Fröhlich M, Fontana F, Schiesser S, Avanzini F (2018) Implementation and characterization of vibrotactile interfaces. In: Musical haptics. Springer, Berlin, pp 257–282
Phinyomark A, Hirunviriya S, Limsakul C, Phukpattaranont P (2010) Evaluation of EMG feature extraction for hand movement recognition based on Euclidean distance and standard deviation. ECTI-CON2010: The 2010 ECTI international confernce on electrical engineering/electronics. Computer, Telecommunications and Information Technology. IEEE, pp 856–860
Pons JL, Ceres R, Rocon E, Levin S, Markovitz I, Saro B, Reynaerts D, Moorleghem WV, Bueno L (2005) Virtual reality training and EMG control of the MANUS hand prosthesis. Robotica 23(3):311–317
Pulliam CL, Lambrecht JM, Kirsch RF (2011) Electromyogram-based neural network control of transhumeral prostheses. J Rehabil Res Dev 48(6):739
Reaz MBI, Hussain MS, Mohd-Yasin F (2006) Techniques of EMG signal analysis: detection, processing, classification and applications. Biol Proced Online 8(1):11–35
Ryge A, Thomsen L, Berthelsen T, Hvass JS, Koreska L, Vollmers C, Nilsson NC, Nordahl R, Serafin S (2017) Effect on high versus low fidelity haptic feedback in a virtual reality baseball simulation. In: IEEE virtual reality, pp 365–366
Saponas TS, Tan DS, Morris D, Balakrishnan R, Turner J, Landay JA (2009) Enabling always-available input with muscle-computer interfaces. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology, pp 167–176
Saponas TS, Tan DS, Morris D, Turner J, Landay JA (2010) Making muscle-computer interfaces more practical. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 851–854
Shima K, Tsuji T (2009) An MMG-based human-assisting manipulator using acceleration sensors. In: 2009 IEEE international conference on systems, man and cybernetics. IEEE, pp 2433–2438
Ström P, Hedman L, Särnå L, Kjellin A, Wredmark T, Felländer-Tsai L (2006) Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Endosc Interv Tech 20(9):1383–1388
Van der Meijden OA, Schijven MP (2009) The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surg Endosc 23(6):1180–1190
Wellman P, Howe RD (1995) Towards realistic vibrotactile display in virtual environments. ASME Dyn Syst Control Div 57:713–718
Westling G, Johansson RS (1987) Responses in glabrous skin mechanoreceptors during precision grip in humans. Exp Brain Res 66(1):128–140
Wirth M, Grad S, Poimann D, Richer R, Ottmann J, Eskofier BM (2018) Exploring the feasibility of EMG based interaction for assessing cognitive capacity in virtual reality. In: 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 4953–4956
Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence 7(3):225–240
Wu CM, Hsu CW, Lee TK, Smith S (2017) A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment. Virtual Real 21(1):19–29
Yem V, Kajimoto H (2017) Wearable tactile device using mechanical and electrical stimulation for fingertip interaction with virtual world. In: IEEE virtual reality, pp 99–104
Acknowledgements
This work was supported by National Key R&D Program of China (No.2017YFB1402105, 2017YFB1002604), National Key Cooperation between the BRICS Program of China(No.2017YFE0100500) and Beijing Natural Science Foundation of China (No. 4172033).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 284061 KB)
Rights and permissions
About this article
Cite this article
Xu, P., Wu, Z., Wang, X. et al. An electromyogram-based tapping gesture model with differentiated vibration feedback by low-fidelity actuators. Virtual Reality 25, 383–397 (2021). https://doi.org/10.1007/s10055-020-00458-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-020-00458-2