Exploring Potential of Electromyography-based Avatar Operation Using Residual Muscles of ALS Individuals: Case Study on Avatar DJ Performance
Article No.: 146, Pages 1 - 7
Abstract
Assistive medical device, such as gastrostomy and ventilation, plays a crucial role in extending the lifespan of individuals with amyotrophic lateral sclerosis (ALS). Additionally, augmentative and alternative communication (AAC) systems are essential for maintaining their ability to communicate. Designing an effective AAC system requires balancing the user’s expressive capability with the usability of the user interface. In this study, we focus on not gaze input and electroencephalograph (EEG) signal input commonly used in AAC systems but electromyography (EMG) signal input as a novel modality to augment their embodiment into a virtual space. Since EMG signals can be measured even with slight movements of residual muscles, EMG-based AAC systems provide a potential avenue for communication. We successfully demonstrated the feasibility of our EMG-based avatar communication system by enabling Masatane Muto, a DJ musician and creator living with ALS, to perform live remotely from Japan to the Ars Electronica Festival 2023 in Linz, Austria using the system. Through this case study and a pilot study, we aimed to explore the potential of EMG-based AAC systems by discussing three perspectives: augmented embodiment, prolonged use, and extensibility into real environments in diverse contexts.
Supplemental Material
- Download
- 19.40 MB
- Transcript
- Download
- 51.04 MB
- Transcript
References
[1]
WITH ALS. 2023. WITH ALS. Retrieved October 11, 2023 from http://withals.com/
[2]
Krzysztof Barć and Magdalena Kuźma-Kozakiewicz. 2020. Gastrostomy and mechanical ventilation in amyotrophic lateral sclerosis: how best to support the decision-making process?Neurologia i Neurochirurgia Polska 54, 5 (2020), 366–377. https://doi.org/10.5603/PJNNS.a2020.0081
[3]
Jiamin Dai, Karyn Moffatt, Jinglan Lin, and Khai Truong. 2022. Designing for Relational Maintenance: New Directions for AAC Research. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 602, 15 pages. https://doi.org/10.1145/3491102.3502011
[4]
Molly Doyle and Barbara Phillips. 2001. Trends in augmentative and alternative communication use by individuals with amyotrophic lateral sclerosis. Augmentative and Alternative Communication 17, 3 (2001), 167–178.
[5]
Ars electronica. 2023. Project Humanity. Retrieved October 11, 2023 from https://ars.electronica.art/who-owns-the-truth/en/events/project-humanity/
[6]
Alexander Fiannaca, Ann Paradiso, Mira Shah, and Meredith Ringel Morris. 2017. AACrobat: Using Mobile Devices to Lower Communication Barriers and Provide Autonomy with Gaze-Based AAC. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17). Association for Computing Machinery, New York, NY, USA, 683–695. https://doi.org/10.1145/2998181.2998215
[7]
Yuko Hirano. 2017. Illness experiences of Amyotrophic lateral sclerosis patients under tracheostomy invasive positive pressure ventilation: A longitudinal study of psychological changes and related factors using the life-line method. Japanese Journal of Health and Human Ecology 83, 2 (2017), 55–69. https://doi.org/10.3861/kenko.83.2_55
[8]
Yuko HIRANO and Yoshihiko YAMAZAKI. 2015. Illness experience of amyotrophic lateral sclerosis patients under tracheostomy invasive positive pressure ventilation: Use of the life-line method to elucidate changes in and correlations among psychological well-being, demographic characteristics, and physical or psychosocial factors. Japanese Journal of Health and Human Ecology 81, 5 (2015), 143–158. https://doi.org/10.3861/jshhe.81.143
[9]
Esther V Hobson and Christopher J McDermott. 2016. Supportive and symptomatic management of amyotrophic lateral sclerosis. Nature Reviews Neurology 12, 9 (2016), 526–538. https://doi.org/10.1038/nrneurol.2016.111
[10]
Dentsu Inc.2023. Dentsu Lab Tokyo’s All Players Welcome Enables ’Physical’ Expression in Metaverse. Retrieved October 11, 2023 from https://www.lbbonline.com/news/dentsu-lab-tokyos-all-players-welcome-enables-physical-expression-in-metaverse
[11]
Balazs Andras Ivanyi, Truls Bendik Tjemsland, Christian Vasileios Tsalidis de Zabala, Lilla Julia Toth, Marcus Alexander Dyrholm, Scott James Naylor, Ann Paradiso, Dwayne Lamb, Jarnail Chudge, Ali Adjorlu, and Stefania Serafin. 2023. DuoRhythmo: Design and Remote User Experience Evaluation (UXE) of a Collaborative Accessible Digital Musical Interface (CADMI) for People with ALS (PALS). In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 56, 13 pages. https://doi.org/10.1145/3544548.3581285
[12]
Shaun K. Kane, Meredith Ringel Morris, Ann Paradiso, and Jon Campbell. 2017. "At Times Avuncular and Cantankerous, with the Reflexes of a Mongoose": Understanding Self-Expression through Augmentative and Alternative Communication Devices. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17). Association for Computing Machinery, New York, NY, USA, 1166–1179. https://doi.org/10.1145/2998181.2998284
[13]
So-Yeon Kim, Hyojin Park, Myeongul Jung, and Kwanguk Kim. 2020. Impact of body size match to an avatar on the body ownership illusion and user’s subjective experience. Cyberpsychology, Behavior, and Social Networking 23, 4 (2020), 234–241.
[14]
Magdalena Kuźma-Kozakiewicz. 2018. Edaravone in the treatment of amyotrophic lateral sclerosis. Neurologia i Neurochirurgia Polska 52, 2 (2018), 124–128.
[15]
Magdalena Kuzma-Kozakiewicz, Peter M Andersen, Katarzyna Ciecwierska, Cynthia Vázquez, Olga Helczyk, Markus Loose, Ingo Uttner, Albert C Ludolph, and Dorothée Lulé. 2019. An observational study on quality of life and preferences to sustain life in locked-in state. Neurology 93, 10 (2019), e938–e945. https://doi.org/10.1212/WNL.0000000000008064
[16]
Dorothée Lulé, Claudia Zickler, Sonja Häcker, Marie-Aurelie Bruno, Athena Demertzi, Frederic Pellas, Steven Laureys, and A Kübler. 2009. Life can be worth living in locked-in syndrome. Progress in brain research 177 (2009), 339–351. https://doi.org/10.1016/S0079-6123(09)17723-3
[17]
Päivi Majaranta, Ulla-Kaija Ahola, and Oleg Špakov. 2009. Fast Gaze Typing with an Adjustable Dwell Time. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 357–360. https://doi.org/10.1145/1518701.1518758
[18]
Willie McClinton, Derek Caprio, Denis Laesker, Blanche Pinto, Sarah Garcia, and Marvin Andujar. 2019. P300-Based 3D Brain Painting in Virtual Reality. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312968
[19]
Dennis J McFarland. 2020. Brain-computer interfaces for amyotrophic lateral sclerosis. Muscle & nerve 61, 6 (2020), 702–707. https://doi.org/10.1002/mus.26828
[20]
Hiroshi Mitsumoto. 2009. Amyotrophic lateral sclerosis: a guide for patients and families. Demos Medical Publishing.
[21]
Martez E. Mott, Shane Williams, Jacob O. Wobbrock, and Meredith Ringel Morris. 2017. Improving Dwell-Based Gaze Typing with Dynamic, Cascading Dwell Times. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2558–2570. https://doi.org/10.1145/3025453.3025517
[22]
NTT. 2023. Initiatives to Enrich Communication for ALS Symbionts To create a world where people with ALS can freely move, express themselves and interact with others. Retrieved June 14, 2023 from https://group.ntt/en/newsrelease/2023/06/14/230614a.html
[23]
NTT. 2023. Musician with ALS Performs at Ars Electronica Festival: Experience freedom through his avatar motions synchronized with remaining physical functions. Retrieved October 11, 2023 from https://group.ntt/en/newsrelease/2023/08/25/230825a.html
[24]
Diogo Pedrosa, Maria Da Graça Pimentel, Amy Wright, and Khai N. Truong. 2015. Filteryedping: Design Challenges and User Performance of Dwell-Free Eye Typing. ACM Trans. Access. Comput. 6, 1, Article 3 (mar 2015), 37 pages. https://doi.org/10.1145/2724728
[25]
Susana Pinto, Stefano Quintarelli, and Vincenzo Silani. 2020. New technologies and amyotrophic lateral sclerosis–which step forward rushed by the COVID-19 pandemic?Journal of the neurological sciences 418 (2020), 117081. https://doi.org/10.1016/j.jns.2020.117081
[26]
Lewis P Rowland and Neil A Shneider. 2001. Amyotrophic lateral sclerosis. New England Journal of Medicine 344, 22 (2001), 1688–1700. https://doi.org/10.1056/NEJM200105313442207
[27]
Mel Slater, Daniel Pérez Marcos, Henrik Ehrsson, and Maria V Sanchez-Vives. 2009. Inducing illusory ownership of a virtual body. Frontiers in neuroscience (2009), 29.
[28]
Kiley Sobel, Alexander Fiannaca, Jon Campbell, Harish Kulkarni, Ann Paradiso, Ed Cutrell, and Meredith Ringel Morris. 2017. Exploring the Design Space of AAC Awareness Displays. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2890–2903. https://doi.org/10.1145/3025453.3025610
[29]
S+T+ARTS. 2023. ALL PLAYERS TOOL LAB.Masatane Muto(JP), Dentsu Lab Tokyo (JP). https://starts-prize.aec.at/en/all-players-tool-lab/.
[30]
Tomohiro Tanaka, Ryo Masumura, Mana Ihori, Hiroshi Sato, Taiga Yamane, Takanori Ashihara, Kohei Matsuura, and Takafumi Moriya. 2023. Leveraging Language Embeddings for Cross-Lingual Self-Supervised Speech Representation Learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5.
[31]
Lokesh C Wijesekera and P Nigel Leigh. 2009. Amyotrophic lateral sclerosis. Orphanet journal of rare diseases 4 (2009), 1–22. https://doi.org/10.1186/1750-1172-4-3
[32]
Xiaoyi Zhang, Harish Kulkarni, and Meredith Ringel Morris. 2017. Smartphone-Based Gaze Gesture Communication for People with Motor Disabilities. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2878–2889. https://doi.org/10.1145/3025453.3025790
Index Terms
- Exploring Potential of Electromyography-based Avatar Operation Using Residual Muscles of ALS Individuals: Case Study on Avatar DJ Performance
Recommendations
Augmented control of an avatar using an SSVEP based BCI
AH '12: Proceedings of the 3rd Augmented Human International ConferenceThe demonstration shows the usage of an EEG-based brain-computer interface (BCI) for the real-time control of an avatar in World of Warcraft. Visitors can test the installation during the conference after about 5 minutes of training time. World of ...
Optimizing electrode positions on forearm to increase SNR and myoelectric pattern recognition performance
AbstractWith the advances in electromyography-based human–computer interaction, particularly in myoelectric prosthetic hands, the position of electromyography electrodes has gained less attention from researchers. However, the performance of hand ...
EEG-based brain-computer interface to a virtual walking avatar engages cortical adaptation
2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)Recent advances in brain-computer interface (BCI) technologies have shown the feasibility of neural decoding for both users' gait intent and continuous kinematics. However, the cortical adaptation and the dynamics of cortical involvement in human upright ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI '25
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 173Total Downloads
- Downloads (Last 12 months)173
- Downloads (Last 6 weeks)22
Reflects downloads up to 19 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format