research-article

Open access

E-textile Sleeve with Graphene Strain Sensors for Arm Gesture Classification of Mid-Air Interactions

Authors:

Yangfangzheng Li,

Rebecca StewartAuthors Info & Claims

TEI '24: Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction

Article No.: 26, Pages 1 - 10

https://doi.org/10.1145/3623509.3633374

Published: 11 February 2024 Publication History

All formats PDF

Abstract

Arm gestures play a pivotal role in facilitating natural mid-air interactions. While computer vision techniques aim to detect these gestures, they encounter obstacles like obfuscation and lighting conditions. Alternatively, wearable devices have leveraged interactive textiles to recognize arm gestures. However, these methods predominantly emphasize textile deformation-based interactions, like twisting or grasping the sleeve, rather than tracking the natural body movement.This study bridges this gap by introducing an e-textile sleeve system that integrates multiple ultra-sensitive graphene e-textile strain sensors in an arrangement that captures bending and twisting along with an inertia measurement unit into a sports sleeve. This paper documents a comprehensive overview of the sensor design, fabrication process, seamless interconnection method, and detachable hardware implementation that allows for reconfiguring the processing unit to other body parts. A user study with ten participants demonstrated that the system could classify six different fundamental arm gestures with over 90% accuracy.

References

[1]

Asli Atalay, Vanessa Sanchez, Ozgur Atalay, Daniel M. Vogt, Florian Haufe, Robert J. Wood, and Conor J. Walsh. 2017. Batch Fabrication of Customizable Silicone-Textile Composite Capacitive Strain Sensors for Human Motion Tracking. Advanced Materials Technologies 2, 9 (2017), 1700136. https://doi.org/10.1002/admt.201700136 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/admt.201700136.

[2]

Weiya Chen, Chenchen Yu, Chenyu Tu, Zehua Lyu, Jing Tang, Shiqi Ou, Yan Fu, and Zhidong Xue. 2020. A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods. Sensors 20, 4 (Jan. 2020), 1074. https://doi.org/10.3390/s20041074 Number: 4 Publisher: Multidisciplinary Digital Publishing Institute.

[3]

Xiang Chen, Yu Li, Ruochen Hu, Xu Zhang, and Xun Chen. 2021. Hand Gesture Recognition based on Surface Electromyography using Convolutional Neural Network with Transfer Learning Method. IEEE Journal of Biomedical and Health Informatics 25, 4 (April 2021), 1292–1304. https://doi.org/10.1109/JBHI.2020.3009383 Conference Name: IEEE Journal of Biomedical and Health Informatics.

[4]

Yu Enokibori and Kenji Mase. 2014. Human joint angle estimation with an e-textile sensor. In Proceedings of the 2014 ACM International Symposium on Wearable Computers(ISWC ’14). Association for Computing Machinery, New York, NY, USA, 129–130. https://doi.org/10.1145/2634317.2634331

Digital Library

[5]

Dustin Freeman, Ramadevi Vennelakanti, and Sriganesh Madhvanath. 2012. Freehand pose-based Gestural Interaction: Studies and implications for interface design. In 2012 4th International Conference on Intelligent Human Computer Interaction (IHCI). 1–6. https://doi.org/10.1109/IHCI.2012.6481816

[6]

Darren Guinness, Alvin Jude, G. Michael Poor, and Ashley Dover. 2015. Models for Rested Touchless Gestural Interaction. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction(SUI ’15). Association for Computing Machinery, New York, NY, USA, 34–43. https://doi.org/10.1145/2788940.2788948

Digital Library

[7]

Nur Al-huda Hamdan, Jeffrey R. Blum, Florian Heller, Ravi Kanth Kosuru, and Jan Borchers. 2016. Grabbing at an angle: menu selection for fabric interfaces. In Proceedings of the 2016 ACM International Symposium on Wearable Computers(ISWC ’16). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/2971763.2971786

Digital Library

[8]

Chris Harrison, Shilpa Ramamurthy, and Scott E. Hudson. 2012. On-body interaction: armed and dangerous. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction(TEI ’12). Association for Computing Machinery, New York, NY, USA, 69–76. https://doi.org/10.1145/2148131.2148148

Digital Library

[9]

Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: smartwatch-based 3D pointing for smartphone-based virtual reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology(VRST ’16). Association for Computing Machinery, New York, NY, USA, 365–366. https://doi.org/10.1145/2993369.2996311

Digital Library

[10]

Rushil Khurana, Mayank Goel, and Kent Lyons. 2019. Detachable Smartwatch: More Than A Wearable. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 2 (2019), 50:1–50:14. https://doi.org/10.1145/3328921

Digital Library

[11]

Ozgun Kilic Afsar, Ali Shtarbanov, Hila Mor, Ken Nakagaki, Jack Forman, Karen Modrei, Seung Hee Jeong, Klas Hjort, Kristina Höök, and Hiroshi Ishii. 2021. OmniFiber: Integrated Fluidic Fiber Actuators for Weaving Movement based Interactions into the ‘Fabric of Everyday Life’. In The 34th Annual ACM Symposium on User Interface Software and Technology(UIST ’21). Association for Computing Machinery, New York, NY, USA, 1010–1026. https://doi.org/10.1145/3472749.3474802

Digital Library

[12]

Arun Kulshreshth and Joseph J. LaViola. 2014. Exploring the usefulness of finger-based 3D gesture menu selection. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’14). Association for Computing Machinery, New York, NY, USA, 1093–1102. https://doi.org/10.1145/2556288.2557122

Digital Library

[13]

An Liang, Rebecca Stewart, and Nick Bryan-Kinns. 2019. Analysis of Sensitivity, Linearity, Hysteresis, Responsiveness, and Fatigue of Textile Knit Stretch Sensors. Sensors (Basel, Switzerland) 19, 16 (Aug. 2019), 3618. https://doi.org/10.3390/s19163618

[14]

An Liang, Rebecca Stewart, Rachel Freire, and Nick Bryan-Kinns. 2019. Effect of bonding and washing on electronic textile stretch sensor properties. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers(UbiComp/ISWC ’19 Adjunct). Association for Computing Machinery, New York, NY, USA, 121–124. https://doi.org/10.1145/3341162.3343817

Digital Library

[15]

An Liang, Rebecca Stewart, Rachel Freire, and Nick Bryan-Kinns. 2021. Knit Stretch Sensor Placement for Body Movement Sensing. In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction(TEI ’21). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3430524.3440629

Digital Library

[16]

Ruibo Liu, Qijia Shao, Siqi Wang, Christina Ru, Devin Balkcom, and Xia Zhou. 2019. Reconstructing Human Joint Motion with Computational Fabrics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 1 (2019), 19:1–19:26. https://doi.org/10.1145/3314406

Digital Library

[17]

Xiaolong Lou, Qinping Zhao, Yan Shi, and Preben Hansen. 2022. Arm Posture Changes and Influences on Hand Controller Interaction Evaluation in Virtual Reality. Applied Sciences 12, 5 (Jan. 2022), 2585. https://doi.org/10.3390/app12052585 Number: 5 Publisher: Multidisciplinary Digital Publishing Institute.

[18]

Jordan Lui and Carlo Menon. 2019. Would a thermal sensor improve arm motion classification accuracy of a single wrist-mounted inertial device?BioMedical Engineering OnLine 18, 1 (May 2019), 53. https://doi.org/10.1186/s12938-019-0677-7

[19]

Brad A. Myers, Rishi Bhatnagar, Jeffrey Nichols, Choon Hong Peck, Dave Kong, Robert Miller, and A. Chris Long. 2002. Interacting at a distance: measuring the performance of laser pointers and other devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’02). Association for Computing Machinery, New York, NY, USA, 33–40. https://doi.org/10.1145/503376.503383

Digital Library

[20]

Mathieu Nancel, Emmanuel Pietriga, Olivier Chapuis, and Michel Beaudouin-Lafon. 2015. Mid-Air Pointing on Ultra-Walls. ACM Transactions on Computer-Human Interaction 22, 5 (2015), 21:1–21:62. https://doi.org/10.1145/2766448

Digital Library

[21]

Alex Olwal, Thad Starner, and Gowa Mainini. 2020. E-Textile Microinteractions: Augmenting Twist with Flick, Slide and Grasp Gestures for Soft Electronics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376236

Digital Library

[22]

Patrick Parzer, Adwait Sharma, Anita Vogl, Jürgen Steimle, Alex Olwal, and Michael Haller. 2017. SmartSleeve: Real-time Sensing of Surface and Deformation Gestures on Flexible, Interactive Textiles, using a Hybrid Gesture Detection Pipeline. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology(UIST ’17). Association for Computing Machinery, New York, NY, USA, 565–577. https://doi.org/10.1145/3126594.3126652

Digital Library

[23]

Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E. Robinson. 2016. Project Jacquard: Interactive Digital Textiles at Scale. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). Association for Computing Machinery, New York, NY, USA, 4216–4227. https://doi.org/10.1145/2858036.2858176

Digital Library

[24]

Jagdish Lal Raheja, Mona Chandra, and Ankit Chaudhary. 2018. 3D gesture based real-time object selection and recognition. Pattern Recognition Letters 115 (Nov. 2018), 14–19. https://doi.org/10.1016/j.patrec.2017.09.034

[25]

Seongwoo Ryu, Phillip Lee, Jeffrey B. Chou, Ruize Xu, Rong Zhao, Anastasios John Hart, and Sang-Gook Kim. 2015. Extremely Elastic Wearable Carbon Nanotube Fiber Strain Sensor for Monitoring of Human Motion. ACS Nano 9, 6 (June 2015), 5929–5936. https://doi.org/10.1021/acsnano.5b00599 Publisher: American Chemical Society.

[26]

T. Scott Saponas, Chris Harrison, and Hrvoje Benko. 2011. PocketTouch: through-fabric capacitive touch input. In Proceedings of the 24th annual ACM symposium on User interface software and technology(UIST ’11). Association for Computing Machinery, New York, NY, USA, 303–308. https://doi.org/10.1145/2047196.2047235

Digital Library

[27]

Stefan Schneegass and Alexandra Voit. 2016. GestureSleeve: using touch sensitive fabrics for gestural input on the forearm for controlling smartwatches. In Proceedings of the 2016 ACM International Symposium on Wearable Computers(ISWC ’16). Association for Computing Machinery, New York, NY, USA, 108–115. https://doi.org/10.1145/2971763.2971797

Digital Library

[28]

Shayan Seyedin, Peng Zhang, Maryam Naebe, Si Qin, Jun Chen, Xungai Wang, and Joselito M. Razal. 2019. Textile strain sensors: a review of the fabrication technologies, performance evaluation and applications. Materials Horizons 6, 2 (Feb. 2019), 219–249. https://doi.org/10.1039/C8MH01062E Publisher: The Royal Society of Chemistry.

[29]

Adwait Sharma, Christina Salchow-Hömmen, Vimal Suresh Mollyn, Aditya Shekhar Nittala, Michael A. Hedderich, Marion Koelle, Thomas Seel, and Jürgen Steimle. 2022. SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-Grained Finger Microgestures. ACM Transactions on Computer-Human Interaction (2022). https://doi.org/10.1145/3569894 Just Accepted.

Digital Library

[30]

Yuying Si, Sujie Chen, Ming Li, Siying Li, Yisen Pei, and Xiaojun Guo. 2022. Flexible Strain Sensors for Wearable Hand Gesture Recognition: From Devices to Systems. Advanced Intelligent Systems 4, 2 (2022), 2100046. https://doi.org/10.1002/aisy.202100046 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/aisy.202100046.

[31]

Sophie Skach and Rebecca Stewart. 2019. One leg at a time: towards optimised design engineering of textile sensors in trousers. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers(UbiComp/ISWC ’19 Adjunct). Association for Computing Machinery, New York, NY, USA, 206–209. https://doi.org/10.1145/3341162.3343775

Digital Library

[32]

Cenxiao Tan, Zhigang Dong, Yehua Li, Haiguang Zhao, Xingyi Huang, Zhaocai Zhou, Jin-Wu Jiang, Yun-Ze Long, Pingkai Jiang, Tong-Yi Zhang, and Bin Sun. 2020. A high performance wearable strain sensor with advanced thermal management for motion monitoring. Nature Communications 11, 1 (July 2020), 3530. https://doi.org/10.1038/s41467-020-17301-6 Number: 1 Publisher: Nature Publishing Group.

[33]

Dinh-Son Tran, Ngoc-Huynh Ho, Hyung-Jeong Yang, Eu-Tteum Baek, Soo-Hyung Kim, and Gueesang Lee. 2020. Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network. Applied Sciences 10, 2 (Jan. 2020), 722. https://doi.org/10.3390/app10020722 Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.

[34]

Rafael Veras, Gaganpreet Singh, Farzin Farhadi-Niaki, Ritesh Udhani, Parth Pradeep Patekar, Wei Zhou, Pourang Irani, and Wei Li. 2021. Elbow-Anchored Interaction: Designing Restful Mid-Air Input. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3411764.3445546

Digital Library

[35]

Panagiotis Vogiatzidakis and Panayiotis Koutsabasis. 2022. ‘Address and command’: Two-handed mid-air interactions with multiple home devices. International Journal of Human-Computer Studies 159 (March 2022), 102755. https://doi.org/10.1016/j.ijhcs.2021.102755

Digital Library

[36]

Chi Cuong Vu and Jooyong Kim. 2020. Highly Sensitive E-Textile Strain Sensors Enhanced by Geometrical Treatment for Human Monitoring. Sensors 20, 8 (Jan. 2020), 2383. https://doi.org/10.3390/s20082383 Number: 8 Publisher: Multidisciplinary Digital Publishing Institute.

[37]

Jilong Wang, Chunhong Lu, and Kun Zhang. 2020. Textile-Based Strain Sensor for Human Motion Detection. ENERGY & ENVIRONMENTAL MATERIALS 3, 1 (2020), 80–100. https://doi.org/10.1002/eem2.12041 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/eem2.12041.

[38]

Tony Wu, Shiho Fukuhara, Nicholas Gillian, Kishore Sundara-Rajan, and Ivan Poupyrev. 2020. ZebraSense: A Double-sided Textile Touch Sensor for Smart Clothing. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(UIST ’20). Association for Computing Machinery, New York, NY, USA, 662–674. https://doi.org/10.1145/3379337.3415886

Digital Library

[39]

Guanghua Xu, Quan Wan, Wenwu Deng, Tao Guo, and Jingyuan Cheng. 2022. Smart-Sleeve: A Wearable Textile Pressure Sensor Array for Human Activity Recognition. Sensors 22, 5 (Jan. 2022), 1702. https://doi.org/10.3390/s22051702 Number: 5 Publisher: Multidisciplinary Digital Publishing Institute.

[40]

Haoran Yang, Yong Wang, Huiqing Wang, Yunde Shi, Lifeng Zhu, Yujin Kuang, and Yuan Yang. 2022. Multi-Inertial Sensor-Based Arm 3D Motion Tracking Using Elman Neural Network. Journal of Sensors 2022 (May 2022), e3926417. https://doi.org/10.1155/2022/3926417 Publisher: Hindawi.

[41]

Tianhong Catherine Yu, Riku Arakawa, James McCann, and Mayank Goel. 2023. uKnit: A Position-Aware Reconfigurable Machine-Knitted Wearable for Gestural Interaction and Passive Sensing using Electrical Impedance Tomography. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems(CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3544548.3580692

Digital Library

[42]

Yi Zhou, Connor Myant, and Rebecca Stewart. 2022. Multifunctional and stretchable graphene/textile composite sensor for human motion monitoring. Journal of Applied Polymer Science 139, 32 (2022), e52755. https://doi.org/10.1002/app.52755 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/app.52755.

[43]

Yi Zhou and Rebecca Stewart. 2022. Highly flexible, durable, UV resistant, and electrically conductive graphene based TPU/textile composite sensor. Polymers for Advanced Technologies 33, 12 (2022), 4250–4264. https://doi.org/10.1002/pat.5856 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/pat.5856.

Cited By

Ruston OSharma AFraser M(2024)SeamSleeve: Robust Arm Movement Sensing through Powered StitchingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660726(1134-1147)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660726

Index Terms

E-textile Sleeve with Graphene Strain Sensors for Arm Gesture Classification of Mid-Air Interactions
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices

Recommendations

Bimanual Word Gesture Keyboards for Mid-air Gestures
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Mid-air hand gestural interaction has generally been researched as a pointing device. However, recent research has shown potential for text input with the use of word gesture keyboards (WGK), where these forms of interactions require the input system to ...
‘Address and command’: Two-handed mid-air interactions with multiple home devices
- The A&C approach enables two-handed mid-air interactions with multiple devices
- An ...
Abstract
Mid-air gestural interaction with multiple targets (or devices) in the physical space presents several challenges: gesture design gets more complex in the attempt to avoid gesture conflicts and accidental activations; the potential ...
Exploring a user-defined gesture vocabulary for descriptive mid-air interactions

Gesturing provides an alternative interaction input for design that is more natural and intuitive. However, standard input devices do not completely reflect natural hand motions in design. A key challenge lies in how gesturing can contribute to human---...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

TEI '24: Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 2024

1058 pages

ISBN:9798400704024

DOI:10.1145/3623509

General Chairs:
Luigina Ciolfi
University College Cork
,
Trevor Hogan
Munster Technological University
,
Program Chairs:
Tanja Döring
University of Bremen
,
Tom Jenkins
IT University of Copenhagen
,
Jelle van Dijk
University of Twente
,
Samuel Huron
Télécom Paris
,
Zhuying Li
Southeast University
,
Publications Chairs:
David Coyle
University College Dublin
,
Beat Signer
Vrije Universiteit Brussel

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 February 2024

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

TEI '24

Sponsor:

SIGCHI

TEI '24: Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 11 - 14, 2024

Cork, Ireland

Acceptance Rates

Overall Acceptance Rate 393 of 1,367 submissions, 29%

Upcoming Conference

TEI '25

Sponsor:
sigchi

Nineteenth International Conference on Tangible, Embedded, and Embodied Interaction

March 4 - 7, 2025

Bordeaux / Talence , France

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
507
Total Downloads

Downloads (Last 12 months)507
Downloads (Last 6 weeks)79

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ruston OSharma AFraser M(2024)SeamSleeve: Robust Arm Movement Sensing through Powered StitchingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660726(1134-1147)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660726

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents