research-article

Open access

NICER: A New and Improved Consumed Endurance and Recovery Metric to Quantify Muscle Fatigue of Mid-Air Interactions

Authors:

Robert Crowther,

Barrett EnsAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 43, Issue 4

Article No.: 102, Pages 1 - 14

https://doi.org/10.1145/3658230

Published: 19 July 2024 Publication History

Abstract

Natural gestures are crucial for mid-air interaction, but predicting and managing muscle fatigue is challenging. Existing torque-based models are limited in their ability to model above-shoulder interactions and to account for fatigue recovery. We introduce a new hybrid model, NICER, which combines a torque-based approach with a new term derived from the empirical measurement of muscle contraction and a recovery factor to account for decreasing fatigue during rest. We evaluated NICER in a mid-air selection task using two interaction methods with different degrees of perceived fatigue. Results show that NICER can accurately model above-shoulder interactions as well as reflect fatigue recovery during rest periods. Moreover, both interaction methods show a stronger correlation with subjective fatigue measurement (ρ = 0.978/0.976) than a previous model, Cumulative Fatigue (ρ = 0.966/0.923), confirming that NICER is a powerful analytical tool to predict fatigue across a variety of gesture-based interactive applications.

References

[1]

Ashrant Aryal, Ali Ghahramani, and Burcin Becerik-Gerber. 2017. Monitoring fatigue in construction workers using physiological measurements. Automation in Construction 82 (2017), 154--165.

[2]

Myroslav Bachynskyi, Antti Oulasvirta, Gregorio Palmas, and Tino Weinkauf. 2014. Is motion capture-based biomechanical simulation valid for hci studies? study and implications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3215--3224.

Digital Library

[3]

Myroslav Bachynskyi, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf. 2015. Informing the design of novel input methods with muscle coactivation clustering. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 6 (2015), 1--25.

Digital Library

[4]

Kayne Barclay, Danny Wei, Christof Lutteroth, and Robert Sheehan. 2011. A quantitative quality model for gesture based user interfaces. In Proceedings of the 23rd Australian Computer-Human Interaction Conference. 31--39.

Digital Library

[5]

Gunnar Borg. 1998. Borg's perceived exertion and pain scales. Human kinetics.

[6]

Doug A. Bowman, Ryan P. McMahan, and Eric D. Ragan. 2012. Questioning Naturalism in 3D User Interfaces. Commun. ACM 55, 9 (sep 2012), 78--88.

Digital Library

[7]

Don B Chaffin, Gunnar BJ Andersson, and Bernard J Martin. 2006. Occupational biomechanics. John wiley & sons.

[8]

Noshaba Cheema, Laura A Frey-Law, Kourosh Naderi, Jaakko Lehtinen, Philipp Slusallek, and Perttu Hämäläinen. 2020. Predicting mid-air interaction movements and fatigue using deep reinforcement learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--13.

Digital Library

[9]

Noshaba Cheema, Rui Xu, Nam Hee Kim, Perttu Hämäläinen, Vladislav Golyanik, Marc Habermann, Christian Theobalt, and Philipp Slusallek. 2023. Discovering Fatigued Movements for Virtual Character Animation. In SIGGRAPH Asia 2023 Conference Papers. 1--12.

[10]

Shaozhang Dai, Yi Li, Tim Dwyer, Barrett Ens, and Lonni Besançon. 2024. Magic Portal Interaction to Support Precise Embodied Mid-air and Haptic Selection-at-a-distance. (2024).

[11]

Niklas Elmqvist, Andrew Vande Moere, Hans-Christian Jetter, Daniel Cernea, Harald Reiterer, and T J Jankun-Kelly. 2011. Fluid interaction for information visualization. Information Visualization 10 (2011), 327--340. Issue 4.

Digital Library

[12]

João Marcelo Evangelista Belo, Anna Maria Feit, Tiare Feuchtner, and Kaj Grønbæk. 2021. XRgonomics: Facilitating the Creation of Ergonomic 3D Interfaces. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 290, 11 pages.

Digital Library

[13]

Mohd Farooq and Abid Ali Khan. 2012. Effect of elbow flexion, forearm rotation and upper arm abduction on MVC grip and grip endurance time. International Journal of Occupational Safety and Ergonomics 18, 4 (2012), 487--498.

[14]

Sue A Ferguson, W Gary Allread, Peter Le, Joseph D Rose, and William S Marras. 2011. Shoulder muscle oxygenation during repetitive tasks. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 55. SAGE Publications Sage CA: Los Angeles, CA, 1039--1041.

[15]

Lincoln E Ford, Alvin J Detterline, Kevin K Ho, and Wenyuan Cao. 2000. Gender-and height-related limits of muscle strength in world weightlifting champions. Journal of Applied Physiology 89, 3 (2000), 1061--1064.

[16]

Laura A Frey Law and Keith G Avin. 2010. Endurance time is joint-specific: a modelling and meta-analysis investigation. Ergonomics 53, 1 (2010), 109--129.

[17]

Laura A Frey-Law, John M Looft, and Jesse Heitsman. 2012. A three-compartment muscle fatigue model accurately predicts joint-specific maximum endurance times for sustained isometric tasks. Journal of biomechanics 45, 10 (2012), 1803--1808.

[18]

Laura A Frey-Law, Mitchell Schaffer, and Frank K Urban III. 2021. Muscle fatigue modelling: Solving for fatigue and recovery parameter values using fewer maximum effort assessments. International journal of industrial ergonomics 82 (2021), 103104.

[19]

Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139--183.

[20]

Kimberley Hayes, Judie R Walton, Zoltan L Szomor, and George AC Murrell. 2002. Reliability of 3 methods for assessing shoulder strength. Journal of Shoulder and Elbow surgery 11, 1 (2002), 33--39.

[21]

Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 1063--1072.

Digital Library

[22]

Sujin Jang, Wolfgang Stuerzlinger, Satyajit Ambike, and Karthik Ramani. 2017. Modeling cumulative arm fatigue in mid-air interaction based on perceived exertion and kinetics of arm motion. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 3328--3339.

Digital Library

[23]

Haiyan Jiang, Dongdong Weng, Xiaonuo Dongye, Le Luo, and Zhenliang Zhang. 2023. Commonsense Knowledge-Driven Joint Reasoning Approach for Object Retrieval in Virtual Reality. ACM Transactions on Graphics (TOG) 42, 6 (2023), 1--18.

Digital Library

[24]

Thomas Kosch, Jakob Karolus, Johannes Zagermann, Harald Reiterer, Albrecht Schmidt, and Paweł W. Woźniak. 2023. A Survey on Measuring Cognitive Workload in Human-Computer Interaction. Comput. Surveys (1 2023).

Digital Library

[25]

Bongshin Lee, Arjun Srinivasan, Petra Isenberg, and John Stasko. 2021. Post-WIMP Interaction for Information Visualization. Foundations and Trends® in Human-Computer Interaction 14 (2021), 1--95. Issue 1.

Digital Library

[26]

Yi Li, Robert Crowther, Jim Smiley, Tim Dwyer, Benjamin Tag, Pourang Irani, and Barrett Ens. 2023. Revisiting Consumed Endurance: A NICE Way to Quantify Shoulder Fatigue in Virtual Reality. In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology. 1--10.

Digital Library

[27]

Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--19.

Digital Library

[28]

RS Lindle, EJ Metter, NA Lynch, J vL Fleg, JL Fozard, J Tobin, TA Roy, and BF Hurley. 1997. Age and gender comparisons of muscle strength in 654 women and men aged 20--93 yr. Journal of applied physiology 83, 5 (1997), 1581--1587.

[29]

Jing Z Liu, Robert W Brown, and Guang H Yue. 2002. A dynamical model of muscle activation, fatigue, and recovery. Biophysical journal 82, 5 (2002), 2344--2359.

[30]

Zhe Liu, Daniel Vogel, and James R Wallace. 2018. Applying the cumulative fatigue model to interaction on large, multi-touch displays. In Proceedings of the 7th ACM International Symposium on Pervasive Displays. 1--9.

Digital Library

[31]

Benjamin Long, Sue Ann Seah, Tom Carter, and Sriram Subramanian. 2014. Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Transactions on Graphics (TOG) 33, 6 (2014), 1--10.

Digital Library

[32]

John M Looft and Laura A Frey-Law. 2020. Adapting a fatigue model for shoulder flexion fatigue: Enhancing recovery rate during intermittent rest intervals. Journal of biomechanics 106 (2020), 109762.

[33]

John M Looft, Nicole Herkert, and Laura Frey-Law. 2018. Modification of a three-compartment muscle fatigue model to predict peak torque decline during intermittent tasks. Journal of biomechanics 77 (2018), 16--25.

[34]

Liang Ma, Damien Chablat, Fouad Bennis, and Wei Zhang. 2009. A new simple dynamic muscle fatigue model and its validation. International journal of industrial ergonomics 39, 1 (2009), 211--220.

[35]

Liang Ma, Damien Chablat, Fouad Bennis, Wei Zhang, and François Guillaume. 2010. A new muscle fatigue and recovery model and its ergonomics application in human simulation. Virtual and Physical Prototyping 5, 3 (2010), 123--137.

[36]

Ganesh R Naik. 2014. Applications, challenges, and advancements in electromyography signal processing. IGI Global.

[37]

Donald A. Norman. 1993. Things That Make Us Smart: Defending Human Attributes in the Age of the Machine. Addison-Wesley Longman Publishing Co., Inc., USA.

Digital Library

[38]

Antti Oulasvirta, Niraj Ramesh Dayama, Morteza Shiripour, Maximilian John, and Andreas Karrenbauer. 2020. Combinatorial optimization of graphical user interface designs. Proc. IEEE 108, 3 (2020), 434--464.

[39]

DM Pincivero, MK Timmons, and D Elsing. 2010. RPE angle effects in young and middle-aged adults. International journal of sports medicine (2010), 257--260.

[40]

Pierre Plantard, Edouard Auvinet, Anne-Sophie Le Pierres, and Franck Multon. 2015. Pose estimation with a kinect for ergonomic studies: Evaluation of the accuracy using a virtual mannequin. Sensors 15, 1 (2015), 1785--1803.

[41]

Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa. 1996. The go-go interaction technique: non-linear mapping for direct manipulation in VR. In Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology (Seattle, Washington, USA) (UIST '96). Association for Computing Machinery, New York, NY, USA, 79--80.

Digital Library

[42]

Inmaculada Rodriguez, R Boulic, and D Meziat. 2002. A joint-level model of fatigue for the postural control of virtual humans. CONF (2002).

[43]

Harrison Jesse Smith and Michael Neff. 2018. Communication Behavior in Embodied Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--12.

Digital Library

[44]

Benjamin Tag, Andrew W. Vargo, Aman Gupta, George Chernyshov, Kai Kunze, and Tilman Dingler. 2019. Continuous alertness assessments: Using EOG glasses to unobtrusively monitor fatigue levels in-the-wild. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA.

Digital Library

[45]

Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM Trans. Graph. 35, 4, Article 143 (jul 2016), 12 pages.

Digital Library

[46]

Ana Villanueva, Sujin Jang, Wolfgang Stuerzlinger, Satyajit Ambike, and Karthik Ramani. 2023. Advanced modeling method for quantifying cumulative subjective fatigue in mid-air interaction. International Journal of Human-Computer Studies 169 (2023), 102931.

Digital Library

[47]

Junhong Wang, Shaoming Sun, and Yining Sun. 2021. A muscle fatigue classification model based on LSTM and improved wavelet packet threshold. Sensors 21, 19 (2021), 6369.

[48]

James Wickham, Tania Pizzari, Katie Stansfeld, Amanda Burnside, and Lyn Watson. 2010. Quantifying 'normal'shoulder muscle activity during abduction. Journal of Electromyography and Kinesiology 20, 2 (2010), 212--222.

[49]

Ting Xia and Laura A Frey Law. 2008. A theoretical approach for modeling peripheral muscle fatigue and recovery. Journal of biomechanics 41, 14 (2008), 3046--3052.

[50]

Hayder A Yousif, Ammar Zakaria, Norasmadi Abdul Rahim, Ahmad Faizal Bin Salleh, Mustafa Mahmood, Khudhur A Alfarhan, Latifah Munirah Kamarudin, Syed Muhammad Mamduh, Ali Majid Hasan, and Moaid K Hussain. 2019. Assessment of muscles fatigue based on surface EMG signals using machine learning and statistical approaches: A review. In IOP conference series: materials science and engineering, Vol. 705. IOP Publishing, 012010.

[51]

Difeng Yu, Hai-Ning Liang, Xueshi Lu, Kaixuan Fan, and Barrett Ens. 2019. Modeling endpoint distribution of pointing selection tasks in virtual reality environments. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.

Digital Library

[52]

Yunxiang Zhang, Kenneth Chen, and Qi Sun. 2023. Toward Optimized VR/AR Ergonomics: Modeling and Predicting User Neck Muscle Contraction. In ACM SIGGRAPH 2023 Conference Proceedings. 1--12.

Cited By

Li YXu HKitamura YTag BFujita KKostakos VKay JHoang T(2024)Towards using Eye Gaze Redirection in Immersive Reading Tasks for Visual Fatigue ReductionCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678474(607-611)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678474

Index Terms

NICER: A New and Improved Consumed Endurance and Recovery Metric to Quantify Muscle Fatigue of Mid-Air Interactions
1. Human-centered computing
  1. Interaction design
    1. Systems and tools for interaction design

Recommendations

Revisiting Consumed Endurance: A NICE Way to Quantify Shoulder Fatigue in Virtual Reality
VRST '23: Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology

Virtual Reality (VR) is increasingly being adopted in fitness, gaming, and workplace productivity applications for its natural interaction with body movement. A widely accepted method for quantifying the physical fatigue caused by VR interactions is ...
Consumed endurance: a metric to quantify arm fatigue of mid-air interactions
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Mid-air interactions are prone to fatigue and lead to a feeling of heaviness in the upper limbs, a condition casually termed as the gorilla-arm effect. Designers have often associated limitations of their mid-air interactions with arm fatigue, but do ...
The consumed endurance workbench: a tool to assess arm fatigue during mid-air interactions
DIS Companion '14: Proceedings of the 2014 companion publication on Designing interactive systems

Consumed Endurance (CE) [8] is a metric that captures the degree of arm fatigue during mid-air interactions. Research has shown that CE can assist with the design of new and minimally fatiguing gestural interfaces. We introduce the Consumed Endurance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 43, Issue 4

July 2024

1774 pages

EISSN:1557-7368

DOI:10.1145/3675116

Issue’s Table of Contents

Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2024

Published in TOG Volume 43, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
330
Total Downloads

Downloads (Last 12 months)330
Downloads (Last 6 weeks)122

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li YXu HKitamura YTag BFujita KKostakos VKay JHoang T(2024)Towards using Eye Gaze Redirection in Immersive Reading Tasks for Visual Fatigue ReductionCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678474(607-611)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678474

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents