Model Touch Pointing and Detect Parkinson's Disease via a Mobile Game
Authors: 
Kaiyan Ling, Hang Zhao, 
Xiangmin Fan, Xiaohui Niu, Wenchao Yin, Yue Liu, Cui Wang, 
Xiaojun BiAuthors Info & Claims
Article No.: 74, Pages 1 - 24
Abstract
Touch pointing is one of the primary interaction actions on mobile devices. In this research, we aim to (1) model touch pointing for people with Parkinson's Disease (PD), and (2) detect PD via touch pointing. We created a mobile game called MoleBuster in which a user performs a sequence of pointing actions. Our study with 40 participants shows that PD participants exhibited distinct pointing behavior. PD participants were much slower and had greater variances in movement time (MT), while their error rate was slightly lower than age-matched non-PD participants, indicating PD participants traded speed for accuracy. The nominal width Finger-Fitts law showed greater fitness than Fitts' law, suggesting this model should be adopted in lieu of Fitts' law to guide mobile interface design for PD users. We also proposed a CNN-Transformer-based neural network model to detect PD. Taking touch pointing data and comfort rating of finger movement as input, this model achieved an AUC of 0.97 and sensitivity of 0.95 in leave-one-user-out cross-validation. Overall, our research contributes models that reveal the temporal and spatial characteristics of touch pointing for PD users, and provide a new method (CNN-Transformer model) and a mobile game (MoleBuster) for convenient PD detection.
References
[1]
Ravin Balakrishnan. 2004. "Beating" Fitts' law: Virtual enhancements for pointing facilitation. International Journal of Human-Computer Studies 61 (12 2004), 857--874. https://doi.org/10.1016/j.ijhcs.2004.09.002
[2]
Nikola Banovic, Varun Rao, Abinaya Saravanan, Anind K. Dey, and Jennifer Mankoff. 2017. Quantifying Aversion to Costly Typing Errors in Expert Mobile Text Entry. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). ACM, New York, NY, USA, 4229--4241. https://doi.org/10.1145/3025453.3025695
[3]
Xiaojun Bi, Yang Li, and Shumin Zhai. 2013. FFitts Law: Modeling Finger Touch with Fitts' Law. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). ACM, New York, NY, USA, 1363--1372. https://doi.org/10.1145/2470654.2466180
[4]
Xiaojun Bi and Shumin Zhai. 2016. Predicting Finger-Touch Accuracy Based on the Dual Gaussian Distribution Model. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST '16). ACM, New York, NY, USA, 313--319. https://doi.org/10.1145/2984511.2984546
[5]
Brian M Bot, Christine Suver, Elias Chaibub Neto, Michael Kellen, Arno Klein, Christopher Bare, Megan Doerr, Abhishek Pratap, John Wilbanks, E Dorsey, et al. 2016. The mPower study, Parkinson disease mobile data collected using ResearchKit. Scientific data 3, 1 (2016), 1--9.
[6]
Stuart K. Card, William K. English, and Betty J. Burr. 1978. Evaluation of Mouse, Rate-controlled Isometric Joystick, Step Keys, and Text Keys, for Text Selection on a CRT. Ergonomics 21, 8 (1978), 601--613. https://doi.org/10.1080/00140137808931762
[7]
Olivier Chapuis, Renaud Blanch, and Michel Beaudouin-Lafon. 2007. Fitts' Law in the Wild: A Field Study of Aimed Movements.(Dec. 2007).
[8]
Andy Cockburn, Carl Gutwin, and Saul Greenberg. 2007. A Predictive Model of Menu Performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '07). ACM, New York, NY, USA, 627--636. https://doi.org/10.1145/1240624.1240723
[9]
ERFW Crossman. 1957. The speed and accuracy of simple hand movements. The nature and acquisition of industrial skills (1957).
[10]
Geoff Cumming. 2014. The new statistics: Why and how. Psychological science 25, 1 (2014), 7--29.
[11]
L. M.L. de Lau, P. C.L.M. Giesbergen, M. C. de Rijk, A. Hofman, P. J. Koudstaal, and M. M.B. Breteler. 2004. Incidence of parkinsonism and Parkinson disease in a general population. Neurology 63, 7 (2004), 1240--1244. https://doi.org/10.1212/01.WNL.0000140706.52798.BEarXiv:https://n.neurology.org/content/63/7/1240.full.pdf
[12]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[13]
Rachel Eardley, Anne Roudaut, Steve Gill, and Stephen J Thompson. 2017. Understanding grip shifts: how form factors impact hand movements on mobile phones. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4680--4691.
[14]
Leah Findlater and Lotus Zhang. 2020. Input accessibility: A large dataset and summary analysis of age, motor ability and input performance. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1--6.
[15]
Paul Fitts and James Peterson. 1964. Information capacity of discrete motor responses. Journal of experimental psychology 67 (03 1964), 103--12. https://doi.org/10.1037/h0045689
[16]
Paul M Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. Journal of experimental psychology 47, 6 (1954), 381.
[17]
Paul M Fitts and Barbara K Radford. 1966. Information capacity of discrete motor responses under different cognitive sets. Journal of Experimental psychology 71, 4 (1966), 475.
[18]
Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically Generating User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (Funchal, Madeira, Portugal) (IUI '04). Association for Computing Machinery, New York, NY, USA, 93--100. https://doi.org/10.1145/964442.964461
[19]
Krzysztof Z Gajos, Katharina Reinecke, Mary Donovan, Christopher D Stephen, Albert Y Hung, Jeremy D Schmahmann, and Anoopum S Gupta. 2020. Computer mouse use captures ataxia and parkinsonism, enabling accurate measurement and detection. Movement Disorders 35, 2 (2020), 354--358.
[20]
Krzysztof Z Gajos, Daniel S Weld, and Jacob O Wobbrock. 2010. Automatically generating personalized user interfaces with Supple. Artificial intelligence 174, 12-13 (2010), 910--950.
[21]
Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. 2013. Bayesian data analysis (3rd Edition). Chapman and Hall/CRC.
[22]
Andrew Gelman, Jennifer Hill, and Masanao Yajima. 2012. Why we (usually) don't have to worry about multiple comparisons. Journal of research on educational effectiveness 5, 2 (2012), 189--211.
[23]
Christopher G Goetz, Stanley Fahn, Pablo Martinez-Martin, Werner Poewe, Cristina Sampaio, Glenn T Stebbins, Matthew B Stern, Barbara C Tilley, Richard Dodel, Bruno Dubois, et al. 2007. Movement Disorder Society-sponsored revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS): process, format, and clinimetric testing plan. Movement disorders 22, 1 (2007), 41--47.
[24]
Julien Gori. 2018. Modeling the speed-accuracy tradeoff using the tools of information theory. Ph. D. Dissertation. Université Paris-Saclay (ComUE).
[25]
Julien Gori and Olivier Rioul. 2019. Regression to a Linear Lower Bound With Outliers: An Exponentially Modified Gaussian Noise Model. In 2019 27th European Signal Processing Conference (EUSIPCO). IEEE, A Coruna, Spain, 1--5. https://doi.org/10.23919/EUSIPCO.2019.8902946
[26]
Sophia Gu, Yan Ma, Zhi Li, Xiangmin Fan, Feng Tian, and Xiaojun Bi. 2022. Using Deep Learning to Detect Motor Impairment in Early Parkinson's Disease from Touchscreen Typing. In Proceedings of Graphics Interface 2022 (GI 2022). Canadian Information Processing Society, Montréal, Quebec, 209 - 217. https://doi.org/10.20380/GI2022.21
[27]
Yves Guiard and Michel Beaudouin-Lafon. 2004. Fitts' law 50 years later: applications and contributions from human-computer interaction - Preface. Int. J. Hum.-Comput. Stud. 61 (12 2004), 747--750. https://doi.org/10.1016/j.ijhcs.2004.09.003
[28]
Yves Guiard, Frédéric Bourgeois, Denis Mottet, and Michel Beaudouin-Lafon. 2001. Beyond the 10-bit barrier: Fitts' law in multi-scale electronic worlds. In People and Computers XV---Interaction without Frontiers: Joint Proceedings of HCI 2001 and IHM 2001. Springer, 573--587.
[29]
Margaret M Hoehn, Melvin D Yahr, et al. 1998. Parkinsonism: onset, progression, and mortality. Neurology 50, 2 (1998), 318--318.
[30]
Christian Holz and Patrick Baudisch. 2010. The Generalized Perceived Input Point Model and How to Double Touch Accuracy by Extracting Fingerprints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). ACM, New York, NY, USA, 581--590. https://doi.org/10.1145/1753326.1753413
[31]
Christian Holz and Patrick Baudisch. 2011. Understanding Touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). ACM, New York, NY, USA, 2501--2510. https://doi.org/10.1145/1978942.1979308
[32]
Carl Nikolaus Homann, Klaudia Suppan, Karoline Wenzel, Gavin Giovannoni, Gert Ivanic, Susanne Horner, Erwin Ott, and Hans Peter Hartung. 2000. The bradykinesia akinesia incoordination test (BRAIN TEST©), an objective and user-friendly means to evaluate patients with Parkinsonism. Movement disorders: official journal of the Movement Disorder Society 15, 4 (2000), 641--647.
[33]
Amy Hurst, Scott E Hudson, Jennifer Mankoff, and Shari Trewin. 2013. Distinguishing users by pointing performance in laboratory and real-world tasks. ACM Transactions on Accessible Computing (TACCESS) 5, 2 (2013), 1--27.
[34]
Kaori Ikematsu, Haruna Oshima, Rachel Eardley, and Itiro Siio. 2020. Investigating how smartphone movement is affected by lying down body posture. Proceedings of the ACM on Human-Computer Interaction 4, ISS (2020), 1--17.
[35]
Alvin Jude, Darren Guinness, and G Michael Poor. 2016. Reporting and Visualizing Fitts's Law: Dataset, Tools and Methodologies. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 2519--2525.
[36]
Simeon Keates and Shari Trewin. 2005. Effect of age and Parkinson's disease on cursor positioning using a mouse. In Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility. 68--75.
[37]
Yu-Jung Ko, Hang Zhao, Yoonsang Kim, IV Ramakrishnan, Shumin Zhai, and Xiaojun Bi. 2020. Modeling Two Dimensional Touch Pointing. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 858--868.
[38]
Yu-Jung Ko, Hang Zhao, IV Ramakrishnan, Shumin Zhai, and Xiaojun Bi. 2021. Issues Related to Using Finger-Fitts law to Model One-Dimensional Touch Pointing Tasks. In The Ninth International Symposium of Chinese CHI. 41--49.
[39]
John K Kruschke. 2010. What to believe: Bayesian methods for data analysis. Trends in cognitive sciences 14, 7 (2010), 293--300.
[40]
Anthony E. Lang and Andres M. Lozano. 1998. Parkinson's Disease. New England Journal of Medicine 339, 15 (1998), 1044--1053. https://doi.org/10.1056/NEJM199810083391506 arXiv:https://doi.org/10.1056/NEJM199810083391506 9761807.
[41]
Chae Young Lee, Seong Jun Kang, Sang-Kyoon Hong, Hyeo-Il Ma, Unjoo Lee, and Yun Joong Kim. 2016. A validation study of a smartphone-based finger tapping application for quantitative assessment of bradykinesia in Parkinson's disease. PloS one 11, 7 (2016), e0158852.
[42]
Yuexing Luo and Daniel Vogel. 2014. Crossing-based Selection with Direct Touch Input. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). ACM, New York, NY, USA, 2627--2636. https://doi.org/10.1145/2556288.2557397
[43]
Yan Ma, Shumin Zhai, IV Ramakrishnan, and Xiaojun Bi. 2021. Modeling touch point distribution with rotational dual gaussian model. In The 34th Annual ACM Symposium on User Interface Software and Technology. 1197--1209.
[44]
I. Scott MacKenzie. 1992. Fitts' Law As a Research and Design Tool in Human-computer Interaction. Human-Computer Interaction 7, 1 (March 1992), 91--139. https://doi.org/10.1207/s15327051hci0701_3
[45]
Kanti V Mardia. 1970. Measures of multivariate skewness and kurtosis with applications. Biometrika 57, 3 (1970), 519--530.
[46]
Richard McElreath. 2020. Statistical rethinking: A Bayesian course with examples in R and Stan (2nd Edition). Chapman and Hall/CRC.
[47]
Martez E Mott and Jacob O Wobbrock. 2019. Cluster Touch: Improving touch accuracy on smartphones for people with motor and situational impairments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--14.
[48]
Karin Nieuwenhuizen and Jean-Bernard Martens. 2016. Advanced modeling of selection and steering data: beyond Fitts' law. International Journal of Human-Computer Studies 94 (2016), 35--52.
[49]
Alastair J Noyce, Anna Nagy, Shami Acharya, Shahrzad Hadavi, Jonathan P Bestwick, Julian Fearnley, Andrew J Lees, and Gavin Giovannoni. 2014. Bradykinesia-akinesia incoordination test: validating an online keyboard test of upper limb function. PLoS One 9, 4 (2014), e96260.
[50]
Fernando L Pagan. 2012. Improving outcomes through early diagnosis of Parkinson's disease. American Journal of Managed Care 18, 7 (2012), S176.
[51]
Neil J Perkins and Enrique F Schisterman. 2006. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. American journal of epidemiology 163, 7 (2006), 670--675.
[52]
Katrin Plaumann, Milos Babic, Tobias Drey, Witali Hepting, Daniel Stooss, and Enrico Rukzio. 2018. Improving input accuracy on smartphones for persons who are affected by tremor using motion sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--30.
[53]
Murray Rosenblatt. 1956. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics (1956), 832--837.
[54]
Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika 52, 3/4 (1965), 591--611.
[55]
R William Soukoreff and I Scott MacKenzie. 2004. Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts' law research in HCI. International journal of human-computer studies 61, 6 (2004), 751--789.
[56]
Afroza Sultana and Karyn Moffatt. 2019. Effects of aging on small target selection with touch input. ACM Transactions on Accessible Computing (TACCESS) 12, 1 (2019), 1--35.
[57]
Ana Lisa Taylor Tavares, Gregory SXE Jefferis, Mandy Koop, Bruce C Hill, Trevor Hastie, Gary Heit, and Helen M Bronte-Stewart. 2005. Quantitative measurements of alternating finger tapping in Parkinson's disease correlate with UPDRS motor disability and reveal the improvement in fine motor control from medication and deep brain stimulation. Movement disorders: official journal of the Movement Disorder Society 20, 10 (2005), 1286--1298.
[58]
Feng Tian, Xiangmin Fan, Junjun Fan, Yicheng Zhu, Jing Gao, Dakuo Wang, Xiaojun Bi, and Hongan Wang. 2019. What Can Gestures Tell? Detecting Motor Impairment in Early Parkinson's from Common Touch Gestural Interactions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3290605.3300313
[59]
Shari Trewin and Helen Pain. 1997. Dynamic modelling of keyboard skills: Supporting users with motor disabilities. In User Modeling: Proceedings of the Sixth International Conference UM97 Chia Laguna, Sardinia, Italy June 2-5 1997. Springer, 135--146.
[60]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[61]
Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom, and Han LJ Van Der Maas. 2011. Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). (2011).
[62]
Michael Wang, HANG ZHAO, Xiaolei Zhou, Xiangshi Ren, and Xiaojun Bi. 2021. Variance and Distribution Models for Steering Tasks. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 1122--1143. https://doi.org/10.1145/3472749.3474811
[63]
Alan Traviss Welford. 1968. Fundamentals of skill. Methuen, New York, NY, USA.
[64]
Jacob O Wobbrock, Krzysztof Z Gajos, Shaun K Kane, and Gregg C Vanderheiden. 2018. Ability-based design. Commun. ACM 61, 6 (2018), 62--71.
[65]
Jacob O Wobbrock, Kristen Shinohara, and Alex Jansen. 2011. The effects of task dimensionality, endpoint deviation, throughput calculation, and experiment design on pointing measures and models. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1639--1648.
[66]
Shota Yamanaka and Hiroki Usuba. 2020. Rethinking the Dual Gaussian Distribution Model for Predicting Touch Accuracy in Onscreen-start Pointing Tasks. Proceedings of the ACM on Human-Computer Interaction 4, ISS (2020), 1--20.
[67]
Shota Yamanaka and Hiroki Usuba. 2021. Computing Touch-Point Ambiguity on Mobile Touchscreens for Modeling Target Selection Times. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1--21.
[68]
Hong Yang and Travis Desell. 2022. Robust augmentation for multivariate time series classification. arXiv preprint arXiv:2201.11739 (2022).
[69]
Caitao Zhan, Himanshu Gupta, Arani Bhattacharya, and Mohammad Ghaderibaneh. 2020. Efficient localization of multiple intruders in shared spectrum system. In 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 205--216.
[70]
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017).
[71]
Tengxiang Zhang, Xin Yi, Ruolin Wang, Jiayuan Gao, Yuntao Wang, Chun Yu, Simin Li, and Yuanchun Shi. 2019. Facilitating Temporal Synchronous Target Selection through User Behavior Modeling. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1--24.
[72]
Hang Zhao, Sophia Gu, Chun Yu, and Xiaojun Bi. 2022. Bayesian Hierarchical Pointing Models. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST '22). Association for Computing Machinery, New York, NY, USA, Article 87, 13 pages. https://doi.org/10.1145/3526113.3545708
Index Terms
- Model Touch Pointing and Detect Parkinson's Disease via a Mobile Game
Recommendations
Revisiting peephole pointing: a study of target acquisition with a handheld projector
MobileHCI '12: Proceedings of the 14th international conference on Human-computer interaction with mobile devices and servicesPeephole pointing is a promising interaction technique for large workspaces that contain more information than can be appropriately displayed on a single screen. In peephole pointing a window to the virtual workspace is moved in space to reveal ...
Peephole pointing: modeling acquisition of dynamically revealed targets
CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsPeephole interaction occurs when a spatially aware display is moved and acts as a viewport to reveal different parts of the virtual space that cannot all fit within the display at once. We investigate pointing within this peephole metaphor, where the ...
The Effects of Encumbrance and Mobility on Touch-Based Gesture Interactions for Mobile Phones
MobileHCI '15: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and ServicesIn this paper, we investigate the effects of mobility and encumbrance (holding objects such as shopping bags) on standard gestures commonly performed on touchscreens: tapping, dragging, spreading & pinching and rotating clockwise & anticlockwise when ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 May 2024
Published in IMWUT Volume 8, Issue 2
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 194Total Downloads
- Downloads (Last 12 months)194
- Downloads (Last 6 weeks)40
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in