Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3025453.3025941acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Improving Gesture Recognition Accuracy on Touch Screens for Users with Low Vision

Published: 02 May 2017 Publication History

Abstract

We contribute in this work on gesture recognition to improve the accessibility of touch screens for people with low vision. We examine the accuracy of popular recognizers for gestures produced by people with and without visual impairments, and we show that the user-independent accuracy of $P, the best recognizer among those evaluated, is small for people with low vision (83.8%), despite $P being very effective for gestures produced by people without visual impairments (95.9%). By carefully analyzing the gesture articulations produced by people with low vision, we inform key algorithmic revisions for the P recognizer, which we call P+. We show significant accuracy improvements of $P+ for gestures produced by people with low vision, from 83.8% to 94.7% on average and up to 98.2%, and 3x faster execution times compared to P.

Supplementary Material

MP4 File (p4667-vatavu.mp4)

References

[1]
Lisa Anthony, Radu-Daniel Vatavu, and Jacob O. Wobbrock. 2013. Understanding the Consistency of Users' Pen and Finger Stroke Gesture Articulation. In Proc. of Graphics Interface 2013 (GI '13). Canadian Inf. Processing Society, 87--94. http://dl.acm.org/ citation.cfm?id=2532129.2532145
[2]
Lisa Anthony and Jacob O. Wobbrock. 2010. A Lightweight Multistroke Recognizer for User Interface Prototypes. In Proc. of Graphics Interface 2010 (GI '10). Canadian Inf. Processing Society, 245--252. http://dl.acm.org/citation.cfm?id=1839214.1839258
[3]
Lisa Anthony and Jacob O. Wobbrock. 2012. N-protractor: A Fast and Accurate Multistroke Recognizer. In Proc. of Graphics Interface 2012 (GI '12). Canadian Inf. Processing Society, 117--120. http://dl.acm.org/citation.cfm?id=2305276.2305296
[4]
American Optometric Association. 1997a. Optometric Clinical Practice Guideline: Care of the Patient with Hyperopia. (1997). http://www.aoa.org/documents/optometrists/CPG-16.pdf
[5]
American Optometric Association. 1997b. Optometric Clinical Practice Guideline: Care of the Patient with Myopia. (1997). http://www.aoa.org/documents/optometrists/CPG-15.pdf
[6]
American Optometric Association. 1997c. Optometric Clinical Practice Guideline: Care of the Patient with Visual Impairment (Low Vision Rehabilitation). (1997). http://www.aoa.org/documents/optometrists/CPG-14.pdf
[7]
Shiri Azenkot, Kyle Rector, Richard Ladner, and Jacob Wobbrock. 2012. PassChords: Secure Multi-touch Authentication for Blind People. In Proc. of the 14th Int. ACM Conf. on Computers and Accessibility (ASSETS '12). ACM, NY, NY, USA, 159--166.
[8]
Rachel Blagojevic, Samuel Hsiao-Heng Chang, and Beryl Plimmer. 2010. The Power of Automatic Feature Selection: Rubine on Steroids. In Proc. of the Seventh Sketch-Based Interfaces and Modeling Symposium (SBIM '10). Eurographics Association, 79--86. http://dl.acm. org/citation.cfm?id=1923363.1923377
[9]
Anke Brock, Philippe Truillet, Bernard Oriola, and Christophe Jouffrais. 2014. Making Gestural Interaction Accessible to Visually Impaired People. Springer Berlin Heidelberg, Berlin, Heidelberg, 41--48.
[10]
Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Loredana Martusciello. 2011. Making Visual Maps Accessible to the Blind. In Proc. of the 6th Int. Conf. on Universal Access in Human-Computer Interaction: Users Diversity (UAHCI'11). Springer, 271--280. http://dl.acm.org/citation.cfm?id=2027376.2027408
[11]
Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Amaury Trujillo. 2015. Exploring Visually Impaired People's Gesture Preferences for Smartphones. In Proc. of the 11th Conf. on Italian SIGCHI Chapter (CHItaly 2015). ACM, NY, NY, USA, 94--101.
[12]
Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Amaury Trujillo. 2016. Analyzing visually impaired people's touch gestures on smartphones. Multimedia Tools and Applications (2016), 1--29.
[13]
Tim Dittmar, Claudia Krull, and Graham Horton. 2015. A new approach for touch gesture recognition: Conversive Hidden non-Markovian Models. Joural of Computational Science 10 (2015), 66--76.
[14]
Manfredo Do Carmo. 1976. Differential Geometry of Curves and Surfaces. Prentice-Hall, Englewood Cliffs, New Jersey, USA.
[15]
Krzysztof Z. Gajos, Jacob O. Wobbrock, and Daniel S. Weld. 2007. Automatically Generating User Interfaces Adapted to Users' Motor and Vision Capabilities. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, NY, NY, USA, 231--240.
[16]
Poika Isokoski. 2001. Model for Unistroke Writing Time. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '01). ACM, NY, NY, USA, 357--364.
[17]
Shaun K. Kane, Jeffrey P. Bigham, and Jacob O. Wobbrock. 2008. Slide Rule: Making Mobile Touch Screens Accessible to Blind People Using Multi-touch Interaction Techniques. In Proc. of the 10th Int. ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '08). ACM, NY, NY, USA, 73--80.
[18]
Shaun K. Kane, Chandrika Jayant, Jacob O. Wobbrock, and Richard E. Ladner. 2009. Freedom to Roam: A Study of Mobile Device Adoption and Accessibility for People with Visual and Motor Disabilities. In Proc. of the 11th Int. ACM Conf. on Computers and Accessibility (ASSETS '09). ACM, NY, NY, USA, 115--122.
[19]
Shaun K. Kane, Meredith Ringel Morris, and Jacob O. Wobbrock. 2013. Touchplates: Low-cost Tactile Overlays for Visually Impaired Touch Screen Users. In Proc. of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '13). ACM, NY, NY, USA, Article 22, 8 pages.
[20]
Shaun K. Kane, Jacob O. Wobbrock, and Richard E. Ladner. 2011. Usable Gestures for Blind People: Understanding Preference and Performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, NY, NY, USA, 413--422.
[21]
Levent Burak Kara and Thomas F. Stahovich. 2004. Hierarchical Parsing and Recognition of Hand-sketched Diagrams. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, NY, NY, USA, 13--22.
[22]
Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012. Proton: Multitouch Gestures As Regular Expressions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 2885--2894.
[23]
Per-Ola Kristensson and Shumin Zhai. 2004. SHARK2: A Large Vocabulary Shorthand Writing System for Pen-based Computers. In Proc. of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, NY, NY, USA, 43--52.
[24]
Luis A. Leiva, Daniel Martín-Albo, and Réjean Plamondon. 2015. Gestures À Go Go: Authoring Synthetic Human-Like Stroke Gestures Using the Kinematic Theory of Rapid Movements. ACM Trans. Intell. Syst. Technol. 7, 2, Article 15 (Nov. 2015), 29 pages.
[25]
Luis A. Leiva, Daniel Martín-Albo, and Radu-Daniel Vatavu. 2017. Synthesizing Stroke Gestures Across User Populations: A Case for Users with Visual Impairments. In Proc. of the 35th ACM Conf. on Human Factors in Computing Systems (CHI '17). ACM.
[26]
L.A. Levin and D.M. Albert. 2010. Ocular Disease: Mechanisms and Management. Elsevier.
[27]
Yang Li. 2010. Protractor: A Fast and Accurate Gesture Recognizer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, NY, NY, USA, 2169--2172.
[28]
Hao Lü and Yang Li. 2012. Gesture Coder: A Tool for Programming Multi-touch Gestures by Demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 2875--2884.
[29]
Hao Lü and Yang Li. 2013. Gesture Studio: Authoring Multi-touch Interactions Through Demonstration and Declaration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 257--266.
[30]
David McGookin, Stephen Brewster, and WeiWei Jiang. 2008. Investigating Touchscreen Accessibility for People with Visual Impairments. In Proc. of the 5th Nordic Conf. on Human-Computer Interaction (NordiCHI '08). ACM, NY, NY, USA, 298--307.
[31]
Martez E. Mott, Radu-Daniel Vatavu, Shaun K. Kane, and Jacob O. Wobbrock. 2016. Smart Touch: Improving Touch Accuracy for People with Motor Impairments with Template Matching. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 1934--1946.
[32]
C.S. Myers and L.R. Rabiner. 1981. A comparative study of several dynamic time-warping algorithms for connected word recognition. The Bell System Technical Journal 60, 7 (1981), 1389--1409.
[33]
Uran Oh, Stacy Branham, Leah Findlater, and Shaun K. Kane. 2015. Audio-Based Feedback Techniques for Teaching Touchscreen Gestures. ACM Trans. Access. Comput. 7, 3, Article 9 (Nov. 2015), 29 pages.
[34]
World Health Organization. 2014. Visual impairment and blindness. Fact Sheet N282. (2014). http://www.who.int/mediacentre/factsheets/fs282/en/
[35]
World Health Organization. 2016. ICD-10, Visual disturbances and blindness. (2016). http://apps.who.int/classifications/icd10/browse/2016/en#/H53-H54
[36]
C. H. Papadimitriou and K. Steiglitz. 1998. Combinatorial optimization: algorithms and complexity. Dover Publications, Mineola, NY, USA.
[37]
Thanawin Rakthanmanon, Bilson Campana, Abdullah Mueen, Gustavo Batista, Brandon Westover, Qiang Zhu, Jesin Zakaria, and Eamonn Keogh. 2012. Searching and Mining Trillions of Time Series Subsequences Under Dynamic Time Warping. In Proc. of the 18th ACM Int. Conf. on Knowledge Discovery and Data Mining (KDD '12). ACM, NY, NY, USA, 262--270.
[38]
Yosra Rekik, Radu-Daniel Vatavu, and Laurent Grisoni. 2014a. Match-up & Conquer: A Two-Step Technique for Recognizing Unconstrained Bimanual and Multi-finger Touch Input. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces (AVI '14). ACM, NY, NY, USA, 201--208.
[39]
Yosra Rekik, Radu-Daniel Vatavu, and Laurent Grisoni. 2014b. Understanding Users' Perceived Difficulty of Multi-Touch Gesture Articulation. In Proceedings of the 16th International Conference on Multimodal Interaction (ICMI '14). ACM, NY, NY, USA, 232--239.
[40]
Dean Rubine. 1991. Specifying Gestures by Example. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91). ACM, NY, NY, USA, 329--337.
[41]
Tevfik Metin Sezgin and Randall Davis. 2005. HMM-based Efficient Sketch Recognition. In Proc. of the 10th Int. Conf. on Intelligent User Interfaces (IUI '05). ACM, NY, NY, USA, 281--283.
[42]
Kristen Shinohara and Josh Tenenberg. 2007. Observing Sara: A Case Study of a Blind Person's Interactions with Technology. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '07). ACM, NY, NY, USA, 171--178.
[43]
Radu-Daniel Vatavu. 2011. The Effect of Sampling Rate on the Performance of Template-based Gesture Recognizers. In Proc. of the 13th Int. Conf. on Multimodal Interfaces (ICMI '11). ACM, 271--278.
[44]
Radu-Daniel Vatavu. 2012. 1F: One Accessory Feature Design for Gesture Recognizers. In Proc. of the 2012 ACM Int. Conf. on Intelligent User Interfaces (IUI '12). ACM, NY, NY, USA, 297--300.
[45]
Radu-Daniel Vatavu. 2013. The Impact of Motion Dimensionality and Bit Cardinality on the Design of 3D Gesture Recognizers. Int. J. Hum.-Comput. Stud. 71, 4 (April 2013), 387--409.
[46]
Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2012. Gestures As Point Clouds: A $P Recognizer for User Interface Prototypes. In Proc. of the 14th ACM Int. Conference on Multimodal Interaction (ICMI '12). ACM, NY, NY, USA, 273--280.
[47]
Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2013. Relative Accuracy Measures for Stroke Gestures. In Proc. of the 15th ACM Int. Conf. on Multimodal Interaction (ICMI '13). ACM, 279--286.
[48]
Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2014. Gesture Heatmaps: Understanding Gesture Performance with Colorful Visualizations. In Proc. of the 16th Int. Conf. on Multimodal Interaction (ICMI '14). ACM, NY, NY, USA, 172--179.
[49]
Radu-Daniel Vatavu, Gabriel Cramariuc, and Doina Maria Schipor. 2015. Touch Interaction for Children Aged 3 to 6 Years: Experimental Findings and Relationship to Motor Skills. International Journal of Human-Computer Studies 74 (2015), 54--76.
[50]
Radu-Daniel Vatavu, Laurent Grisoni, and Stefan-Gheorghe Pentiuc. 2009. Gesture-Based Human-Computer Interaction and Simulation. Springer-Verlag, Chapter Gesture Recognition Based on Elastic Deformation Energies, 1--12.
[51]
Radu-Daniel Vatavu, Laurent Grisoni, and Stefan-Gheorghe Pentiuc. 2010. Multiscale Detection of Gesture Patterns in Continuous Motion Trajectories. In Proc. of the 8th Int. Conf. on Gesture in Embodied Communication and Human-Computer Interaction (GW'09). Springer-Verlag, 85--97.
[52]
Radu-Daniel Vatavu, Daniel Vogel, Géry Casiez, and Laurent Grisoni. 2011. Estimating the Perceived Difficulty of Pen Gestures. In Proc. of the 13th IFIP TC 13 Int. Conf. on Human-computer Interaction (INTERACT'11). Springer, 89--106. http://dl.acm. org/citation.cfm?id=2042118.2042130
[53]
Radu-Daniel Vatavu and Jacob O. Wobbrock. 2015. Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit. In Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI '15). ACM, 1325--1334.
[54]
Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. In Proc. of the 29th Annual Symp. on User Interface Software and Technology (UIST '16). ACM, 851--860.
[55]
Sy Bor Wang, Ariadna Quattoni, Louis-Philippe Morency, and David Demirdjian. 2006. Hidden Conditional Random Fields for Gesture Recognition. In Proc. of the 2006 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR '06). 1521--1527.
[56]
Don Willems, Ralph Niels, Marcel van Gerven, and Louis Vuurpijl. 2009. Iconic and Multi-stroke Gesture Recognition. Pattern Recognition 42, 12 (2009), 3303--3312.
[57]
Andrew D. Wilson and Aaron F. Bobick. 1999. Parametric Hidden Markov Models for Gesture Recognition. IEEE TPAMI 21, 9 (1999), 884--900.
[58]
Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, and Jon Froehlich. 2011. Ability-Based Design: Concept, Principles and Examples. ACM Trans. Access. Comput. 3, 3, Article 9 (2011), 27 pages.
[59]
Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures Without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proc. of the 20th Annual ACM Symp. on User Interface Software and Technology (UIST '07). ACM, 159--168.

Cited By

View all
  • (2024)Hands-On Robotics: Enabling Communication Through Direct Gesture ControlCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640635(822-827)Online publication date: 11-Mar-2024
  • (2024)Generating Virtual Reality Stroke Gesture Data from Out-of-Distribution Desktop Stroke Gesture Data2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00093(732-742)Online publication date: 16-Mar-2024
  • (2023)Effective 2D Stroke-based Gesture Augmentation for RNNsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581358(1-13)Online publication date: 19-Apr-2023
  • Show More Cited By

Index Terms

  1. Improving Gesture Recognition Accuracy on Touch Screens for Users with Low Vision

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
      May 2017
      7138 pages
      ISBN:9781450346559
      DOI:10.1145/3025453
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 02 May 2017

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. 1
      2. P
      3. P+
      4. algorithms
      5. evaluation
      6. gesture recognition
      7. low vision
      8. point clouds
      9. recognition
      10. recognition accuracy
      11. touch gestures
      12. touch screens
      13. visual impairments

      Qualifiers

      • Research-article

      Funding Sources

      • UEFISCDI

      Conference

      CHI '17
      Sponsor:

      Acceptance Rates

      CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI '25
      CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)58
      • Downloads (Last 6 weeks)5
      Reflects downloads up to 13 Nov 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Hands-On Robotics: Enabling Communication Through Direct Gesture ControlCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640635(822-827)Online publication date: 11-Mar-2024
      • (2024)Generating Virtual Reality Stroke Gesture Data from Out-of-Distribution Desktop Stroke Gesture Data2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00093(732-742)Online publication date: 16-Mar-2024
      • (2023)Effective 2D Stroke-based Gesture Augmentation for RNNsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581358(1-13)Online publication date: 19-Apr-2023
      • (2023)iFAD Gestures: Understanding Users’ Gesture Input Performance with Index-Finger Augmentation DevicesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580928(1-17)Online publication date: 19-Apr-2023
      • (2023)Gesture-Based InteractionHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_20-1(1-47)Online publication date: 9-Feb-2023
      • (2023)Gesture‐Based ComputingHandbook of Human‐Machine Systems10.1002/9781119863663.ch32(397-408)Online publication date: 7-Jul-2023
      • (2022)The Hybrid Stylus: A Multi-Surface Active Stylus for Interacting with and Handwriting on Paper, Tabletop Display or BothSensors10.3390/s2218705822:18(7058)Online publication date: 18-Sep-2022
      • (2022)Theoretically-Defined vs. User-Defined Squeeze GesturesProceedings of the ACM on Human-Computer Interaction10.1145/35678056:ISS(73-102)Online publication date: 14-Nov-2022
      • (2022)QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion ControllerProceedings of the ACM on Human-Computer Interaction10.1145/35322116:EICS(1-47)Online publication date: 17-Jun-2022
      • (2022)µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Multi-stroke Gesture RecognizerProceedings of the ACM on Human-Computer Interaction10.1145/35322006:EICS(1-25)Online publication date: 17-Jun-2022
      • Show More Cited By

      View Options

      Get Access

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media