Article

Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes

Authors:

Jacob O. Wobbrock,

Andrew D. Wilson,

Yang LiAuthors Info & Claims

UIST '07: Proceedings of the 20th annual ACM symposium on User interface software and technology

Pages 159 - 168

https://doi.org/10.1145/1294211.1294238

Published: 07 October 2007 Publication History

Abstract

Although mobile, tablet, large display, and tabletop computers increasingly present opportunities for using pen, finger, and wand gestures in user interfaces, implementing gesture recognition largely has been the privilege of pattern matching experts, not user interface prototypers. Although some user interface libraries and toolkits offer gesture recognizers, such infrastructure is often unavailable in design-oriented environments like Flash, scripting environments like JavaScript, or brand new off-desktop prototyping environments. To enable novice programmers to incorporate gestures into their UI prototypes, we present a "$1 recognizer" that is easy, cheap, and usable almost anywhere in about 100 lines of code. In a study comparing our $1 recognizer, Dynamic Time Warping, and the Rubine classifier on user-supplied gestures, we found that $1 obtains over 97% accuracy with only 1 loaded template and 99% accuracy with 3+ loaded templates. These results were nearly identical to DTW and superior to Rubine. In addition, we found that medium-speed gestures, in which users balanced speed and accuracy, were recognized better than slow or fast gestures for all three recognizers. We also discuss the effect that the number of templates or training examples has on recognition, the score falloff along recognizers' N-best lists, and results for individual gestures. We include detailed pseudocode of the $1 recognizer to aid development, inspection, extension, and testing.

References

[1]

Anderson, D., Bailey, C. and Skubic, M. (2004) Hidden Markov Model symbol recognition for sketch-based interfaces. AAAI Fall Symposium. Menlo Park, CA: AAAI Press, 15--21.

[2]

Cao, X. and Balakrishnan, R. (2003) VisionWand: Interaction techniques for large displays using a passive wand tracked in 3D. Proc. UIST '03. New York: ACM Press, 173--182.

Digital Library

[3]

Cao, X. and Balakrishnan, R. (2005) Evaluation of an on-line adaptive gesture interface with command prediction. Proc. Graphics Interface '05. Waterloo, Ontario: CHCCS, 187--194.

Digital Library

[4]

Cho, M.G. (2006) A new gesture recognition algorithm and segmentation method of Korean scripts for gesture-allowed ink editor. Information Sciences 176 (9), 1290--1303.

Digital Library

[5]

Guimbretière, F., Stone, M. and Winograd, T. (2001) Fluid interaction with high-resolution wall-size displays. Proc. UIST '01. New York: ACM Press, 21--30.

Digital Library

[6]

Henry, T.R., Hudson, S.E. and Newell, G.L. (1990) Integrating gesture and snapping into a user interface toolkit. Proc. UIST '90. New York: ACM Press, 112--122.

Digital Library

[7]

Hinckley, K., Ramos, G., Guimbretiere, F., Baudisch, P. and Smith, M. (2004) Stitching: Pen gestures that span multiple displays. Proc. AVI '04. New York: ACM Press, 23--31.

Digital Library

[8]

Hong, J.I. and Landay, J.A. (2000) SATIN: A toolkit for informal ink-based applications. Proc. UIST '00. New York: ACM Press, 63--72.

Digital Library

[9]

Kara, L.B. and Stahovich, T.F. (2004) An image-based trainable symbol recognizer for sketch-based interfaces. AAAI Fall Symposium. Menlo Park, CA: AAAI Press, 99--105.

[10]

Karlson, A.K., Bederson, B.B. and SanGiovanni, J. (2005) AppLens and LaunchTile: Two designs for one-handed thumb use on small devices. Proc. CHI '05. New York: ACM Press, 201--210.

Digital Library

[11]

Kristensson, P. and Zhai, S. (2004) SHARK2: A large vocabulary shorthand writing system for pen-based computers. Proc. UIST '04. New York: ACM Press, 43--52.

Digital Library

[12]

Landay, J. and Myers, B.A. (1993) Extending an existing user interface toolkit to support gesture recognition. Adjunct Proc. CHI '93. New York: ACM Press, 91--92.

Digital Library

[13]

Lin, J., Newman, M.W., Hong, J.I. and Landay, J.A. (2000) DENIM: Finding a tighter fit between tools and practice for web site design. Proc. CHI '00. New York: ACM Press, 510--517.

Digital Library

[14]

Long, A.C., Landay, J.A. and Rowe, L.A. (1999) Implications for a gesture design tool. Proc. CHI '99. New York: ACM Press, 40--47.

Digital Library

[15]

Mitchell, T.M. (1997) Machine Learning. New York: McGraw-Hill.

Digital Library

[16]

Morris, M.R., Huang, A., Paepcke, A. and Winograd, T. (2006) Cooperative gestures: Multi-user gestural interactions for co-located groupware. Proc. CHI '06. New York: ACM Press, 1201--1210.

Digital Library

[17]

Myers, B.A., McDaniel, R.G., Miller, R.C., Ferrency, A.S., Faulring, A., Kyle, B.D., Mickish, A., Klimovitski, A. and Doane, P. (1997) The Amulet environment: New models for effective user interface software development. IEEE Trans. Software Engineering 23 (6), 347--365.

Digital Library

[18]

Myers, C.S. and Rabiner, L.R. (1981) A comparative study of several dynamic time-warping algorithms for connected word recognition. The Bell System Technical J. 60 (7), 1389--1409.

[19]

Notowidigdo, M. and Miller, R.C. (2004) Off-line sketch interpretation. AAAI Fall Symposium. Menlo Park, CA: AAAI Press, 120--126.

[20]

Pittman, J.A. (1991) Recognizing handwritten text. Proc. CHI '91. New York: ACM Press, 271--275.

Digital Library

[21]

Plamondon, R. and Srihari, S.N. (2000) On-line and off-line handwriting recognition: A comprehensive survey. IEEE Trans. Pattern Analysis & Machine Int. 22 (1), 63--84.

Digital Library

[22]

Press, W.H., Teukolsky, S.A., Vetterling, W.T. and Flannery, B.P. (1992) Numerical Recipes in C. Cambridge Univ. Press.

[23]

Rubine, D. (1991) Specifying gestures by example. Proc. SIGGRAPH '91. New York: ACM Press, 329--337.

Digital Library

[24]

Salvador, S. and Chan, P. (2004) FastDTW: Toward accurate dynamic time warping in linear time and space. 3rd Wkshp. on Mining Temporal and Sequential Data, ACM KDD '04. Seattle, Washington (August 22--25, 2004).

[25]

Sezgin, T.M. and Davis, R. (2005) HMM-based efficient sketch recognition. Proc. IUI '05. New York: ACM Press, 281--283.

Digital Library

[26]

Stojmenovi., M., Nayak, A. and Zunic, J. (2006) Measuring linearity of a finite set of points. Proc. CIS '06. Los Alamitos, CA: IEEE Press, 1--6.

[27]

Swigart, S. (2005) Easily write custom gesture recognizers for your Tablet PC applications. Tablet PC Technical Articles.

[28]

Tappert, C.C. (1982) Cursive script recognition by elastic matching. IBM J. of Research & Development 26 (6), 765--771.

[29]

Tappert, C.C., Suen, C.Y. and Wakahara, T. (1990) The state of the art in online handwriting recognition. IEEE Trans. Pattern Analysis & Machine Int. 12 (8), 787--808.

Digital Library

[30]

Vermunt, J.K. (1997) Log-linear Models for Event Histories. Thousand Oaks, CA: Sage Publications.

[31]

Wilson, A.D. and Shafer, S. (2003) XWand: UI for intelligent spaces. Proc. CHI '03. New York: ACM Press, 545--552.

Digital Library

[32]

Zhai, S. and Kristensson, P. (2003) Shorthand writing on stylus keyboard. Proc. CHI '03. New York: ACM Press, 97--104.

Digital Library

Cited By

Bai YWang JChen PGong ZXiong Q(2024)Hand Trajectory Recognition by Radar with a Finite-State Machine and a Bi-LSTMApplied Sciences10.3390/app1415678214:15(6782)Online publication date: 3-Aug-2024
https://doi.org/10.3390/app14156782
Rekik YGuettaf AJamalzadeh MGrisoni L(2024)Comparing Eyes-free Gestures to Gestures Produced in the Presence or Absence of Visual Feedback on Mobile DeviceProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656651(1-5)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656651
Qiwei LLameiro FPatel SIsaula-Reyes CAdar EGilbert ESchoenebeck S(2024)Feminist Interaction Techniques: Social Consent Signals to Deter NCIM ScreenshotsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676380(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676380
Show More Cited By

Index Terms

Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Classification and regression trees
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens
    2. Interactive systems and tools

Recommendations

A lightweight multistroke recognizer for user interface prototypes
GI '10: Proceedings of Graphics Interface 2010

With the expansion of pen- and touch-based computing, new user interface prototypes may incorporate stroke gestures. Many gestures comprise multiple strokes, but building state-of-the-art multistroke gesture recognizers is nontrivial and time-consuming. ...
Gestures à Go Go: Authoring Synthetic Human-Like Stroke Gestures Using the Kinematic Theory of Rapid Movements
Special Issue on Causal Discovery and Inference

Training a high-quality gesture recognizer requires providing a large number of examples to enable good performance on unseen, future data. However, recruiting participants, data collection, and labeling, etc., necessary for achieving this goal are ...
G3: bootstrapping stroke gestures design with synthetic samples and built-in recognizers
MobileHCI '16: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct

Stroke gestures are becoming increasingly important with the ongoing success of touchscreen-capable devices. However, training a high-quality gesture recognizer requires providing a large number of examples to enable good performance on unseen, future ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '07: Proceedings of the 20th annual ACM symposium on User interface software and technology

October 2007

306 pages

ISBN:9781595936790

DOI:10.1145/1294211

General Chairs:
Chia Shen
MERL, USA
,
Robert Jacob
Tufts University, USA
,
Program Chair:
Ravin Balakrishnan
University of Toronto, Canada

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 October 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

UIST07

Sponsor:

UIST07: The 20th Annual ACM Symposium on User Interface Software and Technology

October 7 - 10, 2007

Rhode Island, Newport, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

601
Total Citations
View Citations
5,137
Total Downloads

Downloads (Last 12 months)197
Downloads (Last 6 weeks)47

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bai YWang JChen PGong ZXiong Q(2024)Hand Trajectory Recognition by Radar with a Finite-State Machine and a Bi-LSTMApplied Sciences10.3390/app1415678214:15(6782)Online publication date: 3-Aug-2024
Rekik YGuettaf AJamalzadeh MGrisoni L(2024)Comparing Eyes-free Gestures to Gestures Produced in the Presence or Absence of Visual Feedback on Mobile DeviceProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656651(1-5)Online publication date: 3-Jun-2024
Qiwei LLameiro FPatel SIsaula-Reyes CAdar EGilbert ESchoenebeck S(2024)Feminist Interaction Techniques: Social Consent Signals to Deter NCIM ScreenshotsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676380(1-14)Online publication date: 13-Oct-2024
Shi LCampbell RChi PCirimele MCleron MClimer KFleming CGanti AGervais PGonnet PKarim TMaksai AMelancon CMickle RMusat CNandy PQu XRobishaw DSingh AVenkatesan M(2024)Inkeraction: An Interaction Modality Powered by Ink Recognition and SynthesisProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642498(1-26)Online publication date: 11-May-2024
Pascher MSaad ALiebers JHeger RGerken JSchneegass SGruenefeld UGrollman DBroadbent EJu WSoh HWilliams T(2024)Hands-On Robotics: Enabling Communication Through Direct Gesture ControlCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640635(822-827)Online publication date: 11-Mar-2024
Yuan LLi BWang JQu HZeng W(2024)Generating Virtual Reality Stroke Gesture Data from Out-of-Distribution Desktop Stroke Gesture Data2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00093(732-742)Online publication date: 16-Mar-2024
Sawada NYamamoto TSugiura Y(2024)Converting Tatamis into Touch Sensors by Measuring Capacitance2024 IEEE/SICE International Symposium on System Integration (SII)10.1109/SII58957.2024.10417676(554-558)Online publication date: 8-Jan-2024
Sun LZhang HChen PJiang ZXie XZhou ZLiu XChen X(2024)Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed RealityInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103198183:COnline publication date: 14-Mar-2024
Hosseini MHamidi HKermanshahani S(2024)The impacts of situational visual impairment on usability of touch screensMultimedia Tools and Applications10.1007/s11042-024-18689-983:34(81685-81709)Online publication date: 9-Mar-2024
Myers B(2024)Pick, Click, Flick!undefinedOnline publication date: 14-Mar-2024
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents