article

Gesture Formalization for Multitouch

Authors:

R. GrohAuthors Info & Claims

Software—Practice & Experience, Volume 45, Issue 4

Pages 527 - 548

https://doi.org/10.1002/spe.2247

Published: 01 April 2015 Publication History

Abstract

Since multitouch interfaces are steadily moving out of the experimental niche, the demand for comprehensive programming support increases. Between the formation of de facto standards for multitouch gestures by the industry, and continued research and experimentation by developers or academia, the abstraction of gesture properties becomes essential. This paper contributes a revision of the Gesture Formalization for Multitouch GeForMT gesture description language. This language aims at a short and concise syntax, which is still able to capture the complexity of multitouch gestures. We report on a previously missing reference architecture and practical implementation. Based on a study comparing the revised GeForMT language with other formalization approaches for multitouch gestures, we introduce an editor for GeForMT.Copyright © 2013 John Wiley & Sons, Ltd.

References

[1]

Norman DA, Nielsen J. Gestural interfaces: a step backward in usability. Interactions 2010; Volume 17 Issue 5: pp.46. Available at: "http://portal.acm.org/citation.cfm?doid=1836216.1836228" [last accessed 22 November 2013].

[2]

Lee JC. In search of a natural gesture. XRDS 2010; Volume 16: pp.912. Available at: "http://doi.acm.org/10.1145/1764848.1764853" [last accessed 22 November 2013].

[3]

Kammer D, Wojdziak J, Keck M, Groh R, Taranko S. Towards a formalization of multi-touch gestures. In ACM International Conference on Interactive Tabletops and Surfaces, ITS '10. ACM: New York, NY, USA, 2010; pp.4958. Available at: "http://doi.acm.org/10.1145/1936652.1936662" [last accessed 22 November 2013].

Digital Library

[4]

Görg MT, Cebulla M, Garzon SR. A framework for abstract representation and recognition of gestures in multi-touch applications. In 2010 Third International Conference on Advances in Computer-Human Interactions, St Maarten, Netherlands Antilles, 2010; pp.143-147. Available at: "http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5430107" [last accessed 22 November 2013].

[5]

Scholliers C, Hoste L, Signer B, DeMeuter W. Midas: A declarative multi-touch interaction framework. In Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '11, Funchal, Portugal, 2011; pp.49. Available at: "http://portal.acm.org/citation.cfm?doid=1935701.1935712" [last accessed 22 November 2013].

Digital Library

[6]

Gabbay DM. Labelled Deductive Systems Vol. Volume 1. Clarendon: Oxford, 1996.

[7]

Kin K, Hartmann B, DeRose T, Agrawala M. Proton: multitouch gestures as regular expressions. In Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems, ACM: New York, NY, USA, 2012; pp.2885-2894. Available at: "http://doi.acm.org/10.1145/2208636.2208694" [last accessed 22 November 2013].

[8]

Khandkar SH, Sohan SM, Sillito J, Maurer F. Tool support for testing complex multi-touch gestures. In ACM International Conference on Interactive Tabletops and Surfaces - ITS '10, Saarbrücken, Germany, 2010; pp.59. Available at: "http://portal.acm.org/citation.cfm?doid=1936652.1936663" [last accessed 22 November 2013].

Digital Library

[9]

Echtler F, Klinker G. A multitouch software architecture. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction: Building Bridges, NordiCHI '08, ACM: New York, NY, USA, 2008; pp.463-466. Available at: "http://doi.acm.org/10.1145/1463160.1463220" [last accessed 22 November 2013].

Digital Library

[10]

Echtler F, Butz A. GISpL: gestures made easy. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction, TEI '12, 2012; pp.233-240. Available at: "http://doi.acm.org/10.1145/2148131.2148181" [last accessed 22 November 2013].

[11]

NUIGroup Authors. Gesture recognition - NUI group community wiki. 2009. Available at: "http://wiki.nuigroup.com/Gesture_Recognition".

[12]

Lü H, Li Y. Gesture coder: a tool for programming multi-touch gestures by demonstration. In Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems, CHI '12, ACM: New York, NY, USA, 2012; pp.2875-2884. Available at: "http://doi.acm.org/10.1145/2208636.2208693" [last accessed 22 November 2013].

[13]

Kin K, Hartmann B, DeRose T, Agrawala M. Proton++: a customizable declarative multitouch framework. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST '12, ACM: New York, NY, USA, 2012; pp.477-486. Available at: "http://doi.acm.org/10.1145/2380116.2380176" [last accessed 22 November 2013].

[14]

Lao S, Heng X, Zhang G, Ling Y, Wang P. A gestural interaction design model for multi-touch displays. In Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology, BCS-HCI '09, British Computer Society: Swinton, UK, 2009; pp.440-446. Available at: "http://portal.acm.org/citation.cfm?id=1671011.1671068" [last accessed 22 November 2013].

[15]

Spano LD, Cisternino A, Patern F. A compositional model for gesture definition. In Human-centered software engineering, Winckler M, Forbrig P, Bernhaupt R eds. Springer: Berlin Heidelberg, 2012. Available at: "http://link.springer.com/chapter/10.1007/978-3-642-34347-6_3" [last accessed 22 November 2013].

[16]

Patern F, Santoro C, Spano LD. MARIA: a universal, declarative, multiple abstraction-level language for service-oriented applications in ubiquitous environments. ACM Transactions on Computer-Human Interaction 2009; Volume 16 Issue 4: pp.1-30. Available at: "http://portal.acm.org/citation.cfm?doid=1614390.1614394" [last accessed 22 November 2013].

[17]

Vanacken D. Touch-based interaction and collaboration in walk-up-and-use and multi-user environments. Ph.D. Thesis, 2012. Available at: "http://hdl.handle.net/1942/14432" [last accessed 22 November 2013].

[18]

Microsoft. Application gestures and semantic behavior, 2012. Available at: "http://msdn.microsoft.com/en-us/library/windows/desktop/ms704830v=vs.85.aspx" [last accessed 22 November 2013].

[19]

Lee R. Gesturecons - icons for multi-touch interfaces, 2012. Available at: "http://gesturecons.com/" [last accessed 22 November 2013].

[20]

Brade M, Kammer D, Keck M, Groh R. Immersive data grasping using the explore table. In Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction, TEI '11, ACM: New York, NY, USA, 2011; pp.419-420. Available at: "http://doi.acm.org/10.1145/1935701.1935809" [last accessed 22 November 2013].

Digital Library

[21]

Laufs U, Ruff C, Weisbecker A. MT4j an open source platform for multi-touch software development. VIMation Journal 2010; Volume 1: pp.58-64. Available at: "http://arxiv.org/abs/1012.0467" [last accessed 22 November 2013].

[22]

Wobbrock JO, Wilson AD, Li Y. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST '07, ACM: New York, NY, USA, 2007; pp.159-168. Available at: "http://doi.acm.org/10.1145/1294211.1294238" [last accessed 22 November 2013].

[23]

Appert C, Zhai S. Using strokes as command shortcuts. In Proceedings of the 27th International Conference on Human Factors in Computing Systems - CHI'09, Boston, MA, USA, 2009; pp.2289. Available at: "http://portal.acm.org/citation.cfm?doid=1518701.1519052" [last accessed 22 November 2013].

Digital Library

[24]

Freeman D, Benko H, Morris MR, Wigdor D. ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces - ITS '09, Banff, Alberta, Canada, 2009; pp.165. Available at: "http://portal.acm.org/citation.cfm?doid=1731903.1731935" [last accessed 22 November 2013].

Digital Library

[25]

Blickenstorfer CH. Graffiti: Wow!. Pen Computing Magazine 1995; pp.30-31.

[26]

Franke IS, Kammer D, Groh R, Happ S, Steinhauf J, Schönefeld F. Developing multi-touch software through creative destruction. In Mensch & Computer 2010 Entertainment Interfaces Track, Duisburg, Germany, 2010; pp.45-51. Available at: "http://CEUR-WS.org/Vol-634/Entertainment-Interfaces-Proceedings04.pdf" [last accessed 22 November 2013].

[27]

Groh R, Kammer D, Franke I. Students as catalyst of the knowledge economy: evolution of synergies between companies and universities. In Proceedings of the 6th Annual European Computer Science Summit, ECSS 2010, Prague, 2010.

[28]

Kammer D, Franke IS, Groh R, Steinhauf J, Kirchner M, Schönefeld F. The eleventh finger: levels of manipulation in multi-touch interaction. In Proceedings of the 29th Annual European Conference on Cognitive Ergonomics, ECCE '11, ACM: New York, NY, USA, 2011; pp.141-148. Available at: "http://doi.acm.org/10.1145/2074712.2074741" [last accessed 22 November 2013].

[29]

Müller M, Franke IS, Kammer D, Groh R. Business oriented design of collaborative applications. In Workshop designing collaborative interactive spaces DCIS 2012 at AVI 2012, HCI Group, Univ. of Konstanz, 2012.

[30]

Keck S, Kammer D, Koalick G, Iwan R, Taranko S, Wuttig D. DelViz - deep exploration and lookup of visualizations, 2011.

[31]

Parr T, Fisher K. LL*: the foundation of the ANTLR parser generator. SIGPLAN Notices 2011; Volume 46 Issue 6: pp.425-436. Available at: "http://doi.acm.org/10.1145/2345156.1993548" [last accessed 22 November 2013].

[32]

Kaltenbrunner M, Bovermann T, Bencina R, Costanza E. TUIO: a protocol for table-top tangible user interfaces. In Proceedings of the 6th International Workshop on Gesture in Human-Computer Interaction and Simulation GW 2005, Vannes, France, 2005.

[33]

Wigdor D, Williams S, Cronin M, Levy R, White K, Mazeev M, Benko H. Ripples: utilizing per-contact visualizations to improve user interaction with touch displays. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology - UIST '09, Victoria, BC, Canada, 2009; pp.3. Available at: "http://portal.acm.org/citation.cfm?doid=1622176.1622180" [last accessed 22 November 2013].

[34]

Bau O, Mackay WE. OctoPocus: a dynamic guide for learning gesture-based command sets. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST '08, ACM: New York, NY, USA, 2008; pp.37-46.

[35]

Cederberg P. Grammatica: Parser generator, 2010. Available at: "http://grammatica.percederberg.net/" [last accessed 22 November 2013].

[36]

Signer B, Kurmann U, Norrie M. iGesture: a general gesture recognition framework. In International Conference on Document Analysis and Recognition, Vol.Volume 2, Parana, 2007; pp.954-958.

[37]

Sezgin TM, Stahovich T, Davis R. Sketch based interfaces. In Proceedings of the 2001 Workshop on Perceptive User Interfaces - PUI '01, Orlando, Florida, 2001; pp.1. Available at: "http://portal.acm.org/citation.cfm?doid=971478.971487" [last accessed 22 November 2013].

Digital Library

[38]

Swigart S. Easily write custom gesture recognizers for your tablet PC applications, 2005. Available at: "http://sourceforge.net/projects/siger/" [last accessed 22 November 2013].

[39]

Kammer D, Henkens D, Groh R. GeForMTjs: a JavaScript library based on a domain specific language for multi-touch gestures. In Web Engineering - 12th International Conference, ICWE 2012, Berlin, Germany, July 23-27, 2012; pp.444-447.

[40]

Majda D. PEG.js parser generator for JavaScript, 2011. Available at: "http://pegjs.majda.cz/" [last accessed 22 November 2013].

[41]

Ford B. Parsing expression grammars. In Proceedings of the 31st ACM SIGPLAN-SIGNET Symposium on Principles of Programming Languages, ACM Press, 2004; pp.111-122. Available at: "http://portal.acm.org/citation.cfm?doid=964001.964011" [last accessed 22 November 2013].

[42]

Sizzle. Sizzle JavaScript selector library, 2012. Available at: "http://sizzlejs.com/" [last accessed 22 November 2013].

[43]

Anthony L, Wobbrock JO. A lightweight multistroke recognizer for user interface prototypes. In Proceedings of Graphics Interface 2010, GI '10, Canadian Information Processing Society: Toronto, Ontario, Canada, 2010; pp.245-252. Available at: "http://dl.acm.org/citation.cfm?id=1839214.1839258" [last accessed 22 November 2013].

[44]

Li Y. Protractor. ACM Press: New York, 2010. Available at: "http://portal.acm.org/citation.cfm?doid=1753326.1753654" [last accessed 22 November 2013].

[45]

Wu Y, Hernandez F, Ortega F, Clarke PJ, France R. Measuring the effort for creating and using domain-specific models. In Proceedings of the 10th Workshop on Domain-Specific Modeling, DSM '10, ACM: New York, NY, USA, 2010; pp.14:1-14:6.

[46]

Li Y. Gesture search: a tool for fast mobile data access. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, UIST '10, ACM: New York, NY, USA, 2010; pp.87-96. Available at: "http://doi.acm.org/10.1145/1866029.1866044" [last accessed 22 November 2013].

[47]

Dietz P, Leigh D. DiamondTouch: a multi-user touch technology. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, UIST '01, ACM: New York, NY, USA, 2001; pp.219-226. Available at: "http://doi.acm.org/10.1145/502348.502389" [last accessed 22 November 2013].

[48]

Morris MR, Huang A, Paepcke A, Winograd T. Cooperative gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '06, Montreal, Quebec, Canada, 2006; pp.1201. Available at: "http://portal.acm.org/citation.cfm?doid=1124772.1124952" [last accessed 22 November 2013].

Digital Library

[49]

Walther-Franks B, Schwarten L, Teichert J, Krause M, Herrlich M. User detection for a multi-touch table via proximity sensors. IEEE Tabletops and Interactive Surfaces 2008, IEEE Computer Society: Los Alamitos, CA, USA, 2008.

[50]

Ramakers R, Vanacken D, Luyten K, Coninx K, Schning J. Carpus: a non-intrusive user identification technique for interactive surfaces. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST '12, 2012; pp.35-44. Available at: "http://doi.acm.org/10.1145/2380116.2380123" [last accessed 22 November 2013].

Cited By

Bachert MHesenius M(2024)Towards a Framework for Evaluating Synthetic Surface GesturesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661327(22-30)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661327
Chaffangeon Caillet AGoguey ANigay L(2023)µGlyph: a Microgesture NotationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580693(1-28)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580693
Börsting IHeikamp MHesenius MKoop WGruhn V(2022)Software Engineering for Augmented Reality - A Research AgendaProceedings of the ACM on Human-Computer Interaction10.1145/35322056:EICS(1-34)Online publication date: 17-Jun-2022
https://dl.acm.org/doi/10.1145/3532205
Show More Cited By

Recommendations

Towards a formalization of multi-touch gestures
ITS '10: ACM International Conference on Interactive Tabletops and Surfaces

Multi-touch is a technology which offers new styles of interaction compared to traditional input devices like keyboard and mouse. Users can quickly manipulate objects or execute commands by means of their fingers and hands. Current multi-touch ...
Comparing elicited gestures to designer-created gestures for selection above a multitouch surface
ITS '12: Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces

Many new technologies are emerging that make it possible to extend interaction into the three-dimensional space directly above or in front of a multitouch surface. Such techniques allow people to control these devices by performing hand gestures in the ...
Extended multitouch: recovering touch posture and differentiating users using a depth camera
UIST '12: Proceedings of the 25th annual ACM symposium on User interface software and technology

Multitouch surfaces are becoming prevalent, but most existing technologies are only capable of detecting the user's actual points of contact on the surface and not the identity, posture, and handedness of the user. In this paper, we define the concept ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Software

Software Volume 45, Issue 4

April 2015

143 pages

ISSN:0038-0644

EISSN:1097-024X

Issue’s Table of Contents

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 01 April 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bachert MHesenius M(2024)Towards a Framework for Evaluating Synthetic Surface GesturesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661327(22-30)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661327
Chaffangeon Caillet AGoguey ANigay L(2023)µGlyph: a Microgesture NotationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580693(1-28)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580693
Börsting IHeikamp MHesenius MKoop WGruhn V(2022)Software Engineering for Augmented Reality - A Research AgendaProceedings of the ACM on Human-Computer Interaction10.1145/35322056:EICS(1-34)Online publication date: 17-Jun-2022
https://dl.acm.org/doi/10.1145/3532205
Hesenius MGruhn V(2019)GestureCardsProceedings of the ACM on Human-Computer Interaction10.1145/33311643:EICS(1-35)Online publication date: 13-Jun-2019
https://dl.acm.org/doi/10.1145/3331164
Hesenius MSternal SGruhn VCampos JNunes N(2017)A multi-touch-recognizer for gesturecardsProceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3102113.3102132(75-80)Online publication date: 26-Jun-2017
https://dl.acm.org/doi/10.1145/3102113.3102132
Vivian R(2017)Propositions for a Mid-Air Interactions System Using Leap-Motion for a Collaborative Omnidirectional Immersive EnvironmentDistributed, Ambient and Pervasive Interactions10.1007/978-3-319-58697-7_15(203-216)Online publication date: 9-Jul-2017
https://dl.acm.org/doi/10.1007/978-3-319-58697-7_15
Hesenius MGruhn VBjörk SEriksson EFjeld MBødker SBarendregt WObaid M(2016)Introducing GestureCardsProceedings of the 9th Nordic Conference on Human-Computer Interaction10.1145/2971485.2996746(1-6)Online publication date: 23-Oct-2016
https://dl.acm.org/doi/10.1145/2971485.2996746

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents