research-article

M.Gesture: An Acceleration-Based Gesture Authoring System on Multiple Handheld and Wearable Devices

Authors:

Tek-Jin NamAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 2307 - 2318

https://doi.org/10.1145/2858036.2858358

Published: 07 May 2016 Publication History

Abstract

Gesture-based interaction is still underutilized in the mobile context despite the large amount of attention it has been given. Using accelerometers that are widely available in mobile devices, we developed M.Gesture, a software system that supports accelerometer-based gesture authoring on single or multiple mobile devices. The development was based on a formative study that showed users' preferences for subtle, simple motions and synchronized, multi-device gestures. M.Gesture adopts an acceleration data space and interface components based on mass-spring analogy and combines the strengths of both demonstration-based and declarative approaches. Also, gesture declaration is done by specifying a mass-spring trajectory with planes in the acceleration space. For iterative gesture modification, multi-level feedbacks are provided as well. The results of evaluative studies have shown good usability and higher recognition performance than that of dynamic time warping for simple gesture authoring. Later, we discuss the benefits of applying a physical metaphor and hybrid approach.

Supplementary Material

suppl.mov (pn1600-file3.mp4)

Supplemental video

Download
36.70 MB

MP4 File (p2307-kim.mp4)

Download
193.25 MB

References

[1]

Daniel Ashbrook and Thad Starner. 2010. MAGIC: a motion gesture design tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 10), 2159--2168.

Digital Library

[2]

Mehmet Aydın Baytaş, Yücel Yemez, and Oğuzhan Özcan. 2014. Hotspotizer: end-user authoring of mid-air gestural interactions. In Proceedings of the Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI 14), 677--686.

Digital Library

[3]

Ari Y. Benbasat, and Joseph A. Paradiso. 2002. An inertial measurement framework for gesture recognition and applications. In Gesture and Sign Language in Human-Computer Interaction, Ipke Wachsmuth and Timo Sowa (eds.). Springer, 9--20.

Digital Library

[4]

Liwei Chan, Chi-Hao Hsieh, Yi-Ling Chen, Shuo Yang, Da-Yuan Huang, Rong-Hao Liang, and Bing-Yu Chen. 2015. Cyclops: Wearable and Single-Piece Full-Body Gesture Input Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15). 3001--3009.

Digital Library

[5]

Xiang 'Anthony' Chen, Tovi Grossman, Daniel J. Wigdor, and George Fitzmaurice. 2014. Duet: exploring joint interactions on a smart phone and a smart watch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 159--168.

Digital Library

[6]

Xiang 'Anthony' Chen, Nicolai Marquardt, Anthony Tang, Sebastian Boring, and Saul Greenberg. 2012. Extending a mobile device's interaction space through body-centric interaction. In Proceedings of the international conference on Human-computer interaction with mobile devices and services (MobileHCI '12). 151--160.

Digital Library

[7]

Pei-Yu (Peggy) Chi and Yang Li. 2015. Weave: Scripting Cross-Device Wearable Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15). 3923--3932.

Digital Library

[8]

Andrew Crossan, John Williamson, Stephen Brewster, and Rod Murray-Smith. 2008. Wrist rotation for interaction in mobile contexts. In Proceedings of the international conference on Human computer interaction with mobile devices and services (MobileHCI '08). 435--438.

Digital Library

[9]

Allen Cypher. 1993. Watch what I do: programming by demonstration. MIT Press.

Digital Library

[10]

Tovi Grossman, Xiang Anthony Chen, and George Fitzmaurice. 2015. Typing on Glasses: Adapting Text Entry to Smart Eyewear. In Proceedings of the International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). 144--152.

Digital Library

[11]

Björn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer. 2007. Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). 145--154.

Digital Library

[12]

Lode Hoste, Brecht De Rooms, and Beat Signer. 2013. Declarative Gesture Spotting Using Inferred and Refined Control Points. In Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM 2013).

[13]

Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smartwatch Centric Cross-Device Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15). 1247--1256.

Digital Library

[14]

Edwin L. Hutchins, James D. Hollan, and Donald A. Norman. 1985. Direct manipulation interfaces. Human-Computer Interaction, 1, 4: 311--338.

Digital Library

[15]

Ideum. 2013. CreativeML. Retrieved September, 2015 from http://www.creativeml.org

[16]

Ideum. 2014. Gesture Works. Retrieved September, 2015 from http://gestureworks.com

[17]

Ju-Whan Kim and Tek-Jin Nam. 2013. EventHurdle: supporting designers' exploratory interaction prototyping with gesture-based sensors. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 267--276.

Digital Library

[18]

Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012. Proton: multitouch gestures as regular expressions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). 2885--2894.

Digital Library

[19]

Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012. Proton++: a customizable declarative multitouch framework. In Proceedings of the ACM symposium on User interface software and technology (UIST '12). 477--486.

Digital Library

[20]

Sven Kratz and Michael Rohs. 2010. A $3 gesture recognizer: simple gesture recognition for devices equipped with 3D acceleration sensors. In Proceedings of the international conference on Intelligent user interfaces (IUI '10). 341--344.

Digital Library

[21]

Yang Li. 2010. Protractor: a fast and accurate gesture recognizer. In Proceesdings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). 2169--2172.

Digital Library

[22]

Jiayang Liu, Zhen Wang, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. 2009. uWave: Accelerometer-based personalized gesture recognition and its applications. In Proceedings of IEEE International Conference on Pervasive Computing and Communications (PerCom '09), 1--9.

Digital Library

[23]

Hao Lü and Yang Li. 2012. Gesture coder: a tool for programming multi-touch gestures by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). 2875--2884.

Digital Library

[24]

Hao Lü and Yang Li. 2013. Gesture studio: authoring multi-touch interactions through demonstration and declaration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 257--266.

Digital Library

[25]

Sushmita Mitra and Tinku Acharya. 2007. Gesture recognition: A survey. IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 37, 3: 311--324.

Digital Library

[26]

Miguel A. Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of pre-designed and user-defined gesture sets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 1099--1108.

Digital Library

[27]

Donald A. Norman and Stephen W. Draper. 1986. User Centered System Design; New Perspectives on Human Computer Interaction. L. Erlbaum Assoc. Inc.

Digital Library

[28]

Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. EGesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In Proceedings of the ACM Conference on Embedded Networked Sensor Systems (SenSys '11). 260--273.

Digital Library

[29]

Dean Rubine. 1991. Specifying gestures by example. SIGGRAPH Comput. Graph. 25, 4: 329--337.

Digital Library

[30]

Jaime Ruiz, Yang Li, and Edward Lank. 2011. User defined motion gestures for mobile interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). 197--206.

Digital Library

[31]

Hiroaki Sakoe and Seibi Chiba. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech and Signal Processing, 26, 1: 43--49.

[32]

Lucio Davide Spano, Antonio Cisternino, Fabio Paternò, and Gianni Fenu. 2013. GestIT: a declarative and compositional framework for multi-platform gesture definition. In Proceedings of the SIGCHI symposium on Engineering interactive computing systems (EICS '13). 187--196.

Digital Library

[33]

Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined gestures for surface computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). 1083--1092.

Digital Library

[34]

Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the ACM symposium on User interface software and technology (UIST '07). 159--168.

Digital Library

Cited By

Kim C(2024)Transforming Everyday Objects into IoT Control Interfaces: Design and Evaluation of the 'e-Rings' SystemArchives of Design Research10.15187/adr.2024.11.37.5.2937:5(29-49)Online publication date: 30-Nov-2024
https://doi.org/10.15187/adr.2024.11.37.5.29
Villarreal-Narvaez SSluÿters AVanderdonckt JVatavu R(2023)Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation StudiesACM Computing Surveys10.1145/363645856:5(1-55)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3636458
Sharma ASalchow-Hömmen CMollyn VNittala AHedderich MKoelle MSeel TSteimle J(2023)SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-grained Finger MicrogesturesACM Transactions on Computer-Human Interaction10.1145/356989430:3(1-40)Online publication date: 10-Jun-2023
https://dl.acm.org/doi/10.1145/3569894
Show More Cited By

Index Terms

M.Gesture: An Acceleration-Based Gesture Authoring System on Multiple Handheld and Wearable Devices
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Enabling Hand Gesture Customization on Wrist-Worn Devices
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. To achieve this, we first deployed a large-scale study (N=500+) to collect data and train an ...
SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-grained Finger Microgestures
Gestural interaction with freehands and while grasping an everyday object enables always-available input. To sense such gestures, minimal instrumentation of the user’s hand is desirable. However, the choice of an effective but minimal IMU layout remains ...
Serendipity: Finger Gesture Recognition using an Off-the-Shelf Smartwatch
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Previous work on muscle activity sensing has leveraged specialized sensors such as electromyography and force sensitive resistors. While these sensors show great potential for detecting finger/hand gestures, they require additional hardware that adds to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science, ICT and Future Planning

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
800
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)4

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kim C(2024)Transforming Everyday Objects into IoT Control Interfaces: Design and Evaluation of the 'e-Rings' SystemArchives of Design Research10.15187/adr.2024.11.37.5.2937:5(29-49)Online publication date: 30-Nov-2024
https://doi.org/10.15187/adr.2024.11.37.5.29
Villarreal-Narvaez SSluÿters AVanderdonckt JVatavu R(2023)Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation StudiesACM Computing Surveys10.1145/363645856:5(1-55)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3636458
Sharma ASalchow-Hömmen CMollyn VNittala AHedderich MKoelle MSeel TSteimle J(2023)SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-grained Finger MicrogesturesACM Transactions on Computer-Human Interaction10.1145/356989430:3(1-40)Online publication date: 10-Jun-2023
https://dl.acm.org/doi/10.1145/3569894
Azim MRahman AHeo S(2023)SequenceSense: A Tool for Designing Usable Foot-Based Gestures Using a Sequence-Based Gesture RecognizerInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103035176(103035)Online publication date: Aug-2023
https://doi.org/10.1016/j.ijhcs.2023.103035
Kim CNam T(2022)Exploration on Everyday Objects as an IoT Control InterfaceProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533472(1654-1668)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533472
Sun ZWang SLiu CMa X(2022)Metaphoraction: Support Gesture-based Interaction Design with Metaphorical MeaningsACM Transactions on Computer-Human Interaction10.1145/351189229:5(1-33)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3511892
Wang TQian XHe FHu XHuo KCao YRamani KIqbal SMacLean KChevalier FMueller S(2020)CAPturAR: An Augmented Reality Tool for Authoring Human-Involved Context-Aware ApplicationsProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415815(328-341)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3379337.3415815
Hesenius MGruhn V(2019)GestureCardsProceedings of the ACM on Human-Computer Interaction10.1145/33311643:EICS(1-35)Online publication date: 13-Jun-2019
https://dl.acm.org/doi/10.1145/3331164
Korzetz MKühn RSchlegel T(2019)Turn it, Pour it, Twist itProceedings of the 5th International ACM In-Cooperation HCI and UX Conference10.1145/3328243.3328246(20-23)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1145/3328243.3328246
Pohl HMuresan AHornbæk KBrewster SFitzpatrick GCox AKostakos V(2019)Charting Subtle Interaction in the HCI LiteratureProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300648(1-15)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300648
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten