research-article

CommandBoard: Creating a General-Purpose Command Gesture Input Space for Soft Keyboard

Authors:

Jessalyn Alvina,

Carla F. Griggio,

Wendy E. MackayAuthors Info & Claims

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

Pages 17 - 28

https://doi.org/10.1145/3126594.3126639

Published: 20 October 2017 Publication History

Abstract

CommandBoard offers a simple, efficient and incrementally learnable technique for issuing gesture commands from a soft keyboard. We transform the area above the keyboard into a command-gesture input space that lets users draw unique command gestures or type command names followed by execute. Novices who pause see an in-context dynamic guide, whereas experts simply draw. Our studies show that CommandBoard's inline gesture shortcuts are significantly faster (almost double) than markdown symbols and significantly preferred by users. We demonstrate additional techniques for more complex commands, and discuss trade-offs with respect to the user's knowledge and motor skills, as well as the size and structure of the command space.

Supplementary Material

suppl.mov (uistf3598.mp4)

Supplemental video

Download
36.10 MB

MP4 File (p17-alvina.mp4)

Download
141.46 MB

References

[1]

Jessalyn Alvina, Joseph Malloch, and Wendy E. Mackay. 2016. Expressive Keyboards: Enriching Gesture-Typing on Mobile Devices. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, NY, NY, USA, 583--593.

Digital Library

[2]

Caroline Appert and Shumin Zhai. 2009. Using Strokes As Command Shortcuts: Cognitive Benefits and Toolkit Support. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, NY, NY, USA, 2289--2298.

Digital Library

[3]

Olivier Bau and Wendy E. Mackay. 2008. OctoPocus: A Dynamic Guide for Learning Gesture-based Command Sets. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, NY, NY, USA, 37--46.

Digital Library

[4]

Barry Brown, Moira McGregor, and Donald McMillan. 2014. 100 Days of iPhone Use: Understanding the Details of Mobile Device Use. In Proceedings of the 16th International Conference on Human-computer Interaction with Mobile Devices & Services (MobileHCI '14). ACM, NY, NY, USA, 223--232.

Digital Library

[5]

Daniel Buschek, Alexander De Luca, and Florian Alt. 2015. There is More to Typing Than Speed: Expressive Mobile Touch Keyboards via Dynamic Font Personalisation. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, NY, NY, USA, 125--130.

Digital Library

[6]

Vittorio Fuccella, Poika Isokoski, and Benoit Martin. 2013. Gestures and Widgets: Performance in Text Editing on Multi-touch Capable Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 2785--2794.

Digital Library

[7]

Emilien Ghomi, Stéphane Huot, Olivier Bau, Michel Beaudouin-Lafon, and Wendy E. Mackay. 2013. ArpèGe: Learning Multitouch Chord Gestures Vocabularies. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (ITS '13). ACM, NY, NY, USA, 209--218.

Digital Library

[8]

François Guimbretiére and Terry Winograd. 2000. FlowMenu: Combining Command, Text, and Data Entry. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST '00). ACM, NY, NY, USA, 213--216.

Digital Library

[9]

Sunjun Kim and Geehyuk Lee. 2016. TapBoard 2: Simple and Effective Touchpad-like Interaction on a Multi-Touch Surface Keyboard. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 5163--5168.

Digital Library

[10]

Per-Ola Kristensson and Shumin Zhai. 2004. SHARK2: A Large Vocabulary Shorthand Writing System for Pen-based Computers. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, NY, NY, USA, 43--52.

Digital Library

[11]

Per Ola Kristensson and Shumin Zhai. 2007. Command Strokes with and Without Preview: Using Pen Gestures on Keyboard for Command Selection. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, NY, NY, USA, 1137--1146.

Digital Library

[12]

Gordon Kurtenbach and William Buxton. 1991. Issues in Combining Marking and Direct Manipulation Techniques. In Proceedings of the 4th Annual ACM Symposium on User Interface Software and Technology (UIST '91). ACM, NY, NY, USA, 137--144.

Digital Library

[13]

Gordon Kurtenbach and William Buxton. 1993. The Limits of Expert Performance Using Hierarchic Marking Menus. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (CHI '93). ACM, NY, NY, USA, 482--487.

Digital Library

[14]

David M. Lane, H. Albert Napier, S. Camille Peres, and Aniko Sandor. 2005. Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. Int. J. Hum. Comput. Interaction 18, 2 (2005), 133--144. http://dblp.uni-trier.de/db/journals/ijhci/ijhci18.html#LaneNPS05

[15]

Yang Li. 2010. Gesture Search: A Tool for Fast Mobile Data Access. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, NY, NY, USA, 87--96.

Digital Library

[16]

Wendy E. Mackay. 2002. Which Interaction Technique Works when?: Floating Palettes, Marking Menus and Toolglasses Support Different Task Strategies. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '02). ACM, NY, NY, USA, 203--208.

Digital Library

[17]

Joseph Malloch, Carla F. Griggio, Joanna McGrenere, and Wendy E. Mackay. 2017. fieldward and Pathward: Dynamic Guides for Defining Your Own Gestures. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 4266--4277.

Digital Library

[18]

Miguel A. Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of Pre-designed and User-defined Gesture Sets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 1099--1108.

Digital Library

[19]

Petra Neumann, Annie Tat, Torre Zuk, and Sheelagh Carpendale. 2007. KeyStrokes: Personalizing Typed Text with Visualization. In Proceedings of the 9th Joint Eurographics / IEEE VGTC Conference on Visualization (EUROVIS'07). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 43--50.

Digital Library

[20]

Richard C. Omanson, Craig S. Miller, Elizabeth Young, and David Schwantes. 2010. Comparison of Mouse and Keyboard Efficiency. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 54, 6 (2010), 600--604.

[21]

Robert Pastel. 2006. Measuring the Difficulty of Steering Through Corners. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, NY, NY, USA, 1087--1096.

Digital Library

[22]

Dario D. Salvucci, Niels A. Taatgen, and Jelmer P. Borst. 2009. Toward a Unified Theory of the Multitasking Continuum: From Concurrent Performance to Task Switching, Interruption, and Resumption. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, NY, NY, USA, 1819--1828.

Digital Library

[23]

Christopher D Wickens, Justin G Hollands, Simon Banbury, and Raja Parasuraman. 2015. Engineering psychology & human performance. Psychology Press. 197--244 pages.

[24]

Shumin Zhai and Per Ola Kristensson. 2012. The Word-gesture Keyboard: Reimagining Keyboard Interaction. Commun. ACM 55, 9 (Sept. 2012), 91--101.

Digital Library

Cited By

Rakhmetulla GRen YArif A(2023)GeShort: One-Handed Mobile Text Editing and Formatting with Gestural Shortcuts and a Floating ClipboardProceedings of the ACM on Human-Computer Interaction10.1145/36042597:MHCI(1-23)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604259
Chu JAn DMa YCui WZhai SGu XBi X(2023)WordGesture-GAN: Modeling Word-Gesture Movement with Generative Adversarial NetworkProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581279(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581279
Lemarquis DYilma BLeiva L(2023)What can a swiped word tell us more? Demographic and behavioral correlates from shape-writing text entryNeural Computing and Applications10.1007/s00521-023-08559-035:21(15531-15548)Online publication date: 13-Apr-2023
https://doi.org/10.1007/s00521-023-08559-0
Show More Cited By

Index Terms

CommandBoard: Creating a General-Purpose Command Gesture Input Space for Soft Keyboard
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

MK-pad: a Mouse+Keyboard input technique for distance interaction through a mobile tablet device
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction

We introduce MK-Pad (Mouse+Keyboard Pad), a technique for entering text and performing mouse operations using a tablet device for distance interaction. MK-Pad uses the screen of a 10.1" tablet touchscreen to host a soft keyboard and turns the screen ...
Soft Keyboard UX Evaluation: An Eye Tracking Study
MEDES '14: Proceedings of the 6th International Conference on Management of Emergent Digital EcoSystems

Touch screen devices are used daily in a variety of tasks, such as sending emails and typing notes. These devices usually come equipped with soft keyboards where their design needs to strike a careful balance to meet user needs. In this empirical study, ...
i'sFree: Eyes-Free Gesture Typing via a Touch-Enabled Remote Control
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Entering text without having to pay attention to the keyboard is compelling but challenging due to the lack of visual guidance. We propose i'sFree to enable eyes-free gesture typing on a distant display from a touch-enabled remote control. i'sFree does ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

October 2017

870 pages

ISBN:9781450349819

DOI:10.1145/3126594

General Chair:
Krzysztof Gajos
Harvard University
,
Program Chairs:
Jennifer Mankoff
Carnegie Mellon University
,
Chris Harrison
Carnegie Mellon University

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Research Council (ERC)

Conference

UIST '17

Sponsor:

UIST '17: The 30th Annual ACM Symposium on User Interface Software and Technology

October 22 - 25, 2017

QC, Québec City, Canada

Acceptance Rates

UIST '17 Paper Acceptance Rate 73 of 324 submissions, 23%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
839
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)4

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rakhmetulla GRen YArif A(2023)GeShort: One-Handed Mobile Text Editing and Formatting with Gestural Shortcuts and a Floating ClipboardProceedings of the ACM on Human-Computer Interaction10.1145/36042597:MHCI(1-23)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604259
Chu JAn DMa YCui WZhai SGu XBi X(2023)WordGesture-GAN: Modeling Word-Gesture Movement with Generative Adversarial NetworkProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581279(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581279
Lemarquis DYilma BLeiva L(2023)What can a swiped word tell us more? Demographic and behavioral correlates from shape-writing text entryNeural Computing and Applications10.1007/s00521-023-08559-035:21(15531-15548)Online publication date: 13-Apr-2023
https://doi.org/10.1007/s00521-023-08559-0
Mackay W(2023)Creating Human-Computer PartnershipsComputer-Human Interaction Research and Applications10.1007/978-3-031-49425-3_1(3-17)Online publication date: 23-Dec-2023
https://doi.org/10.1007/978-3-031-49425-3_1
Babic TReiterer HHaller M(2022)Understanding and Creating Spatial Interactions with Distant Displays Enabled by Unmodified Off-The-Shelf SmartphonesMultimodal Technologies and Interaction10.3390/mti61000946:10(94)Online publication date: 19-Oct-2022
https://doi.org/10.3390/mti6100094
Zhao MCui WRamakrishnan IZhai SBi X(2021)Voice and Touch Based Error-tolerant Multimodal Text Editing and Correction for SmartphonesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474742(162-178)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474742
Beaudouin-Lafon MBødker SMackay W(2021)Generative Theories of InteractionACM Transactions on Computer-Human Interaction10.1145/346850528:6(1-54)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1145/3468505
Leiva LKim SCui WBi XOulasvirta A(2021)How We Swipe: A Large-scale Shape-writing Dataset and Empirical FindingsProceedings of the 23rd International Conference on Mobile Human-Computer Interaction10.1145/3447526.3472059(1-13)Online publication date: 27-Sep-2021
https://dl.acm.org/doi/10.1145/3447526.3472059
Griggio CSato AMackay WYatani KKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Mediating Intimacy with DearBoard: a Co-Customizable Keyboard for Everyday MessagingProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445757(1-16)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445757
Cui WZhu SLi ZXu ZYang XRamakrishnan IBi XKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)BackSwipe: Back-of-device Word-Gesture Interaction on SmartphonesProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445081(1-12)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445081
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents