research-article

Speech and sketching: an empirical study of multimodal interaction

Authors:

R. DavisAuthors Info & Claims

SBIM '07: Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling

Pages 83 - 90

https://doi.org/10.1145/1384429.1384449

Published: 02 August 2007 Publication History

Abstract

Sketch recognition can capture the sketching component of a multimodal conversation about design, but it does not capture information conveyed in the other modalities. The informal speech that accompanies a sketch often has a considerable amount of additional information. We want to develop a digital whiteboard capable of understanding both sketching and speech, and capable of participating in a conversation similar to one that the user would have with a human design partner. We conducted a user study to help us understand what kinds of conversations users would have with a whiteboard capable of recognizing a sketch. We report results that we believe will help guide the design of an effective multimodal interface, and discuss implications for system architectures.

References

[1]

{AD01} Alvarado C., Davis R.: Resolving ambiguities to create a natural sketch based interface. In IJCAI (2001). 7

Digital Library

[2]

{AD04} Adler A., Davis R.: Speech and sketching for multimodal design. In IUI (2004), ACM Press, pp. 214--216. 8

Digital Library

[3]

{AHWA04} Anderson R. J., Hoyer C., Wolfman S. A., Anderson R.: A study of digital ink in lecture presentation. In CHI '04 (New York, NY, USA, 2004), ACM Press, pp. 567--574. 4

Digital Library

[4]

{CJM*97} Cohen P. R., Johnston M., Mcgee D. R., Oviatt S. L., Pittman J., Smith I., Chen L., Clowi J.: Quickset: Multimodal interaction for distributed applications. In Multimedia (1997), ACM Press, pp. 31--40. 1

Digital Library

[5]

{Dav02} Davis R.: Sketch understanding in design: Overview of work at the MIT AI lab. AAAI Spring Symposium (2002), 24--31. 1

[6]

{DKD05} Demirdjian D., Ko T., Darrell T.: Untethered gesture acquisition and recognition for virtual world manipulation. Virtual Reality 8, 4 (2005), 222--230. 1, 7

Digital Library

[7]

{HAH*93} Huang X., Alleva R, Hon H.-W., Hwang M.-Y., Rosenfeld R.: The SPHINX-II speech recognition system: an overview. Computer Speech and Language 7, 2 (1993), 137--148. 3

[8]

{JEW*02} Johnston M., Ehlen P., Walker M., Whittaker S., Maloor P.: MATCH: An architecture for multimodal dialogue systems. In ACL (2002), pp. 276--383. 2, 7

Digital Library

[9]

{Kai05} Kaiser E. C.: Multimodal new vocabulary recognition through speech and handwriting in a whiteboard scheduling application. In IUI '05 (New York, NY, USA, January 2005), ACM Press, pp. 51--58. 1, 2, 4, 7

Digital Library

[10]

{Kai06} Kaiser E. C.: Using redundant speech and handwriting for learning new vocabulary and understanding abbreviations. In ICMI (Banff, Canada, 2006). 7

Digital Library

[11]

{KBEC07} Kaiser E. C., Barthelmess P., Erdmann C., Cohen P.: Multimodal redundancy across handwriting and speech during computer mediated human-human interactions. In CHI '07 (New York, NY, USA, April 2007), ACM Press, pp. 1009--1018. 5, 7

Digital Library

[12]

{OCFF92} Oviatt S., Cohen P., Fong M., Frank M.: A rapid semi-automatic simulation technique for investigating interactive speech and handwriting. In ICSLP (1992), University of Alberta, pp. 1351--1354. 7

[13]

{OCW*00} Oviatt S., Cohen P., Wu L., Vergo J., Duncan L., Suhm B., Bers J., Holzman T., Wino-Grad T., Landay J., Larson J., Ferro D.: Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions. In HCI (2000), pp. 263--322. 2, 7

Digital Library

[14]

{OD01} Oltmans M., Davis R.: Naturally conveyed explanations of device behavior. In PUI (2001). 2, 7

Digital Library

[15]

{ODK97} Oviatt S., Deangeli A., Kuhn K.: Integration and synchronization of input modes during multimodal human-computer interaction. In Human Factors in Computing Systems (1997), ACM Press, pp. 415--422. 5, 6, 7

Digital Library

[16]

{RS98} Rich C., Sidner C.: COLLAGEN: A collaboration manager for software interface agents. User Modeling and User-Adapter Interaction 8, 3--4 (1998), 315--350. 7

Digital Library

[17]

{Ske} MIT SketchML Format. http://rationale.csail.mit.edu/ETCHASketches/format/. 3

[18]

{UWC90} Ullman D. G., Wood S., Craig D.: The importance of drawing in the mechanical design process. Computers and Graphics 14, 2 (1990), 263--274. 1

Cited By

Giunchi DSztrajman AJames SSteed A(2021)Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual RealityProceedings of the 2021 ACM International Conference on Interactive Media Experiences10.1145/3452918.3458806(144-155)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3452918.3458806
Chen TMohanty RKrishnamurthy V(2021)Queries and Cues: Textual Stimuli for Reflective Thinking in Digital Mind-MappingJournal of Mechanical Design10.1115/1.4052297144:2Online publication date: 21-Sep-2021
https://doi.org/10.1115/1.4052297
Bonnici AAkman ACalleja GCamilleri KFehling PFerreira AHermuth FIsrael JLandwehr TLiu JPadfield NSezgin TRosin P(2019)Sketch-based interaction and modeling: where do we stand?Artificial Intelligence for Engineering Design, Analysis and Manufacturing10.1017/S089006041900034933:4(370-388)Online publication date: 29-Nov-2019
https://doi.org/10.1017/S0890060419000349
Show More Cited By

Index Terms

Speech and sketching: an empirical study of multimodal interaction
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Sound-based input / output
2. Human-centered computing
  1. Human computer interaction (HCI)
  2. Interaction design
    1. Interaction design process and methods
      1. User centered design

Recommendations

Speech and sketching for multimodal design
SIGGRAPH '07: ACM SIGGRAPH 2007 courses

While sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. ...
Speech and sketching for multimodal design
IUI '04: Proceedings of the 9th international conference on Intelligent user interfaces

While sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. ...
Speech and sketching for multimodal design
SIGGRAPH '06: ACM SIGGRAPH 2006 Courses

While sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SBIM '07: Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling

August 2007

152 pages

ISBN:9781595939135

DOI:10.1145/1384429

Conference Chairs:
Thomas Stahovich
University of California, Riverside, CA
,
Christine Alvarado
Harvey Mudd College, Claremont, CA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
EUROGRAPHICS: The European Association for Computer Graphics

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 August 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SBIM07

Sponsor:

SIGGRAPH
EUROGRAPHICS

SBIM07: 4th Eurographics Workshop on Sketch-Based Interfaces and Modeling

August 2 - 3, 2007

California, Riverside

Acceptance Rates

Overall Acceptance Rate 20 of 36 submissions, 56%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
274
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Giunchi DSztrajman AJames SSteed A(2021)Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual RealityProceedings of the 2021 ACM International Conference on Interactive Media Experiences10.1145/3452918.3458806(144-155)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3452918.3458806
Chen TMohanty RKrishnamurthy V(2021)Queries and Cues: Textual Stimuli for Reflective Thinking in Digital Mind-MappingJournal of Mechanical Design10.1115/1.4052297144:2Online publication date: 21-Sep-2021
https://doi.org/10.1115/1.4052297
Bonnici AAkman ACalleja GCamilleri KFehling PFerreira AHermuth FIsrael JLandwehr TLiu JPadfield NSezgin TRosin P(2019)Sketch-based interaction and modeling: where do we stand?Artificial Intelligence for Engineering Design, Analysis and Manufacturing10.1017/S089006041900034933:4(370-388)Online publication date: 29-Nov-2019
https://doi.org/10.1017/S0890060419000349
Altıok OSezgin TWinnemoeller HBartram LGooch BGingold Y(2017)Characterizing user behavior for speech and sketch-based video retrieval interfacesProceedings of the Symposium on Non-Photorealistic Animation and Rendering10.1145/3092919.3122801(1-2)Online publication date: 29-Jul-2017
https://dl.acm.org/doi/10.1145/3092919.3122801
Altıok OSezgin TWinnemoeller HBartram LGooch BGingold Y(2017)Characterizing user behavior for speech and sketch-based video retrieval interfacesProceedings of the symposium on Computational Aesthetics10.1145/3092912.3122801(1-2)Online publication date: 29-Jul-2017
https://dl.acm.org/doi/10.1145/3092912.3122801
Altıok OSezgin TWinnemoeller HBartram LGooch BGingold Y(2017)Characterizing user behavior for speech and sketch-based video retrieval interfacesProceedings of the Symposium on Sketch-Based Interfaces and Modeling10.1145/3092907.3122801(1-2)Online publication date: 29-Jul-2017
https://dl.acm.org/doi/10.1145/3092907.3122801
Cohen POviatt S(2017)Multimodal speech and pen interfacesThe Handbook of Multimodal-Multisensor Interfaces10.1145/3015783.3015795(403-447)Online publication date: 24-Apr-2017
https://dl.acm.org/doi/10.1145/3015783.3015795
Palou ERamírez-Apud ZRamírez-Corona NLópez-Malo A(2016)Analysis of Student Perspectives on Using Tablet PCs in Junior and Senior Level Chemical Engineering CoursesRevolutionizing Education with Digital Ink10.1007/978-3-319-31193-7_21(307-319)Online publication date: 19-May-2016
https://doi.org/10.1007/978-3-319-31193-7_21
Prasad MHammond TKonstan JChi EHöök K(2012)Observational study on teaching artifacts created using tablet PCCHI '12 Extended Abstracts on Human Factors in Computing Systems10.1145/2212776.2212809(301-316)Online publication date: 5-May-2012
https://dl.acm.org/doi/10.1145/2212776.2212809
Walny JCarpendale SHenry Riche NVenolia GFawcett P(2011)Visual Thinking In ActionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2011.25117:12(2508-2517)Online publication date: 1-Dec-2011
https://dl.acm.org/doi/10.1109/TVCG.2011.251
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten