Article

Free access

Something from nothing: augmenting a paper-based work practice via multimodal interaction

Authors:

David R. McGee,

Philip R. Cohen,

Lizhong WuAuthors Info & Claims

DARE '00: Proceedings of DARE 2000 on Designing augmented reality environments

Pages 71 - 80

https://doi.org/10.1145/354666.354674

Published: 01 April 2000 Publication History

Abstract

In this paper, we describe Rasa: an environment designed to augment, rather than replace, the work habits of its users. These work habits include drawing on Post-it™ notes using a symbolic language. Rasa observes and understands this language, assigning meaning simultaneously to objects in both the physical and virtual worlds. With Rasa, users rollout a paper map, register it, and move the augmented objects from one place to another on it. Once an object is augmented, users can modify the meaning represented by it, ask questions about that representation, view it in virtual reality, or give directions to it, all with speech and gestures. We examine the way Rasa uses language to augment objects, and compare it with prior methods, arguing that language is a more visible, flexible, and comprehensible method for creating augmentations than other approaches.

References

[1]

Bolt, R.A., "Put-That-There": Voice and gesture at the graphics interface. Computer Graphics, 1980. 14(3): 262-270.]]

Digital Library

[2]

Carpenter, R., Typed feature structures: Inheritance, (in)equality, and extensionality, in the Proceedings of the ITK Workshop: Inheritance in Natural Language Processing, Institute for Language Technology and Artificial Intelligence, Tilburg University, Tilburg, 9-18.]]

[3]

Carpenter, R., The logic of typed feature structures. 1992, Cambridge England: Cambridge University Press.]]

Digital Library

[4]

Coen, M.H., Design principles for intelligent environments, in the Proceedings of the Conference on Artificial Intelligence (AAAI '98), July 1998, American Association for Artificial Intelligence, 547-554.]]

Digital Library

[5]

Cohen, P.R., Integrated interfaces for decision support with simulation, in the Proceedings of the Winter Simulation Conference, ACM Press, 1066-1072.]]

Digital Library

[6]

Cohen, P.R., Cheyer, A., Wang, M., and Baeg, S.C., An open agent architecture, in the Proceedings of the AAAI Spring Symposium, Mar. 1994, Reprinted in Readings in Agents, Huhns, M. and Singh, M. (eds.), Morgan Kaufmann Publishers, San Francisco, 1-8.]]

Digital Library

[7]

Cohen, P.R., Johnston, M., McGee, D.R., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J., Quick- Set: multimodal interaction for distributed applications, in the Proceedings of the Fifth Annual ACM In-ternational Multimedia Conference (Multimedia '97), November 1997, ACM Press, 31-40.]]

Digital Library

[8]

Cohen, P.R., McGee, D.R., and Clow, J., The efficiency of multimodal interaction, in the Proceedings of the Applied Natural Language Programming Conference (ANLP'00), April, 2000, ACL.]]

Digital Library

[9]

Ishii, H. and Ullmer, B., Tangible bits: towards seamless interfaces between people, bits and atoms, in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '97), March 1997, ACM Press, 234-241.]]

Digital Library

[10]

Johnston, M., Unification-based multimodal parsing, in the Proceedings of the 17th International Conference on Computational Linguistics and the 36th An-nual Meeting of the Association for Computational Linguistics (COLING-ACL 98), August 98, ACL Press, 624-630.]]

Digital Library

[11]

Johnston, M., Cohen, P.R., McGee, D.R., Oviatt, S.L., Pittman, J.A., and Smith, I., Unification-based multimodal integration, in the Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL '97), March 1997, ACL Press, 281-288.]]

Digital Library

[12]

Koons, D.B., Sparrell, C.J., and Thorisson, K.R., Integrating simultaneous input from speech, gaze, and hand gestures, in Intelligent Multimedia Interfaces, M.T. Maybury, Editor. 1993, AAAI Press/MIT Press: Cambridge, MA. 257-276.]]

Digital Library

[13]

Krueger, M.W., Artificial Reality II. 1991, Reading, MA: Addison-Wesley. p. 286.]]

[14]

McGee, D.R., Cohen, P.R., and Oviatt, S., Confirmation in multimodal systems, in the Proceedings of the International Joint Conference of the Association for Computational Linguistics and the International Committee on Computational Linguistics (COLING- ACL '98), August 1998, ACL Press, 823-829.]]

Digital Library

[15]

Moran, T.P., Saund, E., Melle, W.V., Bryll, R., Gujar, A.U., Fishkin, K.P., and Harrison, B.L., The ins and outs of collaborative walls: Demonstrating the Collaborage concept, in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '99), May 15-20, 1999, ACM Press, CHI'99 Extended Abstracts, 192-193.]]

Digital Library

[16]

Neal, J.G. and Shapiro, S.C., Intelligent multi-media interface technology, in Intelligent User Interfaces, J.W. Sullivan and S.W. Tyler, Editors. 1991, ACM Press, Frontier Series, Addison Wesley Publishing Co.: New York, New York. 45-68.]]

Digital Library

[17]

Newman, W. and Wellner, P., A desk supporting computer-based interaction with paper documents, in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '92), May 1992, ACM Press, 587-592.]]

Digital Library

[18]

Nigay, L. and Coutaz, J., A generic platform for addressing the multimodal challenge, in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '95), May 7-11, 1995, ACM Press, 98-105.]]

Digital Library

[19]

Oviatt, S.L., Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, 1997. 12 (special issue on "Multimodal interfaces"): 93-129.]]

[20]

Oviatt, S.L., DeAngeli, A., and Kuhn, K., Integration and synchronization of input modes during multimodal human-computer interaction, in the Proceedings of the Conference on Human Factors in Computing Systems (CHI '97), March 1997, ACM Press, 415- 422.]]

Digital Library

[21]

Streitz, N.A., Geibler, J., and Holmer, T., Roomware for cooperative buildings: integrated design of architectural spaces and information spaces, in the Proceedings of the First International Workshop on Cooperative Buildings: Integrating Information, Organization, and Architecture (CoBuild '98), February 1998, Springer-Verlag, 4-21.]]

Digital Library

[22]

Ullmer, B. and Ishii, H., The metaDESK: models and prototypes for tangible user Interfaces, in the Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '97), October 1997, ACM Press, 223-232.]]

Digital Library

[23]

Ullmer, B. and Ishii, H., mediaBlocks: physical containers, transports, and controls for online media, in the Proceedings of the 25th International ACM Conference on Computer Graphics and Interactive Techniques, July 1998, ACM Press, 379-386.]]

Digital Library

[24]

Underkoffler, J. and Ishii, H., Urp: a luminoustangible workbench for urban planning and design, in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '99), May 1999, ACM Press, 386-393.]]

Digital Library

[25]

Want, R., Fishkin, K.P., Gujar, A., and Harrison, B.L., Bridging physical and virtual worlds with electronic tags, in the Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '99), May 1999, ACM Press, 370-377.]]

Digital Library

[26]

Weiser, M., Some computer science issues in ubiquitous computing. Communications of the ACM, 1993. 36(7): 75-84.]]

Digital Library

[27]

Wellner, P., Interacting with paper on the DigitalDesk. Communications of the ACM, 1993. 36(7): 87-96.]]

Digital Library

[28]

Wu, L., Oviatt, S., and Cohen, P., Multimodal integration - A statistical view. IEEE Transactions on Multimedia, 1999. 1(4): 334-341.]]

Cited By

Cheng YZhu GYang CMiao GGe W(2022)Characteristics of augmented map research from a cartographic perspectiveCartography and Geographic Information Science10.1080/15230406.2022.205957149:5(426-442)Online publication date: 22-Apr-2022
https://doi.org/10.1080/15230406.2022.2059571
Fischel AHalskov KMandryk RHancock MPerry MCox A(2018)A Survey of the Usage of Sticky NotesExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3188526(1-6)Online publication date: 20-Apr-2018
https://dl.acm.org/doi/10.1145/3170427.3188526
Dumas BPirau JLalanne DLank EVinciarelli AHoggan ESubramanian SBrewster S(2017)Modelling fusion of modalities in multimodal interactive systems with MMMMProceedings of the 19th ACM International Conference on Multimodal Interaction10.1145/3136755.3136768(288-296)Online publication date: 3-Nov-2017
https://dl.acm.org/doi/10.1145/3136755.3136768
Show More Cited By

Index Terms

Something from nothing: augmenting a paper-based work practice via multimodal interaction
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Recommendations

Creating tangible interfaces by augmenting physical objects with multimodal language
IUI '01: Proceedings of the 6th international conference on Intelligent user interfaces

Rasa is a tangible augmented reality environment that digitally enhances the existing paper-based command and control capability in a military command post. By observing and understanding the users' speech, pen, and touch-based multimodal language, Rasa ...
Natural and tangible human-computer interfaces for augmented environments
SIGDOC '08: Proceedings of the 26th annual ACM international conference on Design of communication

In this talk, the author will introduce Augmented Reality as a new media to improve human-computer communication and industrial processes. The author will make a special emphasis in introducing and demonstrating new human-computer interfaces appropriate ...
REVEL: tactile feedback technology for augmented reality

REVEL is an augmented reality (AR) tactile technology that allows for change to the tactile feeling of real objects by augmenting them with virtual tactile textures using a device worn by the user. Unlike previous attempts to enhance AR environments ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DARE '00: Proceedings of DARE 2000 on Designing augmented reality environments

April 2000

171 pages

ISBN:9781450373265

DOI:10.1145/354666

Chairman:
Wendy E. Mackay
Univ. of Aarhus, Aarhus, Denmark

Copyright © 2000 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2000

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

DARE00

Sponsor:

ACM

DARE00: Designing Augmented Reality Environments

Elsinore, Denmark

Acceptance Rates

Overall Acceptance Rate 5 of 7 submissions, 71%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
1,018
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)4

Reflects downloads up to 18 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng YZhu GYang CMiao GGe W(2022)Characteristics of augmented map research from a cartographic perspectiveCartography and Geographic Information Science10.1080/15230406.2022.205957149:5(426-442)Online publication date: 22-Apr-2022
https://doi.org/10.1080/15230406.2022.2059571
Fischel AHalskov KMandryk RHancock MPerry MCox A(2018)A Survey of the Usage of Sticky NotesExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3170427.3188526(1-6)Online publication date: 20-Apr-2018
https://dl.acm.org/doi/10.1145/3170427.3188526
Dumas BPirau JLalanne DLank EVinciarelli AHoggan ESubramanian SBrewster S(2017)Modelling fusion of modalities in multimodal interactive systems with MMMMProceedings of the 19th ACM International Conference on Multimodal Interaction10.1145/3136755.3136768(288-296)Online publication date: 3-Nov-2017
https://dl.acm.org/doi/10.1145/3136755.3136768
Dolata MSchwabe G(2017)Paper Practices in Institutional TalkComputer Supported Cooperative Work10.1007/s10606-017-9279-826:4-6(769-805)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1007/s10606-017-9279-8
Oviatt SCohen P(2015)The Paradigm Shift to Multimodality in Contemporary Computer InterfacesSynthesis Lectures on Human-Centered Informatics10.2200/S00636ED1V01Y201503HCI0308:3(1-243)Online publication date: 13-Apr-2015
https://doi.org/10.2200/S00636ED1V01Y201503HCI030
Duque CDe la Rosa FHernandez J(2013)Multimodal interaction architecture applied to navigation in maps2013 8th Computing Colombian Conference (8CCC)10.1109/ColombianCC.2013.6637520(1-6)Online publication date: Aug-2013
https://doi.org/10.1109/ColombianCC.2013.6637520
Ullmer BDever ZSankaran RToole CFreeman CCassady BWiley CDiabi MWallace ADeLatin MTregre BLiu KJandhyala SKooima RBranton CParker RCoelho MZigelbaum JIshii HJacob RMaes PPederson TShaer OWakkary R(2010)CartoucheProceedings of the fourth international conference on Tangible, embedded, and embodied interaction10.1145/1709886.1709904(93-100)Online publication date: 24-Jan-2010
https://dl.acm.org/doi/10.1145/1709886.1709904
Morrison AOulasvirta APeltonen PLemmela SJacucci GReitmayr GNäsänen JJuustila AOlsen DArthur RHinckley KMorris MHudson SGreenberg S(2009)Like bees around the hiveProceedings of the SIGCHI Conference on Human Factors in Computing Systems10.1145/1518701.1518991(1889-1898)Online publication date: 4-Apr-2009
https://dl.acm.org/doi/10.1145/1518701.1518991
Gurses AXiao YHu P(2009)User-designed information tools to support communication and care coordination in a trauma hospitalJournal of Biomedical Informatics10.1016/j.jbi.2009.03.00742:4(667-677)Online publication date: 1-Aug-2009
https://dl.acm.org/doi/10.1016/j.jbi.2009.03.007
Mistry PMaes P(2009)Augmenting Sticky Notes as an I/O InterfaceProceedings of the 5th International on ConferenceUniversal Access in Human-Computer Interaction. Part II: Intelligent and Ubiquitous Interaction Environments10.1007/978-3-642-02710-9_61(547-556)Online publication date: 14-Jul-2009
https://dl.acm.org/doi/10.1007/978-3-642-02710-9_61
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents