Article

Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks

Authors:

Susan R. Fussell,

Leslie D. Setlock,

Jie YangAuthors Info & Claims

ICMI '03: Proceedings of the 5th international conference on Multimodal interfaces

Pages 242 - 249

https://doi.org/10.1145/958432.958477

Published: 05 November 2003 Publication History

Abstract

We present a system integrating gesture and live video to support collaboration on physical tasks. The architecture combines network IP cameras, desktop PCs, and tablet PCs to allow a remote helper to draw on a video feed of a workspace as he/she provides task instructions. A gesture recognition component enables the system both to normalize freehand drawings to facilitate communication with remote partners and to use pen-based input as a camera control device. Results of a preliminary user study suggest that our gesture over video communication system enhances task performance over traditional video-only systems. Implications for the design of multimodal systems to support collaborative physical tasks are also discussed.

References

[1]

Bekker, M. M., Olson, J. S., & Olson, G. M. (1995). Analysis of gestures in face-to-face design teams provides guidance for how to use groupware in design. Proceedings of DIS 95. NY: ACM Press.]]

Digital Library

[2]

Flor, N. V. (1998). Side-by-side collaboration: A case study. International Journal of Human-Computer Studies, 49, 201--222.]]

Digital Library

[3]

Fussell, S. R., Kraut, R. E., & Siegel, J. (2000). Coordination of communication: Effects of shared visual context on collaborative work. Proceedings of CSCW 2000 (pp. 21--30). NY: ACM Press.]]

Digital Library

[4]

Fussell, S. R., Setlock, L. D., & Kraut, R. E. (2003). Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks. Proceedings of CHI 2003. NY: ACM Press.]]

Digital Library

[5]

Fussell, S., Setlock, L., Parker, E., & Yang, J. (2003). Assessing the value of a cursor pointing device for remote collaboration on physical tasks. Proceedings of CHI '2003. NY: ACM Press.]]

Digital Library

[6]

Fussell, S. R., Setlock, L. D., Yang, J., Ou, J., & Parker, E. M. (under review). Gestures over video streams to support remote collaboration on physical tasks.]]

[7]

Gutwin, C., & Penner, R. (2002). Improving interpretation of remote gestures with telepointer traces. Proceedings of CSCW 2002. (pp.49-57). NY: ACM Press.]]

Digital Library

[8]

Ishii, H., Kobayashi, M., & Grudin, J. (1993). Integration of interpersonal space and shared workspace: ClearBoard design and experiments. ACM Transactions on Information Systems, 11, 349--375.]]

Digital Library

[9]

Jin, X., Liu, W., Sun, J., & Sun, Z. (2002). On-line Graphics Recognition. Proceedings of PG'02 (pp. 256--265).]]

Digital Library

[10]

Jorge, J., & Fonseca, M. (1999). A Simple Approach to Recognise Geometric Shapes Interactively. Proceedings of GREC'99 (pp. 266--276).]]

Digital Library

[11]

Kraut, R. E., Fussell, S. R., & Siegel, J. (2003). Visual information as a conversational resource in collaborative physical tasks. Human-Computer Interaction, 18, 13--49.]]

Digital Library

[12]

Kraut, R. E., Miller, M. D., & Siegel, J. (1996) Collaboration in performance of physical tasks: Effects on outcomes and communication, Proceedings of CSCW'96 (57--66). NY: ACM Press.]]

Digital Library

[13]

Kuzuoka, H., & Shoji, H. (1994). Results of observational studies of spatial workspace collaboration. Electronics and Communications in Japan, 77, 58--68.]]

[14]

Kuzuoka, H., Kosuge, T., & Tanaka, K. (1994) GestureCam: A video communication system for sympathetic remote collaboration. Proceedings of CSCW 94 (35--43). NY: ACM Press.]]

Digital Library

[15]

Kuzuoka, H., Oyama, S., Yamazaki, K., Suzuki, K., & Mitsuishi, M. (2000). GestureMan: A mobile robot that embodies a remote instructor's actions. Proceedings of CSCW 2000 (pp. 155--162). NY: ACM Press]]

Digital Library

[16]

McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.]]

[17]

Ou, J., Chen, X., Fussell, S. R., & Yang, J. (in press). DOVE: Drawing over Video Environment. Multimedia 2003 Demonstrations.]]

Digital Library

[18]

Rubine, D. (1991). Specifying gestures by example. Computer Graphics, 25, 329--337.]]

Digital Library

[19]

Sezgin, M., Stahovich, T., & Davis, R. (2001). Sketch based interfaces: Early processing for sketch understanding. Proceedings of PUI-2001. NY: ACM Press.]]

Digital Library

[20]

Tang, J. C. (1991). Findings from observational studies of collaborative work. International Journal of Man-Machine Studies, 34, 143--160.]]

Digital Library

Cited By

Barnett THuang WMather CKim S(2024)HelpingHands: Developing Health Procedure Skills Through Augmented Reality-Based Remote Guidance2024 28th International Conference Information Visualisation (IV)10.1109/IV64223.2024.00065(338-343)Online publication date: 22-Jul-2024
https://doi.org/10.1109/IV64223.2024.00065
Huang WAlem LHuang WBillinghurst MAlem LXiao CRasmussen T(2024)Gesturing in the Air in Supporting Full MobilityComputer‐Supported Collaboration10.1002/9781119719830.ch8(195-213)Online publication date: 24-May-2024
https://doi.org/10.1002/9781119719830.ch8
Huang WWakefield MRasmussen TKim SBillinghurst MHuang WBillinghurst MAlem LXiao CRasmussen T(2024)Communication Cues for Remote GuidanceComputer‐Supported Collaboration10.1002/9781119719830.ch4(81-114)Online publication date: 24-May-2024
https://doi.org/10.1002/9781119719830.ch4
Show More Cited By

Index Terms

Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing design and evaluation methods
    2. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
2. Social and professional topics
  1. Professional topics
    1. Computing and business
      1. Computer supported cooperative work

Recommendations

DOVE: drawing over video environment
MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia

We demonstrate a multimedia system that integrates pen-based gesture and live video to support collaboration on physical tasks. The system combines network IP cameras, desktop PCs, and tablet PCs (or PDAs) to allow a remote helper to draw on a video ...
An exploratory analysis of partner action and camera control in a video-mediated collaborative task
CSCW '06: Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work

This paper reports on an exploratory experimental study of the relationships between physical movement and desired visual information in the performance of video-mediated collaborative tasks in the real world by geographically distributed groups. Twenty-...
Weak gaze awareness in video-mediated communication
CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems

We present a video mediated communication system that conveys gaze information to a remote location. Unlike existing video mediated communication system, this system does not send visual information directly, only gaze position and face direction. The ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '03: Proceedings of the 5th international conference on Multimodal interfaces

November 2003

318 pages

ISBN:1581136218

DOI:10.1145/958432

Conference Chair:
Sharon Oviatt
Oregon Health & Science University
,
Program Chairs:
Trevor Darrell
Massachusetts Institute of Technology
,
Mark Maybury
MITRE
,
Wolfgang Wahlster
DFKI, Germany

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI-PUI03

Sponsor:

ICMI-PUI03: International Conference on Multimodal User Interfaces

November 5 - 7, 2003

British Columbia, Vancouver, Canada

Acceptance Rates

ICMI '03 Paper Acceptance Rate 45 of 130 submissions, 35%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

93
Total Citations
View Citations
1,382
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)4

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Barnett THuang WMather CKim S(2024)HelpingHands: Developing Health Procedure Skills Through Augmented Reality-Based Remote Guidance2024 28th International Conference Information Visualisation (IV)10.1109/IV64223.2024.00065(338-343)Online publication date: 22-Jul-2024
https://doi.org/10.1109/IV64223.2024.00065
Huang WAlem LHuang WBillinghurst MAlem LXiao CRasmussen T(2024)Gesturing in the Air in Supporting Full MobilityComputer‐Supported Collaboration10.1002/9781119719830.ch8(195-213)Online publication date: 24-May-2024
https://doi.org/10.1002/9781119719830.ch8
Huang WWakefield MRasmussen TKim SBillinghurst MHuang WBillinghurst MAlem LXiao CRasmussen T(2024)Communication Cues for Remote GuidanceComputer‐Supported Collaboration10.1002/9781119719830.ch4(81-114)Online publication date: 24-May-2024
https://doi.org/10.1002/9781119719830.ch4
Huang WAlem LTecchia FDuh HHuang WBillinghurst MAlem LXiao CRasmussen T(2024) Augmenting Hand Gestures in 3D Mixed Reality Computer‐Supported Collaboration10.1002/9781119719830.ch10(243-268)Online publication date: 24-May-2024
https://doi.org/10.1002/9781119719830.ch10
LC RLiu SLyu QEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)IN/ACTive: A Distance-Technology-Mediated Stage for Performer-Audience Telepresence and Environmental ControlProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613791(6989-6997)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3613791
Poh ETang ALee JShengdong Z(2023)Supporting Artefact Awareness in Partially-Replicated Workspaces2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00019(55-59)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00019
Letter MGeorge CWolf K(2023)A Survey of Computer-Supported Remote Collaboration on Physical ObjectsHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42286-7_19(346-368)Online publication date: 25-Aug-2023
https://doi.org/10.1007/978-3-031-42286-7_19
Syed TSiddiqui MAbdullah HJan SNamoun AAlzahrani ANadeem AAlkhodre A(2022)In-Depth Review of Augmented Reality: Tracking Technologies, Development Tools, AR Displays, Collaborative AR, and Security ConcernsSensors10.3390/s2301014623:1(146)Online publication date: 23-Dec-2022
https://doi.org/10.3390/s23010146
Dell NEstrin DHaraldsson HJu W(2022)Designing Extended Reality Guidance for Physical Caregiving Tasks2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW55335.2022.00091(419-422)Online publication date: Mar-2022
https://doi.org/10.1109/VRW55335.2022.00091
Huang WWakefield MRasmussen TKim SBillinghurst M(2022)A review on communication cues for augmented reality based remote guidanceJournal on Multimodal User Interfaces10.1007/s12193-022-00387-116:2(239-256)Online publication date: 12-Apr-2022
https://doi.org/10.1007/s12193-022-00387-1
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents