research-article

Modeling ontology for multimodal interaction in ubiquitous computing systems

Authors:

Amar Ramdane Cherif,

Chakib TadjAuthors Info & Claims

UbiComp '12: Proceedings of the 2012 ACM Conference on Ubiquitous Computing

Pages 842 - 849

https://doi.org/10.1145/2370216.2370408

Published: 05 September 2012 Publication History

Abstract

People communicate with each other using different ways, such as words, gestures, etc. to give information about their status, emotions and intentions. But how may this information be described in a way that autonomous systems (e.g. Robots) can react with a human being in a given environment?

A multimodal interface allows a more flexible and natural interaction between a user and a computing system. This paper presents a methodological approach for designing an architecture that facilitates the work of a fusion engine. The selection of modalities and the fusion of events invoked by the fusion engine are based upon the definition of an ontology that describes the environment where a multimodal interaction system exists.

References

[1]

Yuen, P. C., Y. Y. Tang, and P. S.-p. Wang, Multimodal Interface for Human-Machine Communication. Machine Perception Artificial Intelligence. Vol. 48. 2002, Singapore: World Scientific Publishing Co., Pte. Ltd.

Digital Library

[2]

Oviatt, S. L. and P. R. Cohen, Multimodal Interfaces that Process What Comes Naturally. Communications of the ACM, 2000. 43(3): p. 45--53.

Digital Library

[3]

Shin, B.-S., H. Ahn, and G. Y. Kim, Wearable multimodal interface for helping visually handicapped persons, in 16th international conference on artificial reality and telexistence. 2006, LNCS vol. 4282: Hangzhou, China, p. 989--988.

Digital Library

[4]

Raisamo, R., et al., Testing usability of multimodal applications with visually impaired children. IEE, Institute of Electrical and Electronics Engineers Computer Society, 2006. 13(3): p. 70--76.

Digital Library

[5]

Lai, J., S. Mitchell, and C. Pavlovski, Examining modality usage in a conversational multimodal application for mobile e-mail access. International Journal of Speech Technology, 2007. 10(1): p. 17--30.

[6]

Debevc, M., et al., Accessible multimodal Web pages with sign language translations for deaf and hard of hearing users, in DEXA 2009, 20th International Workshop on Database and Expert Systems Application. 2009, IEEE: Linz, Austria. p. 279--283.

Digital Library

[7]

Bolt., "Put-that-there": Voice and gesture at the graphics interface ACMSIGGRAPH Computer Graphics, v.14 n.3, p.262--270, July 1980

Digital Library

[8]

Holzapfel, H., Nickel, K., and Stiefelhagen, R. I., mplementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures. In Proceedings of the 6th international Conference on Multimodal interfaces (State College, PA, USA. ICMI '04. ACM, New York, NY, 175--182. 2004.

Digital Library

[9]

Duarte, C.a.C., L., A conceptual framework for developing adaptive multimodal applications. In Proceedings of the 11th international Conference on intelligent User interfaces (Sydney, Australia, January 29 - February 01, 2006). IUI '06. ACM, New York, NY, 132--139. 2006.

Digital Library

[10]

Latoschik, M. E., Designing transition networks for multimodal VR-interactions using a markup language. Multimodal Interfaces, 2002. In Proceedings of the Fourth IEEE International Conference on Multimodal Interfaces. IEEE, 411--416. 2002.

Digital Library

[11]

Schlomer, T., Poppinga B., Henze N. & Boll S., Gesture Recognition with a Wii Controller, Proceedings of the 2nd international conference on Tangible and embedded interaction, ACM, 2008. 2008.

Digital Library

[12]

Jobs, S. P.e.a., Touch Screen Device, Method, and Graphical User Interface for Determining Commands by Applying Heuristics. United States Patent Application 20080122796. Kind Code A, May 29, 2008. 2008.

[13]

Schroeter, J., et al. Multimodal Speech Synthesis. 2000. New York, NY.

[14]

Engel, R. P., Norbert., Modality Fusion.SmartKom: Foundations of Multimodal Dialogue Systems. Springer, Berlin. 2006.

[15]

Norbert Reithinger, JA., Tilman Becker, Anselm Blocher, Ralf Engel, Markus Löckelt, Jochen Müller, Norbert Pfleger, Peter Poller, Michael Streit, Valentin Tschernomas, SmartKom - Adaptive and Flexible Multimodal Access to Multiple Applications, DFKI GmbH -- German Research Center for Artificial Intelligence2003.

Digital Library

[16]

Daniel Sonntag, M. R., A Multimodal Result Ontology for Integrated Semantic Web Dialogue Applications Daniel Sonntag_, Massimo Romanelli DFKI GmbH. German Research Center for Arti_cial Intelligence Stuhlsatzenhausweg 3, d-66123 2006.

[17]

www.ontologyportal.org.

[18]

www.loa-cnr.it/DOLCE.html.

[19]

Dumas, B., D. Lalanne, and R. Ingold, Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML. Journal on Multimodal User Interfaces, 2010. 3(3): p. 237--247.

[20]

Little, T. D. C., Ghafoor, A., Chen, C. Y. R., Chang, C. S., Berra, P. B., Multimedia Synchronization. IEEE Data Engineering Bulletin, 1991. 14: p. 26--35.

Digital Library

[21]

Henry, T. R., Hudson, S. E., Newell, G. L., Integrating Gesture and Snapping into a User Interface Toolkit, in Proc, in Symposium on User Interface Software and Technology 1990, ACM Press. p. 112--121.

Digital Library

[22]

Nigay, L., Coutaz, J., Salber, D. MATIS, A multimodal airline travel information system, E. B. A. SM/WP10, Editor. 1993.

[23]

http://tt.Stanford.edu/.

[24]

http://www.w3.org/Submission/SWRL/.

[25]

http://www.w3.org/TR/owl-guide/.

Cited By

Erazo-Garzon LAvila JPinos SCedillo P(2022)A Systematic Review on the Use of Ontologies in the Internet of ThingsApplied Technologies10.1007/978-3-031-03884-6_37(509-524)Online publication date: 6-Apr-2022
https://doi.org/10.1007/978-3-031-03884-6_37
Pereira Dda S. e Silva Fde Salles S. Neto Cdos Santos DCoutinho LGuedes Á(2021)An ontology-based approach to integrate TV and IoT middlewaresMultimedia Tools and Applications10.1007/s11042-020-09645-480:2(1813-1837)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s11042-020-09645-4

Index Terms

Modeling ontology for multimodal interaction in ubiquitous computing systems
1. Human-centered computing

Recommendations

MULTIMODAL FUSION AS COMMUNICATIVE ACTS DURING HUMAN–ROBOT INTERACTION

Research on dialog systems is a very active area in social robotics. During the last two decades, these systems have evolved from those based only on speech recognition and synthesis to the current and modern systems, which include new components and ...
Multimodal Interaction within Ambient Environments: An Exploratory Study
INTERACT '09: Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part II

Inputs and outputs are not two independent phenomena in multimodal systems. This paper examines the relationship that exists between them. We present the results of a Wizard of Oz experiment which shows that output modalities used by the system have an ...
Multimodal floor control shift detection
ICMI-MLMI '09: Proceedings of the 2009 international conference on Multimodal interfaces

Floor control is a scheme used by people to organize speaking turns in multi-party conversations. Identifying the floor control shifts is important for understanding a conversation's structure and would be helpful for more natural human computer ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '12: Proceedings of the 2012 ACM Conference on Ubiquitous Computing

September 2012

1268 pages

ISBN:9781450312240

DOI:10.1145/2370216

General Chair:
Anind K. Dey
Carnegie Mellon University
,
Program Chairs:
Hao-Hua Chu
National Taiwan University, Taiwan
,
Gillian Hayes
University of California, Irvine

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

SIGSPATIAL: ACM Special Interest Group on Spatial Information

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 September 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

Ubicomp '12

Sponsor:

Ubicomp '12: The 2012 ACM Conference on Ubiquitous Computing

September 5 - 8, 2012

Pennsylvania, Pittsburgh

Acceptance Rates

UbiComp '12 Paper Acceptance Rate 58 of 301 submissions, 19%;

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
156
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Erazo-Garzon LAvila JPinos SCedillo P(2022)A Systematic Review on the Use of Ontologies in the Internet of ThingsApplied Technologies10.1007/978-3-031-03884-6_37(509-524)Online publication date: 6-Apr-2022
https://doi.org/10.1007/978-3-031-03884-6_37
Pereira Dda S. e Silva Fde Salles S. Neto Cdos Santos DCoutinho LGuedes Á(2021)An ontology-based approach to integrate TV and IoT middlewaresMultimedia Tools and Applications10.1007/s11042-020-09645-480:2(1813-1837)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s11042-020-09645-4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten