research-article

Modeling input modality choice in mobile graphical and speech interfaces

Authors:

Stefan Schaffer,

Robert Schleicher,

Sebastian MöllerAuthors Info & Claims

International Journal of Human-Computer Studies, Volume 75, Issue C

Pages 21 - 34

https://doi.org/10.1016/j.ijhcs.2014.11.004

Published: 01 March 2015 Publication History

Abstract

In this paper, we review three experiments with a mobile application that integrates graphical input with a touch-screen and a speech interface and develop a model for input modality choice in multimodal interaction. The model aims to enable simulation of multimodal human-computer interaction for automatic usability evaluation. The experimental results indicate that modality efficiency and input performance are important moderators of modality choice. Accordingly, we establish a utility-driven model that provides probability estimations of modality usage, based on the parameters of modality efficiency and input performance. Four variants of the model that differ in training data are fitted by means of Sequential Least Squares Programming. The analysis reveals a considerable fit regarding averaged modality usage. When applied to individual modality usage profiles, the accuracy decreases significantly. In an application example it is shown how the modality choice mechanism can be deployed for simulating interaction in the field of automatic usability evaluation. Results and possible limitations are discussed. Decreased input performance of a specific modality, decreases the usage of this modality.Decreased efficiency of the GUI, increases VUI usage.A model predicting modality choice is proposed.The model can be deployed for automatic usability evaluation.System designers are provided with working knowledge about expected modality usage.

References

[1]

J.R. Anderson, M. Matessa, C. Lebiere, Act-r: a theory of higher level cognition and its relation to visual attention, Hum.-Comput. Interact., 12 (1997) 439-462.

[2]

Apple, 2011. Siri, Online}. Available : {http://www.apple.com/ios/siri/} (25.01.13).

[3]

Bellamy, R., John, B., Kogan, S., 2011. Deploying cogtool: integrating quantitative usability assessment into real-world software development. In: Proceedings of the 33rd International Conference on IEEE Software Engineering (ICSE), pp. 691-700.

[4]

N.O. Bernsen, Multimodality theory, in: Multimodal User Interfaces, Springer, Berlin Heidelberg, 2008, pp. 5-29.

[5]

N. Bevan, Usability is quality of use, Adv. Hum. Factors Ergon., 20 (1995) 349-354.

[6]

Bilici, V., Krahmer, E., te Riele, S., Veldhuis, R.N., 2000. Preferred modalities in dialog systems. In: Proceedings of the Interspeech, pp. 727-730.

[7]

J. Bohn, V. Coroama, M. Langheinrich, F. Mattern, M. Rohs, Social, economic, and ethical implications of ambient intelligence and ubiquitous computing, Ambient Intell. (2005) 5-29.

[8]

Bolt R.A., 1980. "Put-that-there": voice and gesture at the graphics interface, In: Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '80), ACM, pp. 262-270

[9]

Card, S., Mackinlay, J., Robertson, G., 1990. The design space of input devices. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering People, ACM, pp. 117-124.

[10]

Y.S. Chow, H. Teicher, Probability Theory: Independence, Interchangeability, Martingales, Springer, New York, 2003.

[11]

A.K. Dey, J. Häkkilä, Context-awareness and mobile devices, User Interface Design and Evaluation for Mobile Technology, 1 (2008) 205-217.

[12]

Engelbrecht, K.P., Kruppa, M., Mller, S., Quade, M., 2008. Memo workbench for semi-automated usability testing. In: Proceedings of the 9th Annual Conference of the International Speech Communication Association (Interspeech ¿08), pp. 1662-1665.

[13]

J. Fiscus, J. Ajot, J. Garofolo, The rich transcription 2007 meeting recognition evaluation, Multimodal Technologies for Perception of Humans (2008) 373-389.

[14]

Franz, A., Henzinger, M., Brin, S., Milch, B., 2006. Voice interface for a search engine. US Patent 7,027,987.

[15]

W.T. Fu, W.D. Gray, Suboptimal tradeoffs in information seeking, Cogn. Psychol., 52 (2006) 195-242.

[16]

W. Gray, C. Sims, W. Fu, M. Schoelles, The soft constraints hypothesis: a rational analysis approach to resource allocation for interactive behavior, Psychol. Rev., 113 (2006) 461.

[17]

M. Hassenzahl, M. Burmester, F. Koller, AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität, In Mensch & Computer, 2003 (2003) 187-196.

[18]

K.S. Hone, R. Graham, Towards a tool for the subjective assessment of speech system interfaces (SASSI), Nat. Lang. Eng., 6 (2000) 287-305.

[19]

ISO, W., 1998. 9241-11. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs), The International Organization for Standardization.

[20]

M.Y. Ivory, M.A. Hearst, The state of the art in automating usability evaluation of user interfaces, ACM Trans. Comput.-Hum. Interact., 33 (2001) 470-516.

[21]

John, B.E., Prevas, K., Salvucci, D.D., Koedinger, K., 2004. Predictive human performance modeling made easy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 455-462.

[22]

B.E. John, D.E. Kieras, Using GOMS for user interface design and evaluation: which technique?, ACM Trans. Comput.-Hum. Interact. (TOCHI), 3 (1996) 287-319.

[23]

P.W. Jordan, Designing Pleasurable Products: An Introduction to the New Human Factors, Taylor & Francis, London, 2002.

[24]

D. Kahneman, Attention and Effort, Prentice-Hall, Englewood Cliffs, NJ, 1973.

[25]

Kahneman, D., 2011. Thinking, Fast and Slow, Farrar, Straus and Giroux.

[26]

D. Kieras, S. Wood, D. Meyer, Predictive engineering models based on the EPIC architecture for a multimodal high-performance human-computer interaction task, ACM Trans. Comput.-Hum. Interact., 4 (1997) 230-275.

[27]

Kühnel, C., Westermann, T., Weiss, B., Möller, S., 2010. Evaluating multimodal systems: a comparison of established questionnaires and interaction parameters. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, ACM, pp. 286-294.

[28]

Metze, F., Wechsung, I., Schaffer, S., Seebode, J. and Möller, S., 2009. Reliable evaluation of multimodal dialog systems. In: Proceedings of the 13th International Conference on Human-Computer Interaction (HCII ¿09), pp. 75-83.

[29]

Möller, S., Kühnel, C., Weiss, B., 2011. Parameters describing the interaction with multimodal dialog systems. ITU-T Contribution.

[30]

Möller, S., Englert, R., Engelbrecht, K., Hafner, V., Jameson, A., Oulasvirta, A., Raake, A., Reithinger, N., 2006. MeMo: towards automatic usability evaluation of spoken dialog services by user error simulations. In: Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech ¿06), pp. 1786-1789.

[31]

Morrison, J.E., 2003. A review of computer-based human behavior representations and their relation to military simulations. Technical Report DTIC Document.

[32]

Nigay, L. and Coutaz, J., 1993. A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of the INTERACT¿93 and CHI¿93 Conference on Human Factors in Computing Systems, pp. 172-178.

[33]

Perakakis, M., Potamianos, A., 2008. Multimodal system evaluation using modality efficiency and synergy metrics. In: Proceedings of the 10th International Conference on Multimodal Interfaces, ACM, pp. 9-16.

[34]

Rauterberg, M., 1996. A Petri net based analyzing and modeling tool kit for logfiles in human computer interaction. In: Proceedings of the Cognitive Systems Engineering in Process Control, pp. 268-275.

[35]

W. Reisig, G. Rozenberg, Lectures on Petri Nets I: Basic Models: Advances in Petri Nets, Springer, 1998.

[36]

X. Ren, G. Zhang, G. Dai, An experimental study of input modes for multimodal human-computer interaction, Adv. Multimodal Interfaces-ICMI, 2000 (2000) 49-56.

[37]

Rudnicky, A.I., 1993. Mode preference in a simple data-retrieval task, Association for Computational Linguistics. In: Proceedings of the Workshop on Human Language Technology, Association for Computational Linguistics, pp. 364-369.

[38]

D.D. Salvucci, Rapid prototyping and evaluation of in-vehicle interfaces, ACM Trans. Comput.-Hum. Interact., 16 (2009) 9-33.

[39]

Schaffer, S., Jöckel, B., Wechsung, I., Schleicher, R., Möller, S., 2011a. Modality selection and perceived mental effort in a mobile application. In: Proceedings of the 12th Annual Conference of the International Speech Communication Association, Interspeech 2011, pp. 2253-2256.

[40]

Schaffer, S., Minge, M., 2012. Error-prone voice and graphical user interfaces in a mobile application. In: Proceedings of the ITG Symposium on Speech Communication 10, pp. 135-139.

[41]

Schaffer, S., Reitter, D., 2012. Modeling efficiency-guided modality choice in voice and graphical user interfaces. In: ICCM 2012 Proceedings, p. 253.

[42]

S. Schaffer, R. Schleicher, S. Möller, Measuring cognitive load for different input modalities, 2011.

[43]

Schleicher, R., Wechsung, I., 2012. Modelling modality choice using task parameters and perceived quality. In: Proceedings of ITG Symposium on Speech Communication, 10, pp. 1-4.

[44]

T.B. Sheridan, R. Parasuraman, Human versus automation in responding to failures: an expected-value analysis, Hum. Factors: J. Hum. Factors Ergon. Soc., 42 (2000) 403-407.

[45]

Suhm, B., Myers, B., Waibel, A., 1999. Model-based and empirical evaluation of multimodal interactive error correction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: the CHI is the Limit, ACM, pp. 584-591.

[46]

J. Sweller, Cognitive load during problem solving: effects on learning, Cogn. Sci., 12 (1988) 257-285.

[47]

Turunen, M., Hakulinen, J., Heimonen, T., 2010. Assessment of spoken and multimodal applications: lessons learned from laboratory and field studies. In: Proceedings of the Interspeech, pp. 1333-1336.

[48]

I. Varga, S. Aalburg, B. Andrassy, S. Astrov, J.G. Bauer, C. Beaugeant, Gei, H. Hoge, ASR in mobile phones-an industrial approach, IEEE Trans. Speech Audio Process., 10 (2002) 562-569.

[49]

Wechsung, I., Engelbrecht, K.-P., Möller, S., 2012. Using quality ratings to predict modality choice in multimodal systems. In: Proceedings of the Interspeech 2012: the 13th Annual Conference of the International Speech Communication Association, ISCA. pp. 1-5.

[50]

I. Wechsung, K.P. Engelbrecht, A. Naumann, S. Möller, S. Schaffer, R. Schleicher, Investigating modality selection strategies, I, Spoken Language Technology Workshop (SLT) (2010) 31-36.

[51]

Wechsung, I., Engelbrecht, K.-P., Schaffer, S., Seebode, J., Metze, F., Möller, S., 2009. Usability evaluation of multimodal interfaces: is the whole the sum of its parts? In: Proceedings of the 13th International Conference on Human-Computer Interaction, pp. 113-119.

[52]

C.D. Wickens, J.G. Hollands, Engineering Psychology and Human Performance, Prentice-Hall, Upper Saddle River, EUA, 2000.

Cited By

Chhibber NGoh JLaw E(2022)Teachable Conversational Agents for Crowdwork: Effects on Performance and TrustProceedings of the ACM on Human-Computer Interaction10.1145/35552236:CSCW2(1-21)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555223
Heck MShon SBecker C(2022)Does Using Voice Authentication in Multimodal Systems Correlate With Increased Speech Interaction During Non-critical Routine Tasks?Proceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511129(868-877)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511129
Baxter MBleakley AEdwards JClark LCowan BWilliamson J(2021)“You, Move There!”: Investigating the Impact of Feedback on Voice Control in Virtual EnvironmentsProceedings of the 3rd Conference on Conversational User Interfaces10.1145/3469595.3469609(1-9)Online publication date: 27-Jul-2021
https://dl.acm.org/doi/10.1145/3469595.3469609
Show More Cited By

Modeling input modality choice in mobile graphical and speech interfaces
1. Human-centered computing

Recommendations

Mixed-initiative dialog management for speech-based interaction with graphical user interfaces
CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Controlling graphical user interfaces (GUI) by speech is slow, but proves useful for disabled persons with limitations in operating mouse and keyboard. We present conversation-and-control, a new approach for using speech as input modality for GUIs, ...
On timing and modality choice with local danger warnings for drivers
AutomotiveUI '09: Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications

We present an experimental study on the effectiveness of five modality variants (speech, text-only, icon-only, two combinations of text and icons) for presenting local danger warnings for drivers. Hereby, we focus on sudden appearing road obstacles ...
A proposal for the development of adaptive spoken interfaces to access the Web

Spoken dialog systems have been proposed as a solution to facilitate a more natural human-machine interaction. In this paper, we propose a framework to model the user s intention during the dialog and adapt the dialog model dynamically to the user needs ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image International Journal of Human-Computer Studies

International Journal of Human-Computer Studies Volume 75, Issue C

March 2015

52 pages

ISSN:1071-5819

Issue’s Table of Contents

Copyright © Elsevier Ltd.

Publisher

Academic Press, Inc.

United States

Publication History

Published: 01 March 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chhibber NGoh JLaw E(2022)Teachable Conversational Agents for Crowdwork: Effects on Performance and TrustProceedings of the ACM on Human-Computer Interaction10.1145/35552236:CSCW2(1-21)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555223
Heck MShon SBecker C(2022)Does Using Voice Authentication in Multimodal Systems Correlate With Increased Speech Interaction During Non-critical Routine Tasks?Proceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511129(868-877)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511129
Baxter MBleakley AEdwards JClark LCowan BWilliamson J(2021)“You, Move There!”: Investigating the Impact of Feedback on Voice Control in Virtual EnvironmentsProceedings of the 3rd Conference on Conversational User Interfaces10.1145/3469595.3469609(1-9)Online publication date: 27-Jul-2021
https://dl.acm.org/doi/10.1145/3469595.3469609
Guerino GValentim NCavalcante EDantas FBatista T(2020)Usability and User eXperience Evaluation of Conversational SystemsProceedings of the XXXIV Brazilian Symposium on Software Engineering10.1145/3422392.3422421(427-436)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3422392.3422421
Agarwal MSivakumar RBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Charge for a whole day: Extending Battery Life for BCI Wearables using a Lightweight Wake-Up CommandProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376738(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376738
Schaffer SReithinger NCowan BClark L(2019)Conversation is multimodalProceedings of the 1st International Conference on Conversational User Interfaces10.1145/3342775.3342801(1-3)Online publication date: 22-Aug-2019
https://dl.acm.org/doi/10.1145/3342775.3342801
Varghese DOlivier PBalaam M(2017)Citizen Tagger16th IFIP TC 13 International Conference on Human-Computer Interaction --- INTERACT 2017 - Volume 1051610.1007/978-3-319-68059-0_7(116-125)Online publication date: 25-Sep-2017
https://dl.acm.org/doi/10.1007/978-3-319-68059-0_7
Himmelsbach JGarschall MEgger SSteffek STscheligi MHolzmann CMayrhofer RHäkkilä JRukzio ERoland M(2015)Enabling accessibility through multimodality?Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia10.1145/2836041.2836060(195-199)Online publication date: 30-Nov-2015
https://dl.acm.org/doi/10.1145/2836041.2836060

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents