Article

Human perception of intended addressee during computer-assisted meetings

Authors:

Rebecca Lunsford,

Sharon OviattAuthors Info & Claims

ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

Pages 20 - 27

https://doi.org/10.1145/1180995.1181002

Published: 02 November 2006 Publication History

Get Access

Abstract

Recent research aims to develop new open-microphone engagement techniques capable of identifying when a speaker is addressing a computer versus human partner, including during computer-assisted group interactions. The present research explores: (1) how accurately people can judge whether an intended interlocutor is a human versus computer, (2) which linguistic, acoustic-prosodic, and visual information sources they use to make these judgments, and (3) what type of systematic errors are present in their judgments. Sixteen participants were asked to determine a speaker's intended addressee based on actual videotaped utterances matched on illocutionary force, which were played back as: (1) lexical transcriptions only, (2) audio-only, (3) visual-only, and (4) audio-visual information. Perhaps surprisingly, people's accuracy in judging human versus computer addressees did not exceed chance levels with lexical-only content (46%). As predicted, accuracy improved significantly with audio (58%), visual (57%), and especially audio-visual information (63%). Overall, accuracy in detecting human interlocutors was significantly worse than judging computer ones, and specifically worse when only visual information was present because speakers often looked at the computer when addressing peers. In contrast, accuracy in judging computer interlocutors was significantly better whenever visual information was present than with audio alone, and it yielded the highest accuracy levels observed (86%). Questionnaire data also revealed that speakers' gaze, peers' gaze, and tone of voice were considered the most valuable information sources. These results reveal that people rely on cues appropriate for interpersonal interactions in determining computer- versus human-directed speech during mixed human-computer interactions, even though this degrades their accuracy. Future systems that process actual rather than expected communication patterns potentially could be designed that perform better than humans.

References

[1]

Arthur, A.M., Lunsford, R., Wesson, M., & Oviatt, S., Prototyping novel collaborative multimodal systems: Simulation, data collection, and analysis tools for the next decade. In press,ICMI 2006.

Digital Library

Google Scholar

[2]

Jovanovic, N., op den Akker, R., & Nijholt, A. Addressee identification in face-to-face meetings. In Proceedings of the 11th Conference of the European Chapter of the ACL (EACL), 2006 (Trento, Italy). ACL: 169--176.

Google Scholar

[3]

Katzenmaier, M., Stiefelhagen, R., & Schultz, T. Identifying the addressee in human-human-robot interactions based on head pose and speech. In Proceedings of the International Conference on Multimodal Interfaces, 2004 (State College, PA). ACM Press: 144--151.

Digital Library

Google Scholar

[4]

Lunsford, R. & Oviatt, S., Toward open-microphone engagement for multiparty field interactions. In press, ICMI 2006.

Digital Library

Google Scholar

[5]

Lunsford, R., Oviatt, S.L., & Coulston, R. Audio-visual cues distinguishing self- from system-directed speech in younger and older adults. In Proceedings of the International Conference on Multimodal Interfaces, 2005 (Trento, Italy). ACM Press: 265--272.

Digital Library

Google Scholar

[6]

Rienks, R., Poppe, R., & Heylen, D., Differences in head orientation between speakers and listeners in in multi-party conversations. In submission to Elsevier Science, (March 4, 2005).

Google Scholar

[7]

van Turnhout, K., Terken, J., Bakx, I., & Eggen, B. Identifying the intended addressee in mixed human-human and human-computer interaction from non-verbal features. In Proceedings of the 7th International Conference on Multimodal interfaces, 2005 (Trento, Italy). ACM Press: 175--182.

Digital Library

Google Scholar

[8]

Vertegaal, R., Look who is talking to whom, Ph.D. Thesis, September 1998 Cognitive Ergonomics, Twente University.

Google Scholar

Cited By

View all

Huseynov F(2023)Chatbots in Digital MarketingContemporary Approaches of Digital Marketing and the Role of Machine Intelligence10.4018/978-1-6684-7735-9.ch003(46-72)Online publication date: 30-Jun-2023
https://doi.org/10.4018/978-1-6684-7735-9.ch003
Tesema FGu JSong WWu HZhu SLin ZHuang MWang WKumar R(2023)Addressee Detection Using Facial and Audio Features in Mixed Human–Human and Human–Robot Settings: A Deep Learning FrameworkIEEE Systems, Man, and Cybernetics Magazine10.1109/MSMC.2022.32248439:2(25-38)Online publication date: Apr-2023
https://doi.org/10.1109/MSMC.2022.3224843
Siegert IWeißkirchen NKrüger JAkhtiamov OWendemuth A(2022)Admitting the addressee detection faultiness of voice assistants to improve the activation performance using a continuous learning frameworkCognitive Systems Research10.1016/j.cogsys.2021.07.00570:C(65-79)Online publication date: 22-Apr-2022
https://dl.acm.org/doi/10.1016/j.cogsys.2021.07.005
Show More Cited By

Index Terms

Human perception of intended addressee during computer-assisted meetings
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Sound-based input / output
2. Human-centered computing
  1. Human computer interaction (HCI)
  2. Interaction design
    1. Interaction design process and methods
      1. User centered design
    2. Interaction design theory, concepts and paradigms

Recommendations

Toward open-microphone engagement for multiparty interactions
ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

There currently is considerable interest in developing new open-microphone engagement techniques for speech and multimodal interfaces that perform robustly in complex mobile and multiparty field environments. State-of-the-art audio-visual open-...
Spontaneous spoken dialogues with the furhat human-like robot head
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction

Furhat [1] is a robot head that deploys a back-projected animated face that is realistic and human-like in anatomy. Furhat relies on a state-of-the-art facial animation architecture allowing accurate synchronized lip movements with speech, and the ...
Audio-visual cues distinguishing self- from system-directed speech in younger and older adults
ICMI '05: Proceedings of the 7th international conference on Multimodal interfaces

In spite of interest in developing robust open-microphone engagement techniques for mobile use and natural field contexts, there currently are no reliable techniques available. One problem is the lack of empirically-grounded models as guidance for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

November 2006

404 pages

ISBN:159593541X

DOI:10.1145/1180995

General Chairs:
Francis Quek
Virginia Tech, USA
,
Jie Yang
Carnegie Mellon University, USA
,
Program Chairs:
Dominic Massaro
University of California, Santa Cruz, USA
,
Abeer Alwan
University of California, Los Angeles, USA
,
Timothy J. Hazen
Massachusetts Institute of Technology, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI06

Sponsor:

ICMI06: 8th International Conference on Multimodal Interfaces 2006

November 2 - 4, 2006

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
275
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Huseynov F(2023)Chatbots in Digital MarketingContemporary Approaches of Digital Marketing and the Role of Machine Intelligence10.4018/978-1-6684-7735-9.ch003(46-72)Online publication date: 30-Jun-2023
https://doi.org/10.4018/978-1-6684-7735-9.ch003
Tesema FGu JSong WWu HZhu SLin ZHuang MWang WKumar R(2023)Addressee Detection Using Facial and Audio Features in Mixed Human–Human and Human–Robot Settings: A Deep Learning FrameworkIEEE Systems, Man, and Cybernetics Magazine10.1109/MSMC.2022.32248439:2(25-38)Online publication date: Apr-2023
https://doi.org/10.1109/MSMC.2022.3224843
Siegert IWeißkirchen NKrüger JAkhtiamov OWendemuth A(2022)Admitting the addressee detection faultiness of voice assistants to improve the activation performance using a continuous learning frameworkCognitive Systems Research10.1016/j.cogsys.2021.07.00570:C(65-79)Online publication date: 22-Apr-2022
https://dl.acm.org/doi/10.1016/j.cogsys.2021.07.005
Akhtiamov OSiegert IKarpov AMinker W(2020)Using Complexity-Identical Human- and Machine-Directed Utterances to Investigate Addressee Detection for Spoken Dialogue SystemsSensors10.3390/s2009274020:9(2740)Online publication date: 11-May-2020
https://doi.org/10.3390/s20092740
Siegert IKrüger J(2020)“Speech Melody and Speech Content Didn’t Fit Together”—Differences in Speech Behavior for Device Directed and Human Directed InteractionsAdvances in Data Science: Methodologies and Applications10.1007/978-3-030-51870-7_4(65-95)Online publication date: 27-Aug-2020
https://doi.org/10.1007/978-3-030-51870-7_4
Tsai TStolcke ASlaney M(2015)A Study of Multimodal Addressee Detection in Human-Human-Computer InteractionIEEE Transactions on Multimedia10.1109/TMM.2015.245433217:9(1550-1561)Online publication date: Sep-2015
https://doi.org/10.1109/TMM.2015.2454332
Tsai TStolcke ASlaney M(2015)Multimodal addressee detection in multiparty dialogue systems2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2015.7178384(2314-2318)Online publication date: Apr-2015
https://doi.org/10.1109/ICASSP.2015.7178384
Baba NHuang HNakano Y(2013)Identifying the Addressee using Head Orientation and Speech Information in Multiparty Human-Agent ConversationsTransactions of the Japanese Society for Artificial Intelligence10.1527/tjsai.28.14928:2(149-159)Online publication date: 2013
https://doi.org/10.1527/tjsai.28.149
Nakano YBaba NHuang HHayashi YEpps JChen FOviatt SMase KSears AJokinen KSchuller B(2013)Implementation and evaluation of a multimodal addressee identification mechanism for multiparty conversation systemsProceedings of the 15th ACM on International conference on multimodal interaction10.1145/2522848.2522872(35-42)Online publication date: 9-Dec-2013
https://dl.acm.org/doi/10.1145/2522848.2522872
Hajishirzi HLehman JHodgins JLee GGinzburg J(2012)Using group history to identify character-directed utterances in multi-child interactionsProceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue10.5555/2392800.2392838(207-216)Online publication date: 5-Jul-2012
https://dl.acm.org/doi/10.5555/2392800.2392838
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Toward open-microphone engagement for multiparty interactions

Spontaneous spoken dialogues with the furhat human-like robot head

Audio-visual cues distinguishing self- from system-directed speech in younger and older adults