research-article

Estimating user's engagement from eye-gaze behaviors in human-agent conversations

Authors:

Yukiko I. Nakano,

Ryo IshiiAuthors Info & Claims

IUI '10: Proceedings of the 15th international conference on Intelligent user interfaces

Pages 139 - 148

https://doi.org/10.1145/1719970.1719990

Published: 07 February 2010 Publication History

Abstract

In face-to-face conversations, speakers are continuously checking whether the listener is engaged in the conversation and change the conversational strategy if the listener is not fully engaged in the conversation. With the goal of building a conversational agent that can adaptively control conversations with the user, this study analyzes the user's gaze behaviors and proposes a method for estimating whether the user is engaged in the conversation based on gaze transition 3-gram patterns. First, we conduct a Wizard-of-Oz experiment to collect the user's gaze behaviors. Based on the analysis of the gaze data, we propose an engagement estimation method that detects the user's disengagement gaze patterns. The algorithm is implemented as a real-time engagement-judgment mechanism and is incorporated into a multimodal dialogue manager in a conversational agent. The agent estimates the user's conversational engagement and generates probing questions when the user is distracted from the conversation. Finally, we conduct an evaluation experiment using the proposed engagement-sensitive agent and demonstrate that the engagement estimation function improves the user's impression of the agent and the interaction with the agent. In addition, probing performed with proper timing was also found to have a positive effect on user's verbal/nonverbal behaviors in communication with the conversational agent.

References

[1]

Sidner, C.L., et al., Explorations in engagement for humans and robots. Artificial Intelligence, (2005). 166(1-2): pp. 140--164.

Digital Library

[2]

Peters, C. Direction of Attention Perception for Conversation Initiation in Virtual Environments. in Intelligent Virtual Agents. (2005). p. 215--228.

Digital Library

[3]

Sidner, C.L., et al. Where to Look: A Study of Human-Robot Engagement. in ACM International Conference on Intelligent User Interfaces (IUI). (2004). p. 78--84.

Digital Library

[4]

Argyle, M. and Cook, M., Gaze and Mutual Gaze. (1976), Cambridge: Cambridge University Press.

[5]

Duncan, S., On the structure of speaker-auditor interaction during speaking turns. Language in Society, (1974). 3: pp. 161--180.

[6]

Novick, D.G., Hansen, B., and Ward, K. Coordinating turn-taking with gaze. in ICSLP-96. (1996). Philadelphia, PA. p. 1888--1891.

[7]

Kendon, A., Some Functions of Gaze Direction in Social Interaction. Acta Psychologica, (1967). 26: pp. 22--63.

[8]

Argyle, M., et al., The different functions of gaze. Semiotica, (1973). 7: pp. 19--32.

[9]

Argyle, M. and Graham, J., The Central Europe Experiment - looking at persons and looking at things. Journal of Environmental Psychology and Nonverbal Behaviour, (1977). 1: pp. 6--16.

[10]

Anderson, A.H., et al., The effects of face-to-face communication on the intelligibility of speech. Perception and Psychophysics, (1997). 59: pp. 580--592.

[11]

Prasov, Z. and Chai, J.Y. What's in a Gaze? The Role of Eye-Gaze in Reference Resolution in Multimodal Conversational Interfaces. in the 13th international conference on Intelligent user interfaces (2008). p. 20--29.

Digital Library

[12]

Qvarfordt, P. and Zhai, S. Conversing with the User Based on Eye-Gaze Patterns. in the Conference on Human-Factors in Computing Systems, CHI 2005. (2005).

Digital Library

[13]

Eichner, T., et al. Attentive Presentation Agents. in The 7th International Conference on Intelligent Virtual Agents (IVA). (2007). p. 283--295.

Digital Library

[14]

Nakano, I.Y. and Nishida, T., Attentional Behaviors as Nonverbal Communicative Signals in Situated Interactions with Conversational Agents, in Engineering Approaches to Conversational Informatics, Nishida, T., Editor. (2007), John Wiley & Sons Inc.

[15]

Pelachaud, C. and Bilvi, M. Modelling Gaze Behavior for Conversational Agents. in IVA03 International Working Conference on Intelligent Virtual Agents. (2003). Germany.

[16]

Gratch, J., et al., Virtual Rapport, in 6th International Conference on Intelligent Virtual Agents. (2006), Springer: Marina del Rey, CA.

Digital Library

[17]

Nakano, Y.I., et al. Towards a Model of Face-to-Face Grounding. in the 41st Annual Meeting of the Association for Computational Linguistics (ACL03). (2003). Sapporo, Japan. p. 553--561.

Digital Library

[18]

Morency, L.-P., Kok, I.d., and Gratch, J. Predicting Listener Backchannels: A Probabilistic Multimodal Approach. in The 8th International Conference Intelligent Virtual Agents (IVA'08). (2008): Springer. p. 176--190.

Digital Library

[19]

Morency, L.-P., et al., Head gestures for perceptual interfaces: The role of context in improving recognition. Artificial Intelligence (2007). 171(8-9): pp. 568--585.

Digital Library

[20]

Bohus, D. and Horvitz, E. Learning to Predict Engagement with a Spoken Dialog System in Open-World Settings. in SIGdial'09. (2009). London, UK.

Digital Library

[21]

Kendon, A., Spatial organization in social encounters: the F-formation system, Conducting Interaction: Patterns of behavior in focused encounters. Studies in International Sociolinguistics, ed. Gumperz, J.J. (1990): Cambridge University Press.

[22]

Schober, M.F. and Clark, H.H., Understanding by addressees and overhearers. Cognitive Psychology, (1989). 21: pp. 211--232.

[23]

Ishii, R. and Nakano, Y. Estimating User's Conversational Engagement based on Gaze Behaviors. in The 8th International Conference Intelligent Virtual Agents (IVA'08). (2008): Springer. p. 200--207.

Digital Library

[24]

Matheson, C., Poesio, M., and Traum, D. Modelling Grounding and Discourse Obligations Using Update Rules. in 1st Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL2000). (2000). p. 1--8.

Digital Library

[25]

Nakano, Y.I., et al. Converting Text into Agent Animations: Assigning Gestures to Text. in Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2004), Companion Volume. (2004). Boston. p. 153--156.

Digital Library

[26]

Larsson, S., et al., TrindiKit 1.0 (Manual). (1999). p. http://www.ling.gu.se/projekt/trindi//.

[27]

MIDIKI. {cited; Available from: http://midiki.sourceforge.net/.

Cited By

Funk MOkada SAndré E(2024)Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on EngagementProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685757(224-233)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685757
Stefanov KNakano YKobayashi CHoshina ISakato TNihei FTakayama CIshii RTsujii M(2024)Participation Role-Driven Engagement Estimation of ASD Individuals in Neurodiverse Group DiscussionsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685721(556-564)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685721
Ji KPathiyan Cherumanal STrippas JHettiachchi DSalim FScholer FSpina D(2024)Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational SearchAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680245(1-10)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3640471.3680245
Show More Cited By

Index Terms

Estimating user's engagement from eye-gaze behaviors in human-agent conversations
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes
CHI '01: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

In multi-agent, multi-user environments, users as well as agents should have a means of establishing who is talking to whom. In this paper, we present an experiment aimed at evaluating whether gaze directional cues of users could be used for this ...
An empirical study of eye-gaze behaviors: towards the estimation of conversational engagement in human-agent communication
EGIHMI '10: Proceedings of the 2010 workshop on Eye gaze in intelligent human machine interaction

In face-to-face conversations, speakers are continuously checking whether the listener is engaged in the conversation by monitoring the partner's eye-gaze behaviors. In this study, focusing on eye-gaze as information of estimating user's conversational ...
Gaze awareness in conversational agents: Estimating a user's conversational engagement from eye gaze
Special issue on interaction with smart objects, Special section on eye gaze and conversation

In face-to-face conversations, speakers are continuously checking whether the listener is engaged in the conversation, and they change their conversational strategy if the listener is not fully engaged. With the goal of building a conversational agent ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '10: Proceedings of the 15th international conference on Intelligent user interfaces

February 2010

460 pages

ISBN:9781605585154

DOI:10.1145/1719970

General Chairs:
Charles Rich
Worcester Polytechnic Institute, USA
,
Qiang Yang
Hong Kong University of Science & Technology, China
,
Program Chairs:
Marc Cavazza
Teesside University, UK
,
Michelle Zhou
IBM Research, China

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 February 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI '10

Sponsor:

IUI '10: 15th International Conference on Intelligent User Interfaces

February 7 - 10, 2010

Hong Kong, China

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

113
Total Citations
View Citations
1,676
Total Downloads

Downloads (Last 12 months)118
Downloads (Last 6 weeks)8

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Funk MOkada SAndré E(2024)Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on EngagementProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685757(224-233)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685757
Stefanov KNakano YKobayashi CHoshina ISakato TNihei FTakayama CIshii RTsujii M(2024)Participation Role-Driven Engagement Estimation of ASD Individuals in Neurodiverse Group DiscussionsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685721(556-564)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685721
Ji KPathiyan Cherumanal STrippas JHettiachchi DSalim FScholer FSpina D(2024)Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational SearchAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680245(1-10)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3640471.3680245
Grootjen JWeingärtner HMayer S(2024)Uncovering and Addressing Blink-Related Challenges in Using Eye Tracking for Interactive SystemsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642086(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642086
Nagasawa FOkada SIshihara TNitta K(2024)Adaptive Interview Strategy Based on Interviewees’ Speaking Willingness Recognition for Interview RobotsIEEE Transactions on Affective Computing10.1109/TAFFC.2023.330964015:3(942-957)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TAFFC.2023.3309640
Salam HCeliktutan OGunes HChetouani M(2024)Automatic Context-Aware Inference of Engagement in HMI: A SurveyIEEE Transactions on Affective Computing10.1109/TAFFC.2023.327870715:2(445-464)Online publication date: Apr-2024
https://doi.org/10.1109/TAFFC.2023.3278707
Wakita KShimada K(2024)An Utterance is Enough to the gaze? Gaze Detection from Utterance Information in Multiparty Discussion2024 International Conference on Activity and Behavior Computing (ABC)10.1109/ABC61795.2024.10651736(1-8)Online publication date: 29-May-2024
https://doi.org/10.1109/ABC61795.2024.10651736
Juvina IO’Neill KCarson JMenke PWong CMcNett HHolsinger G(2024)Human-AI Coordination to Induce Flow in Adaptive Learning SystemsAI Approaches for Designing and Evaluating Interactive Intelligent Systems10.1007/978-3-031-53957-2_7(139-162)Online publication date: 10-Apr-2024
https://doi.org/10.1007/978-3-031-53957-2_7
Pellet-Rostaing ABertrand RBoudin ARauzy SBlache P(2023)A multimodal approach for modeling engagement in conversationFrontiers in Computer Science10.3389/fcomp.2023.10623425Online publication date: 2-Mar-2023
https://doi.org/10.3389/fcomp.2023.1062342
Guhan PAwasthi NMcDonald KBussell KReeves GManocha DBera A(2023)Developing Machine Learning based Effective and Automated Patient Engagement Estimator for Telehealth: Algorithm Development and Validation Study (Preprint)JMIR Formative Research10.2196/46390Online publication date: 9-Feb-2023
https://doi.org/10.2196/46390
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents