Article

A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances

Authors:

Kazuhiro Otsuka,

Yoshinao Takemae,

Junji YamatoAuthors Info & Claims

ICMI '05: Proceedings of the 7th international conference on Multimodal interfaces

Pages 191 - 198

https://doi.org/10.1145/1088463.1088497

Published: 04 October 2005 Publication History

Abstract

A novel probabilistic framework is proposed for inferring the structure of conversation in face-to-face multiparty communication, based on gaze patterns, head directions and the presence/absence of utterances. As the structure of conversation, this study focuses on the combination of participants and their participation roles. First, we assess the gaze patterns that frequently appear in conversations, and define typical types of conversation structure, called conversational regime, and hypothesize that the regime represents the high-level process that governs how people interact during conversations. Next, assuming that the regime changes over time exhibit Markov properties, we propose a probabilistic conversation model based on Markov-switching; the regime controls the dynamics of utterances and gaze patterns, which stochastically yield measurable head-direction changes. Furthermore, a Gibbs sampler is used to realize the Bayesian estimation of regime, gaze pattern, and model parameters from observed head directions and utterances. Experiments on four-person conversations confirm the effectiveness of the framework in identifying conversation structures.

References

[1]

S. Basu, T. Choudhury, and B. Clarkson. Learning human interactions with the influence model. In MIT Media Lab. TR#539, 2001.

[2]

J. M. Bernardo and A. F. M. Smith. Bayesian Theory. John Wiley & Sons, Ltd., 1994.

[3]

M. Bett, H. Y. R. Gross, X. Zhu, Y. Pan, J. Yang, and A. Waibel. Multimodal meeting tracker. In Proc. RIAO 2000: Content-Based Multimodal Inform. Access, 2000.

[4]

R. Chen and T.-H. Li. Blind restoration of linearly degraded discrete signals by Gibbs sampling. IEEE Trans. Signal Processing, 43(10):2410--2413, 1995.

Digital Library

[5]

H. H. Clark and T. B. Carlson. Hearers and speech acts. Language, 58:332--373, 1982.

[6]

R. Cutler et al. Distributed meetings: A meeting capture and broadchasting system. In Proc. ACM Multimedia '02, pages 503--512, 2002.

Digital Library

[7]

A. Dielmann and S. Renals. Dynamic Bayesian networks for meeting structuring. In Proc. IEEE ICASSP'04, 2004.

[8]

R. Diestel. Graph Theory, 2nd ed. Springer-Verlag New York, Inc., 2000.

[9]

S. Frühwirth-Schnatter. Markov chain Monte Carlo estimation of classical and dynamic switching and mixture models. J. American Statistical Association, 96(453):194--209, 2001.

[10]

W. R. Gilks, S. Richardson, and D. J. Spiegelhalter. Markov Chain Monte Carlo in Practice. Chapman & Hall/CRC, 1996.

[11]

E. Goffman. Forms of Talks. University of Pennsylvania Press, Philadelphia, 1981.

[12]

C. Goodwin. Conversational Organization : Interaction between Speakers and Hearers. Academic Press, 1981.

[13]

D. Heylen, I. V. Es, A. Nijholt, and B. V. Dijk. Experimenting with the gaze of a conversational agent. In Proc. Int. CLASS Workshop on Natural Intelligent and Effective Interaction in Multimodal Dialogue Systems, pages 93--100, 2002.

[14]

N. Jovanovic and R. Akker. Towards automatic addressee identification in multi-party dialogues. In Proc. SIGdial'04, pages 89--92, 2004.

[15]

A. Kendon. Some functions of gaze-direction in social interaction. Acta Psychologica, 26:22--63, 1967.

[16]

C.-J. Kim and C. R. Nelson. State-Space Models with Regime Switching. MIT Press, 1999.

[17]

Y. Matsumoto and A. Zelinsky. An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement. In Proc. Int. Conf. Automatic Face and Gesture Recognition '04, pages 499--504, 2000.

Digital Library

[18]

I. McCowan, D. Perez, S. Bengio, G. Lathoud, M. Barnard, and D. Zhang. Automatic analysis of multimodal group actions in meetings,. IEEE Trans. PAMI, 27(3), 2005.

Digital Library

[19]

L.-P. Morency, A. Rahimi, and T. Darrell. Adaptive view-based appearance model. In Proc. CVPR'03, pages 803--810, 2003.

Digital Library

[20]

D. G. Novic, B. Hansen, and K. Ward. Coordinating turn-taking with gaze. In Proc. Int. Conf. Spoken Language '96., pages 1888--1891, 1996.

[21]

T. Ohno and N. Mukawa. A free-head, simple calibration, gaze tracking system that enables gaze-based interaction. In Proc. Eye Tracking Research & Application Symposium (ETRA)2004, pages 115--122, 2004.

Digital Library

[22]

D. Reidsma et al. Virtual meeting rooms: From observation to simulation. In Proc. Social Intelli. Design '05, 2005.

[23]

R. Stiefelhagen, J. Yang, and A. Waibel. Modeling focus of attention for meeting index based on multiple cues. IEEE Trans. Neural Networks, 13(4), 2002.

Digital Library

[24]

R. Stiefelhagen and J. Zhu. Head orientation and gaze direction in meetings. In Proc. ACM CHI'02, 2002.

Digital Library

[25]

Y. Takemae, K. Otsuka, and N. Mukawa. Impact of video editing based on participants' gaze in multiparty conversation. In Proc. ACM CHI'04, pages 1333--1336, 2004.

Digital Library

[26]

R. Vertegaal, R. Slagter, G. Veer, and A. Nijholt. Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In Proc. ACM CHI'2001, pages 301--308, 2001.

Digital Library

[27]

D. Zhang, D. G. Perez, S. Bengio, I. McCowan, and G. Lathoud. Modeling individual and group actions in meetings: A two-layers HMM framework. In Proc. 2nd. IEEE Workshop on Event Mining, 2004.

Digital Library

Cited By

Tashiro AImamura MKumano SOtsuka K(2024)Exploring Interlocutor Gaze Interactions in Conversations based on Functional Spectrum AnalysisProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685708(86-94)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685708
Gupta AVuillecard PFarkhondeh AOdobez J(2024)Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00066(615-624)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00066
Tashiro AImamura MKumano SOtsuka K(2023)Analyzing and Recognizing Interlocutors' Gaze Functions from Multimodal Nonverbal CuesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614152(33-41)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614152
Show More Cited By

Index Terms

A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Automatic inference of cross-modal nonverbal interactions in multiparty conversations: "who responds to whom, when, and how?" from gaze, head gestures, and utterances
ICMI '07: Proceedings of the 9th international conference on Multimodal interfaces

A novel probabilistic framework is proposed for analyzing cross-modal nonverbal interactions in multiparty face-to-face conversations. The goal is to determine "who responds to whom, when, and how" from multimodal cues including gaze, head gestures, and ...
Conversation scene analysis based on dynamic Bayesian network and image-based gaze detection
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

This paper presents a probabilistic framework, which incorporates automatic image-based gaze detection, for inferring the structure of multiparty face-to-face conversations. This framework aims to infer conversation regimes and gaze patterns from the ...
Quantifying interpersonal influence in face-to-face conversations based on visual attention patterns
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing Systems

A novel measure for automatically quantifying the amount of interpersonal influence present in face-to-face conversations is proposed based on the visual-attention patterns of the participants as inferred from video sequences. First, we focus on the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '05: Proceedings of the 7th international conference on Multimodal interfaces

October 2005

344 pages

ISBN:1595930280

DOI:10.1145/1088463

General Chairs:
Gianni Lazzari
ITC-irst, Trento (Italy)
,
Fabio Pianesi
ITC-irst, Trento (Italy)
,
Program Chairs:
James Crowley
I.N.P. Grenoble (France)
,
Kenji Mase
Nagoya University (Japan)
,
Sharon Oviatt
Oregon Health & Sciences University

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 October 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI05

Sponsor:

ICMI05: Seventh International Conference on Multimodal Interfaces 2005

October 4 - 6, 2005

Torento, Italy

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

85
Total Citations
View Citations
781
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tashiro AImamura MKumano SOtsuka K(2024)Exploring Interlocutor Gaze Interactions in Conversations based on Functional Spectrum AnalysisProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685708(86-94)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685708
Gupta AVuillecard PFarkhondeh AOdobez J(2024)Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00066(615-624)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00066
Tashiro AImamura MKumano SOtsuka K(2023)Analyzing and Recognizing Interlocutors' Gaze Functions from Multimodal Nonverbal CuesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614152(33-41)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614152
Córdoba-Tlaxcalteco MBenítez-Guerrero E(2023)Human Event Recognition in Smart Classrooms Using Computer Vision: A Systematic Literature ReviewProgramming and Computing Software10.1134/S036176882308006649:8(625-642)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1134/S0361768823080066
Lu HBrimijoin W(2022)Sound Source Selection Based on Head Movements in Natural Group ConversationTrends in Hearing10.1177/2331216522109778926Online publication date: 28-Apr-2022
https://doi.org/10.1177/23312165221097789
Jin ADeng QDeng Z(2022)S2M-Net: Speech Driven Three-party Conversational Motion Synthesis NetworksProceedings of the 15th ACM SIGGRAPH Conference on Motion, Interaction and Games10.1145/3561975.3562954(1-10)Online publication date: 3-Nov-2022
https://dl.acm.org/doi/10.1145/3561975.3562954
Skowronek JRaake ABerndtsson GRummukainen OUsai PGunkel SJohanson MHabets EMalfait LLindero DToet A(2022)Quality of Experience in Telemeetings and Videoconferencing: A Comprehensive SurveyIEEE Access10.1109/ACCESS.2022.317636910(63885-63931)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3176369
Xu BSong A(2022)Modelling Eye-Gaze Movement Using Gaussian Auto-regression Hidden MarkovAI 2021: Advances in Artificial Intelligence10.1007/978-3-030-97546-3_16(190-202)Online publication date: 19-Mar-2022
https://doi.org/10.1007/978-3-030-97546-3_16
Clarfeld LGramling RRizzo DEppstein M(2021)A general model of conversational dynamics and an example application in serious illness communicationPLOS ONE10.1371/journal.pone.025312416:7(e0253124)Online publication date: 1-Jul-2021
https://doi.org/10.1371/journal.pone.0253124
Tian L(2021)Improved Gazing Transition Patterns for Predicting Turn-Taking in Multiparty ConversationProceedings of the 2021 5th International Conference on Video and Image Processing10.1145/3511176.3511208(215-219)Online publication date: 22-Dec-2021
https://dl.acm.org/doi/10.1145/3511176.3511208
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents