research-article

Conversation scene analysis based on dynamic Bayesian network and image-based gaze detection

Authors:

Sebastian Gorga,

Kazuhiro OtsukaAuthors Info & Claims

ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

Article No.: 54, Pages 1 - 8

https://doi.org/10.1145/1891903.1891969

Published: 08 November 2010 Publication History

Abstract

This paper presents a probabilistic framework, which incorporates automatic image-based gaze detection, for inferring the structure of multiparty face-to-face conversations. This framework aims to infer conversation regimes and gaze patterns from the nonverbal behaviors of meeting participants, which are captured from image and audio streams with cameras and microphones. The conversation regime corresponds to a global conversational pattern such as monologue and dialogue, and the gaze pattern indicates "who is looking at whom". Input nonverbal behaviors include presence/absence of utterances, head directions, and discrete head-centered eye-gaze directions. In contrast to conventional meeting analysis methods that focus only on the participant's head pose as a surrogate of visual focus of attention, this paper newly incorporates vision-based gaze detection combined with head pose tracking into a probabilistic conversation model based on dynamic Bayesian network. Our gaze detector is able to differentiate 3 to 5 different eye gaze directions, e.g. left, straight and right. Experiments on four-person conversations confirm the power of the proposed framework in identifying conversation structure and in estimating gaze patterns with higher accuracy then previous models.

References

[1]

M. Argyle. Bodily Communication -- 2nd ed. Routledge, London and New York, 1988.

[2]

M. Argyle and M. Cook. Gaze and Mutual Gaze. Cambridge University Press, Cambridge, 1976.

[3]

S. O. Ba and J.-M. Odobez. Multi-party Focus of Attention Recognition in Meetings from Head Pose and Multimodal Contextual Cues In ICASSP, 2008

[4]

J. M. Bernardo and A. F. M. Smith. Bayesian Theory. John Wiley & Sons, Ltd., 1994.

[5]

M. S. Choi and W. Y. Kim. A novel two stage template matching method for rotation and illumination invariance. In Pattern Recognition 35, pages 119--129, 2002.

[6]

A. Doshi and M. M Trivedi. Head and Gaze Dynamics in Visual Attention and Context Learning. In IEEE CVPR Joint Workshop for Visual and Contextual Learning and Visual Scene Understanding, June 2009

[7]

A. T. Duchowski. Eye Tracking Methodology: Theory and Practice. Springer, 2007.

Digital Library

[8]

S. Duncan. Some signals and rules for taking speaking turns in conversations. In Journal of Personality and Social Psychology, 23(2), 283--292, 1972.

[9]

S. Fruhwirth-Schnatter. Finite Mixture and Markov Switching Models. Springer, 2006.

[10]

D. Gatica-Perez. Automatic nonverbal analysis of social interaction in small groups: A review. In Image and Vision Computing, 27 pages 1775--1787, 2009.

Digital Library

[11]

W. R. Gilks, S. Richardson and D. J. Spiegelhalter. Markov Chain Monte Carlo in Practice. Chapman & Hall/CRC, 1996.

[12]

C. Goodwin. Conversational Organization: Interaction between Speakers and Hearers. Academic Press, 1981.

[13]

C.-W. Hsu, C.-C. Chang, C.-J. Lin. A Practical Guide to Support Vector Classification. Technical Report, Department of Computer Science & Information Engineering, National Taiwan University, 2008.

[14]

A. Kendon. Some functions of gaze-direction in social interaction. In Acta Psychologica, 26: pages 22--63, 1967.

[15]

C.-J. Kim and C. R. Nelson. State-Space Models with Regime Switching. MIT Press, 1999.

[16]

G. Loy and A. Zelinsky. Fast Radial Symmetry for Detecting Points of Interest. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 8, August 2003.

Digital Library

[17]

O. M. Lozano and K. Otsuka. Real-time Visual Tracker by Stream Processing: Simultaneous and Fast 3D Tracking of Multiple Faces in Video Sequences by Using a Particle Filter. In Journal of Signal Processing Systems, Vol. 57, No. 2 pages 285--295, November, 2009

Digital Library

[18]

P. Morasso, G. Sandini, V. Tagliasco and R. Zaccaria. Control Strategies in the Eye-Head Coordination System. In IEEE Trans. Systems, Man and Cibernetics, Vol SMC-7 No. 9, pages 639--651, 1977.

[19]

L. P. Morency, C. M. Christoudias, T. Darrell. Recognizing gaze aversion gestures in embodied conversational discourse. In Proc. ICMI'06, pages 287--294, 2006.

Digital Library

[20]

K. Otsuka, Y. Takemae, J. Yamato, and H. Murase. A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances. In Proc. ICMI'05, pages 191--198, 2005.

Digital Library

[21]

K. Otsuka, J. Yamato, and H. Murase. Conversation scene analysis with dynamic Bayesian network based on visual head tracking. In Proc. ICMI'06, pages 949--952, 2006.

[22]

K. Otsuka, H. Sawada and J. Yamato. Automatic Inference of Cross-modal Nonverbal interactions in Multiparty Conversations. In Proc. ICMI'07, 2007.

Digital Library

[23]

P. Qvarfordt, S. Zhai. Conversing with the user based on eye-gaze patterns. In Proc. CHI'05, pages 221--230, 2005.

Digital Library

[24]

V. P. Richmon and J. C. McCroskey. Nonverbal Behavior in Interpersonal Relations -- 5th ed. Allyn and Bacon/Pearson, Boston, 2004.

[25]

H. Sawada, S. Araki, K. Otsuka, M. Fujimoto and K. Ishizuka. Voice activity detection for multiple speakers with multiple pin microphones. In 2007 Spring Meeting, Acoustical Society of Japan, 2007.

[26]

R. Stiefelhagen, J. Zhu. Head Orientation and Gaze Direction in meetings. In Proc. ACM CHI '02, 2002.

Digital Library

[27]

W. H. Zangemeister and L. Stark. Types of Gaze Movement: Variable Interactions of Eye and Head Movements. In Experimental Neurology, 77, pages 563--577, 1982.

Cited By

Lee MLi WDeng Z(2024)A Computational Study on Sentence-based Next Speaker Prediction in Multiparty ConversationsProceedings of the 24th ACM International Conference on Intelligent Virtual Agents10.1145/3652988.3673915(1-4)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3652988.3673915
Gupta ATafasca SOdobez J(2022)A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW56347.2022.00552(5037-5046)Online publication date: Jun-2022
https://doi.org/10.1109/CVPRW56347.2022.00552
Siegfried ROdobez J(2021)Visual Focus of Attention Estimation in 3D Scene with an Arbitrary Number of Targets2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW53098.2021.00352(3147-3155)Online publication date: Jun-2021
https://doi.org/10.1109/CVPRW53098.2021.00352
Show More Cited By

Index Terms

Conversation scene analysis based on dynamic Bayesian network and image-based gaze detection
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Quantifying interpersonal influence in face-to-face conversations based on visual attention patterns
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing Systems

A novel measure for automatically quantifying the amount of interpersonal influence present in face-to-face conversations is proposed based on the visual-attention patterns of the participants as inferred from video sequences. First, we focus on the ...
A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances
ICMI '05: Proceedings of the 7th international conference on Multimodal interfaces

A novel probabilistic framework is proposed for inferring the structure of conversation in face-to-face multiparty communication, based on gaze patterns, head directions and the presence/absence of utterances. As the structure of conversation, this ...
Automatic inference of cross-modal nonverbal interactions in multiparty conversations: "who responds to whom, when, and how?" from gaze, head gestures, and utterances
ICMI '07: Proceedings of the 9th international conference on Multimodal interfaces

A novel probabilistic framework is proposed for analyzing cross-modal nonverbal interactions in multiparty face-to-face conversations. The goal is to determine "who responds to whom, when, and how" from multimodal cues including gaze, head gestures, and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

November 2010

311 pages

ISBN:9781450304146

DOI:10.1145/1891903

General Chairs:
Wen Gao
PKU, China
,
Chin-Hui Lee
Georgia Tech
,
Jie Yang
Carnegie Mellon
,
Program Chairs:
Xilin Chen
ICT, CAS, China
,
Maxine Eskenazi
Carnegie Mellon
,
Zhengyou Zhang
MSR

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI-MLMI '10

Sponsor:

SIGCHI

ICMI-MLMI '10: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES/WORKSHOP ON MACHINE LEARNING FOR MULTIMODAL INTERFACES

November 8 - 10, 2010

Beijing, China

Acceptance Rates

ICMI-MLMI '10 Paper Acceptance Rate 41 of 100 submissions, 41%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
399
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lee MLi WDeng Z(2024)A Computational Study on Sentence-based Next Speaker Prediction in Multiparty ConversationsProceedings of the 24th ACM International Conference on Intelligent Virtual Agents10.1145/3652988.3673915(1-4)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3652988.3673915
Gupta ATafasca SOdobez J(2022)A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW56347.2022.00552(5037-5046)Online publication date: Jun-2022
https://doi.org/10.1109/CVPRW56347.2022.00552
Siegfried ROdobez J(2021)Visual Focus of Attention Estimation in 3D Scene with an Arbitrary Number of Targets2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW53098.2021.00352(3147-3155)Online publication date: Jun-2021
https://doi.org/10.1109/CVPRW53098.2021.00352
Otsuka KKasuga KKöhler MD'Mello SGeorgiou PScherer SProvost ESoleymani MWorsley M(2018)Estimating Visual Focus of Attention in Multiparty Meetings using Deep Convolutional Neural NetworksProceedings of the 20th ACM International Conference on Multimodal Interaction10.1145/3242969.3242973(191-199)Online publication date: 2-Oct-2018
https://dl.acm.org/doi/10.1145/3242969.3242973
Otsuka K(2018)Behavioral Analysis of Kinetic Telepresence for Small Symmetric Group-to-Group MeetingsIEEE Transactions on Multimedia10.1109/TMM.2017.277139620:6(1432-1447)Online publication date: Jun-2018
https://doi.org/10.1109/TMM.2017.2771396
Tian LJia QZhu Z(2018)Predicting Turn-Taking by Compact Gazing Transition Patterns in Multiparty ConversationImage and Video Technology10.1007/978-3-319-75786-5_35(437-447)Online publication date: 15-Feb-2018
https://doi.org/10.1007/978-3-319-75786-5_35
Ding YZhang YXiao MDeng ZMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)A Multifaceted Study on Eye Contact based Speaker Identification in Three-party ConversationsProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3025644(3011-3021)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3025644
Kumano SOtsuka KIshii RYamato J(2017)Collective First-Person Vision for Automatic Gaze Analysis in Multiparty ConversationsIEEE Transactions on Multimedia10.1109/TMM.2016.260800219:1(107-122)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1109/TMM.2016.2608002
Gatica-Perez DAran OJayagopi D(2017)Analysis of Small GroupsSocial Signal Processing10.1017/9781316676202.025(349-367)Online publication date: 13-Jul-2017
https://doi.org/10.1017/9781316676202.025
Otsuka K(2016)MMSpace: Kinetically-augmented telepresence for small group-to-group conversations2016 IEEE Virtual Reality (VR)10.1109/VR.2016.7504684(19-28)Online publication date: Mar-2016
https://doi.org/10.1109/VR.2016.7504684
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten