research-article

Sequence-based multimodal behavior modeling for social agents

Authors:

Soumia Dermouche,

Catherine PelachaudAuthors Info & Claims

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Pages 29 - 36

https://doi.org/10.1145/2993148.2993180

Published: 31 October 2016 Publication History

Abstract

The goal of this work is to model a virtual character able to converse with different interpersonal attitudes. To build our model, we rely on the analysis of multimodal corpora of non-verbal behaviors. The interpretation of these behaviors depends on how they are sequenced (order) and distributed over time. To encompass the dynamics of non-verbal signals across both modalities and time, we make use of temporal sequence mining. Specifically, we propose a new algorithm for temporal sequence extraction. We apply our algorithm to extract temporal patterns of non-verbal behaviors expressing interpersonal attitudes from a corpus of job interviews. We demonstrate the efficiency of our algorithm in terms of significant accuracy improvement over the state-of-the-art algorithms.

References

[1]

M. Argyle. Bodily Communication. Methuen, 1988.

[2]

M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. LOF: Identifying Density-Based Local Outliers. In 2000 ACM SIGMOD International Conference on Management of Data (SIGMOD ’00), pages 93–104, 2000.

Digital Library

[3]

J. Cassell, T. Bickmore, H. Vilhjálmsson, and H. Yan. More Than Just a Pretty Face: Affordances of Embodiment. In 15th International Conference of Intelligent Virtual Agents (IVA 2015), pages 52–59, 2000.

Digital Library

[4]

M. Chollet, M. Ochs, C. Clavel, and C. Pelachaud. A multimodal corpus approach to the design of virtual recruiters. In Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII 2013), pages 19–24, 2013.

Digital Library

[5]

M. Chollet, M. Ochs, and C. Pelachaud. Mining a multimodal corpus for non-verbal behavior sequences conveying attitudes. In 9th International Conference on Language Resources and Evaluation (LREC 2014), pages 3417–3424, 2014.

[6]

R. Cowie, C. Cox, J.-C. Martin, A. Batliner, D. Heylen, and K. Karpouzis. Issues in Data Labelling. In Emotion-Oriented Systems: The Humaine Handbook, pages 213–241. Springer Berlin Heidelberg, 2011.

[7]

D. Fricker, H. Zhang, and C. Yu. Sequential pattern mining of multimodal data streams in dyadic interactions. In IEEE International Conference on Development and Learning (ICDL 2011), 2011.

[8]

T. Guyet and R. Quiniou. Mining temporal patterns with quantitative intervals. In IEEE International Conference on Data Mining Workshops (ICDM Workshops 2008), pages 218–227, 2008.

Digital Library

[9]

T. Guyet and R. Quiniou. Extracting temporal patterns from interval-based sequences. In International Joint Conference on Artificial Intelligence (IJCAI 2011), pages 1306–1311, 2011.

Digital Library

[10]

K. Kalimeri, B. Lepri, B. K. Foundation, D. Gatica-perez, and F. Pianesi. Modeling Dominance Effects on Nonverbal Behaviors Using Granger Causality. In International Conference on Multimodal Interaction, ICMI 2012, pages 23–26, 2012.

Digital Library

[11]

D. Keltner. Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame. Journal of Personality and Social Psychology, 68(3):441–454, 1995.

[12]

M. S. Magnusson. Discovering hidden time patterns in behavior: T-patterns and their detection. Behavior research methods, instruments, & computers, 32(1):93–110, 2000.

[13]

S. Mariooryad and C. Busso. Analysis and Compensation of the Reaction Lag of Evaluators in Continuous Emotional Annotations. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII 2013), pages 85–90, 2013.

Digital Library

[14]

A. Mignault and A. Chaudhuri. The many faces of a neutral face: Head tilt and perception of dominance and emotion. Journal of Nonverbal Behavior, 27(2):111–132, 2003.

[15]

F. Nakagaito, T. Ozaki, and T. Ohkawa. Discovery of quantitative sequential patterns from event sequences. In IEEE International Conference on Data Mining Workshops, number 1, pages 31–36, Pisa, Italy, 2009.

Digital Library

[16]

L. S. Nguyen and D. Gatica-Perez. I Would Hire You in a Minute: Thin Slices of Nonverbal Behavior in Job Interviews. In International Conference on Multimodal Interaction (ICMI 2015), pages 51–58, 2015.

Digital Library

[17]

R. Niewiadomski, S. J. Hyniewska, and C. Pelachaud. Constraint-based model for synthesis of multimodal sequential expressions of emotions. IEEE Transactions on Affective Computing, 2(3):134–146, 2011.

Digital Library

[18]

J. Pei, J. Han, B. Mortazavi-Asl, H. Pinto, Q. Chen, U. Dayal, and M.-C. Hsu. PrefixSpan,: mining sequential patterns efficiently by prefix-projected pattern growth. In 17th International Conference on Data Engineering (ICDE 2001), pages 215–224, 2001.

Digital Library

[19]

B. Ravenet, A. Cafaro, B. Biancardi, M. Ochs, and C. Pelachaud. Conversational behavior reflecting interpersonal attitudes in small group interactions. In 15th International Conference of Intelligent Virtual Agents (IVA 2015), pages 375–388, 2015.

[20]

G. B. Ray and K. Floyd. Nonverbal Expressions of Liking and Disliking in Initial Interaction: Encoding and Decoding Perspectives. Southern Communication Journal, 71(1):45–65, 2006.

[21]

G. Ruan, H. Zhang, and B. Plale. Parallel and Quantitative Sequential Pattern Mining for Large-scale Interval-based Temporal Data. In 2014 IEEE International Conference on Big Data (BigData 2014), pages 32–39, 2014.

[22]

K. R. Scherer. What are emotion? And how can they be measured? Social Science Information, 44(4):695–729, 2005.

[23]

K. R. Scherer and H. Ellgring. Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion, 7(1):113–130, 2007.

[24]

R. Srikant and R. Agrawal. Mining Sequential Patterns: Generalizations and Performance Improvements. In 5th International Conference on Extending Database Technology: Advances in Database Technology (EDBT ’96), pages 3–17, 1996.

Digital Library

[25]

T. Washio, Y. Mitsunaga, and H. Motoda. Using Adaptive Density-based Subspace Clustering. In Proceedings of the Fifth IEEE International Conference on Data Mining (ICDM 2005), pages 793–796, Houston, TX, USA, 2005.

Digital Library

[26]

S. With and S. Kaiser. Sequential patterning of facial actions in the production and perception of emotional expressions. Swiss Journal of Psychology, 70(4):241–252, 2011.

[27]

C. Yu, M. Scheutz, and P. Schermerhorn. Investigating multimodal real-time patterns of joint attention in an HRI word learning task. In 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2010), pages 309–316, 2010.

Digital Library

[28]

H. Zhang, D. Fricker, T. G. Smith, and C. Yu. Real-time adaptive behaviors in multimodal human-avatar interactions. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI 2010), pages 1–8, 2010.

Digital Library

Cited By

Dermouche SPelachaud C(2022)Leveraging the Dynamics of Non-Verbal Behaviors For Social Attitude ModelingIEEE Transactions on Affective Computing10.1109/TAFFC.2020.298926213:2(1072-1085)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TAFFC.2020.2989262
Pelachaud CBusso CHeylen D(2021)Multimodal Behavior Modeling for Socially Interactive AgentsThe Handbook on Socially Interactive Agents10.1145/3477322.3477331(259-310)Online publication date: 10-Sep-2021
https://dl.acm.org/doi/10.1145/3477322.3477331
Rafael SAlmeida V(2021)Implementation Goals for Multimodal Interfaces in Human-Computer InteractionHuman-Computer Interaction. Theory, Methods and Tools10.1007/978-3-030-78462-1_17(230-239)Online publication date: 3-Jul-2021
https://doi.org/10.1007/978-3-030-78462-1_17
Show More Cited By

Index Terms

Sequence-based multimodal behavior modeling for social agents

Recommendations

Computational model for interpersonal attitude expression
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

This paper presents a plan towards a computational model of interpersonal attitudes and its integration in an embodied conversational agent (ECA). The goal is to endow an ECA with the capacity to express different interpersonal attitudes depending on ...
Attitude Modeling for Virtual Character Based on Temporal Sequence Mining: Extraction and Evaluation
MOCO '18: Proceedings of the 5th International Conference on Movement and Computing

Virtual agents are increasingly being integrated in our everyday life thanks to their communicative skills and abilities to express social affects like emotions and attitudes. The goal of this work is to evaluate the perception of agents expressing ...
Analyses of the effects of agents' performing self-adaptors
ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents

This paper introduces the results of a series of experiments on the impression of agents that perform self-adaptors. Human-human interactions were video-taped and analyzed with respect to usage of different types of self-adaptors (relaxed/stressful), ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

October 2016

605 pages

ISBN:9781450345569

DOI:10.1145/2993148

General Chairs:
Yukiko I. Nakano
Seikei University, Japan
,
Elisabeth André
Augsburg University, Germany
,
Toyoaki Nishida
Kyoto University, Japan
,
Program Chairs:
Louis-Philippe Morency
Carnegie Mellon University, USA
,
Carlos Busso
University of Texas at Dallas, USA
,
Catherine Pelachaud
ISIR, France / University of Paris6, France

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '16

Sponsor:

SIGCHI

ICMI '16: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2016

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
267
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Dermouche SPelachaud C(2022)Leveraging the Dynamics of Non-Verbal Behaviors For Social Attitude ModelingIEEE Transactions on Affective Computing10.1109/TAFFC.2020.298926213:2(1072-1085)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TAFFC.2020.2989262
Pelachaud CBusso CHeylen D(2021)Multimodal Behavior Modeling for Socially Interactive AgentsThe Handbook on Socially Interactive Agents10.1145/3477322.3477331(259-310)Online publication date: 10-Sep-2021
https://dl.acm.org/doi/10.1145/3477322.3477331
Rafael SAlmeida V(2021)Implementation Goals for Multimodal Interfaces in Human-Computer InteractionHuman-Computer Interaction. Theory, Methods and Tools10.1007/978-3-030-78462-1_17(230-239)Online publication date: 3-Jul-2021
https://doi.org/10.1007/978-3-030-78462-1_17
Yunus FClavel CPelachaud C(2021)Sequence-to-Sequence Predictive Model: From Prosody to Communicative GesturesDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior10.1007/978-3-030-77817-0_25(355-374)Online publication date: 3-Jul-2021
https://doi.org/10.1007/978-3-030-77817-0_25
Chollet MOchs MPelachaud C(2019)A Methodology for the Automatic Extraction and Generation of Non-Verbal Signals Sequences Conveying Interpersonal AttitudesIEEE Transactions on Affective Computing10.1109/TAFFC.2017.275377710:4(585-598)Online publication date: 1-Oct-2019
https://doi.org/10.1109/TAFFC.2017.2753777
Kantharaju RDe Franco DPease APelachaud CBogdanovych ARichards DSimoff SPelachaud CHeylen DTrescak T(2018)Is Two Better than One?Proceedings of the 18th International Conference on Intelligent Virtual Agents10.1145/3267851.3267890(255-262)Online publication date: 5-Nov-2018
https://dl.acm.org/doi/10.1145/3267851.3267890
Dermouche SPelachaud C(2018)Attitude Modeling for Virtual Character Based on Temporal Sequence MiningProceedings of the 5th International Conference on Movement and Computing10.1145/3212721.3212806(1-8)Online publication date: 28-Jun-2018
https://dl.acm.org/doi/10.1145/3212721.3212806
Pelachaud CChaminade TNgyuen NOchs MLefèvre F(2017)Greta: a conversing socio-emotional agentProceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents10.1145/3139491.3139902(9-10)Online publication date: 13-Nov-2017
https://dl.acm.org/doi/10.1145/3139491.3139902
Porhet COchs MSaubesty Jde Montcheuil GBertrand RLank EVinciarelli AHoggan ESubramanian SBrewster S(2017)Mining a multimodal corpus of doctor’s training for virtual patient’s feedbacksProceedings of the 19th ACM International Conference on Multimodal Interaction10.1145/3136755.3136816(473-478)Online publication date: 3-Nov-2017
https://dl.acm.org/doi/10.1145/3136755.3136816

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents