short-paper

Smile and Laugh Dynamics in Naturalistic Dyadic Interactions: Intensity Levels, Sequences and Roles

Authors:

Kevin El Haddad,

Sandeep Nallan Chakravarthula,

James KennedyAuthors Info & Claims

ICMI '19: 2019 International Conference on Multimodal Interaction

Pages 259 - 263

https://doi.org/10.1145/3340555.3353764

Published: 14 October 2019 Publication History

Abstract

Smiles and laughs have been the subject of many studies over the past decades, due to their frequent occurrence in interactions, as well as their social and emotional functions in dyadic conversations. In this paper we push forward previous work by providing a first study on the influence one interacting partner’s smiles and laughs have on their interlocutor’s, taking into account these expressions’ intensities. Our second contribution is a study on the patterns of laugh and smile sequences during the dialogs, again taking the intensity into account. Finally, we discuss the effect of the interlocutor’s role on smiling and laughing. In order to achieve this, we use a database of naturalistic dyadic conversations which was collected and annotated for the purpose of this study. The details of the collection and annotation are also reported here to enable reproduction.

References

[1]

Véronique Aubergé and Marie Cathiard. 2003. Can we hear the prosody of smile?Speech Communication 40, 1-2 (2003), 87–97.

[2]

Sanjay Bilakhia, Stavros Petridis, Anton Nijholt, and Maja Pantic. 2015. The MAHNOB Mimicry Database: A database of naturalistic human interactions. Pattern Recognition Letters 66 (2015), 52–61.

Digital Library

[3]

Paul Ekman. 1977. Facial Action Coding System. (1977).

[4]

Kevin El Haddad, Hüseyin Çakmak, Emer Gilmartin, Stéphane Dupont, and Thierry Dutoit. 2016. Towards a Listening Agent: A System Generating Audiovisual Laughs and Smiles to Show Interest. In Proceedings of the 18th ACM International Conference on Multimodal Interaction(ICMI 2016). ACM, New York, NY, USA, 248–255. https://doi.org/10.1145/2993148.2993182

Digital Library

[5]

Sebastian Feese, Bert Arnrich, Gerhard Tröster, Bertolt Meyer, and Klaus Jonas. 2012. Quantifying behavioral mimicry by automatic detection of nonverbal cues from body motion. In Proceedings of the 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Conference on Social Computing. IEEE, 520–525.

Digital Library

[6]

Elisa Gironzetti, Lucy Pickering, Meichan Huang, Ying Zhang, Shigehito Menjo, and Salvatore Attardo. 2016. Smiling synchronicity and gaze patterns in dyadic humorous conversations. Humor 29, 2 (2016), 301–324.

[7]

Markku Haakana. 2010. Laughter and smiling: Notes on co-occurrences. Journal of Pragmatics 42, 6 (2010), 1499–1512.

[8]

Louise Heron, Jaebok Kim, Minha Lee, Kevin El Haddad, Stéphane Dupont, Thierry Dutoit, and Khiet Truong. 2018. A Dyadic Conversation Dataset on Moral Emotions. In Proceedings of the 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018). 687–691. https://doi.org/10.1109/FG.2018.00108

Digital Library

[9]

Ursula Hess and Patrick Bourgeois. 2010. You smile–I smile: Emotion expression in social interaction. Biological psychology 84, 3 (2010), 514–520.

[10]

Nadine Lavan, Sophie K Scott, and Carolyn McGettigan. 2016. Laugh like you mean it: Authenticity modulates acoustic, physiological and perceptual properties of laughter. Journal of Nonverbal Behavior 40, 2 (2016), 133–149.

[11]

JS Lockard, CE Fahrenbruch, JL Smith, and CJ Morgan. 1977. Smiling and laughter: Different phyletic origins?Bulletin of the Psychonomic Society 10, 3 (1977), 183–186.

[12]

Gary McKeown and Will Curran. 2015. The relationship between laughter intensity and perceived humour. In The 4th Interdisciplinary Workshop on Laughter and other Non-Verbal Vocalisations in Speech. 27–29.

[13]

Costanza Navarretta. 2016. Mirroring Facial Expressions and Emotions in Dyadic Conversations. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). 469–474.

[14]

Robert R Provine. 1992. Contagious laughter: Laughter is a sufficient stimulus for laughs and smiles. Bulletin of the Psychonomic Society 30, 1 (1992), 1–4.

[15]

Willibald Ruch and Paul Ekman. 2001. The expressive pattern of laughter. In Emotion, qualia and consciousness, A. Kaszniak (Ed.). World Scientific Publishers, Tokyo, 426–443.

[16]

Marc Schröder, Véronique Aubergé, and Marie-Agnes Cathiard. 1998. Can we hear smile?. In Proceedings of the Fifth International Conference on Spoken Language Processing.

[17]

Juan R Terven, Bogdan Raducanu, María Elena Meza-de Luna, and Joaquín Salas. 2016. Head-gestures mirroring detection in dyadic social interactions with computer vision-based wearable devices. Neurocomputing 175(2016), 866–876.

Digital Library

[18]

Jürgen Trouvain. 2001. Phonetic aspects of ”speech laughs”. In Oralité et Gestualité: Actes du colloque ORAGE, Aix-en-Provence. Paris: L’Harmattan. 634–639.

[19]

Peter Wittenburg, Hennie Brugman, Albert Russel, Alex Klassmann, and Han Sloetjes. 2006. ELAN: a professional framework for multimodality research. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006). 1556–1559.

Cited By

Maraev VMazzocconi CHowes CPelachaud C(2023)Towards investigating gaze and laughter coordination in socially interactive agentsProceedings of the 11th International Conference on Human-Agent Interaction10.1145/3623809.3623968(473-475)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3623809.3623968
Quiros JCabrera-Quiros LOertel CHung H(2023)Impact of Annotation Modality on Label Quality and Model Performance in the Automatic Assessment of Laughter In-the-WildIEEE Transactions on Affective Computing10.1109/TAFFC.2023.326900315:2(519-534)Online publication date: 25-May-2023
https://dl.acm.org/doi/10.1109/TAFFC.2023.3269003
Rauzy SAmoyal M(2023)Automatic tool to annotate smile intensities in conversational face-to-face interactionsGesture10.1075/gest.22012.rau21:2-3(320-364)Online publication date: 1-Sep-2023
https://doi.org/10.1075/gest.22012.rau
Show More Cited By

Recommendations

Multi-modal Multi-cultural Dimensional Continues Emotion Recognition in Dyadic Interactions
AVEC'18: Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop

Automatic emotion recognition is a challenging task which can make great impact on improving natural human computer interactions. In this paper, we present our solutions for the Cross-cultural Emotion Sub-challenge (CES) of Audio/Visual Emotion ...
Laughter entrainment in dyadic interactions: Temporal distribution and form
Abstract
It has been established across a wide range of communicative behaviours that conversational partners tend to become more similar during their interaction. This phenomenon, often called entrainment, has been shown to take place not only ...
Highlights
- Laughter entrainment examined across languages: French, German and Mandarin Chinese.
Multimodal Analysis of Interruptions
Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Anthropometry, Human Behavior, and Communication
Abstract
During an interaction, interactants exchange speaking turns. Exchanges can be done smoothly or through interruptions. Listeners can display backchannels, send signals to grab the speaking turn, wait for the speaker to yield the turn, or even ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMI '19: 2019 International Conference on Multimodal Interaction

October 2019

601 pages

ISBN:9781450368605

DOI:10.1145/3340555

Editors:
Wen Gao
Peking University, China
,
Helen Mei Ling Meng
Chinese University of Hong Kong, China
,
Matthew Turk
Toyota Technological Institute at Chicago, USA
,
Susan R. Fussell
Cornell University, USA
,
Björn Schuller
Imperial College London / University of Augsburg, UK
,
Yale Song
Microsoft Research, USA
,
Kai Yu
Shanghai Jiao Tong University, China

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

ICMI '19

ICMI '19: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 14 - 18, 2019

Suzhou, China

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
287
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)5

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Maraev VMazzocconi CHowes CPelachaud C(2023)Towards investigating gaze and laughter coordination in socially interactive agentsProceedings of the 11th International Conference on Human-Agent Interaction10.1145/3623809.3623968(473-475)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3623809.3623968
Quiros JCabrera-Quiros LOertel CHung H(2023)Impact of Annotation Modality on Label Quality and Model Performance in the Automatic Assessment of Laughter In-the-WildIEEE Transactions on Affective Computing10.1109/TAFFC.2023.326900315:2(519-534)Online publication date: 25-May-2023
https://dl.acm.org/doi/10.1109/TAFFC.2023.3269003
Rauzy SAmoyal M(2023)Automatic tool to annotate smile intensities in conversational face-to-face interactionsGesture10.1075/gest.22012.rau21:2-3(320-364)Online publication date: 1-Sep-2023
https://doi.org/10.1075/gest.22012.rau
Amoyal MCappellini MCombe CPallaud BPriego-Valverde BTellier M(2022)Multimodalité(s) dans les interactions humainesMultimodality in human interactionsTIPA. Travaux interdisciplinaires sur la parole et le langage10.4000/tipa.5167Online publication date: 31-Dec-2022
https://doi.org/10.4000/tipa.5167
Bohy HEl Haddad KDutoit T(2022)A New Perspective on Smiling and Laughter Detection: Intensity Levels Matter2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII)10.1109/ACII55700.2022.9953896(1-8)Online publication date: 18-Oct-2022
https://doi.org/10.1109/ACII55700.2022.9953896
Wood ASievert SMartin J(2022)Semantic Similarity of Social Functional Smiles and LaughterJournal of Nonverbal Behavior10.1007/s10919-022-00405-646:4(399-420)Online publication date: 25-Jun-2022
https://doi.org/10.1007/s10919-022-00405-6
Mazzocconi CGinzburg J(2022)A Longitudinal Characterization of Typical Laughter Development in Mother–Child Interaction from 12 to 36 Months: Formal Features and Reciprocal ResponsivenessJournal of Nonverbal Behavior10.1007/s10919-022-00403-846:4(327-362)Online publication date: 7-Aug-2022
https://doi.org/10.1007/s10919-022-00403-8
Mazzocconi C(2022)Laughter Meaning Construction and Use in Development: Children and Spoken Dialogue SystemsHCI International 2022 - Late Breaking Papers. Multimodality in Advanced Interaction Environments10.1007/978-3-031-17618-0_10(113-133)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-17618-0_10
Mazzocconi CMaraev VSomashekarappa VHowes C(2021)Looking for Laughs: Gaze Interaction with Laughter Pragmatics and CoordinationProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479947(636-644)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3479947
Jansen MTruong KHeylen D(2021)How Familiarity Influences the Frequency, Temporal Dynamics and Acoustics of Laughter2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII)10.1109/ACII52823.2021.9597414(1-8)Online publication date: 28-Sep-2021
https://doi.org/10.1109/ACII52823.2021.9597414
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents