research-article

Laughter Recognition Using Non-invasive Wearable Devices

Authors:

Elena Di Lascio,

Silvia SantiniAuthors Info & Claims

PervasiveHealth'19: Proceedings of the 13th EAI International Conference on Pervasive Computing Technologies for Healthcare

Pages 262 - 271

https://doi.org/10.1145/3329189.3329216

Published: 20 May 2019 Publication History

Abstract

A growing number of pervasive systems integrate emotion recognition capabilities with the aim of fostering and promoting human health and wellbeing. In this paper, we argue that enhancing these systems with the ability to detect laughter episodes automatically would improve their effectiveness. This is because laughter is one of the most expressive behavioral cues for positive emotions. However, the existing approaches for laughter recognition rely on the use of obtrusive devices or of video and audio cues and have thus limited applicability in real-world settings. To overcome this limitation, we evaluate the feasibility of a novel, multi-modal approach to recognize laughter episodes using physiological and body movement data gathered with unobtrusive, wrist-worn devices. To assess the performance of our method, we collect an extensive data set of laughter episodes, which we also make publicly available. Our results show that laughter episodes can be distinguished from non-laughter episodes with an accuracy of 81%. Further, we demonstrate that the signatures left by laughter episodes on physiological and body-movement data differ significantly from those caused by slightly intense motions or cognitive load tasks.

References

[1]

Jo-Anne Bachorowski, Moria J Smoski, and Michael J Owren. 2001. The Acoustic Features of Human Laughter. The Journal of the Acoustical Society of America 110, 3 (2001).

[2]

Mary Payne Bennett and Cecile Lengacher. 2008. Humor and Laughter May Influence Health: III. Laughter and Health Outcomes. Evidence-Based Complementary and Alternative Medicine 5, 1 (2008).

[3]

Paul N Bennett, Trisha Parsons, Ros Ben-Moshe, Melissa Weinberg, Merv Neal, Karen Gilbert, Helen Rawson, Cherene Ockerby, Paul Finlay, and Alison Hutchinson. 2014. Laughter and Humor Therapy in Dialysis. In Seminars in Dialysis, Vol. 27.

[4]

Christopher M Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Inc. Secaucus, NJ, USA (2006).

Digital Library

[5]

Wolfram Boucsein. 2012. Electrodermal Activity. Springer Science & Business Media.

[6]

Dongrae Cho, Jinsil Ham, Jooyoung Oh, Jeanho Park, Sayup Kim, Nak-Kyu Lee, and Boreom Lee. 2017. Detection of Stress Levels from Biosignals Measured in Virtual Reality Environments Using a Kernel-Based Extreme Learning Machine. Sensors 17, 10 (2017).

[7]

Sarah Cosentino, Tatsuhiro Kishi, Massimiliano Zecca, Salvatore Sessa, Luca Bartolomeo, Kenji Hashimoto, Takashi Nozawa, and Atsuo Takanishi. 2013. Human-humanoid Robot Social Interaction: Laughter. In Proceedings of the International Conference on Robotics and Biomimetics (ROBIO).

[8]

Sarah Cosentino, Salvatore Sessa, and Atsuo Takanishi. 2016. Quantitative Laughter Detection, Measurement, and Classification -- A Critical Survey. IEEE Reviews in Biomedical Engineering 9 (2016).

[9]

Elena Di Lascio, Shkurta Gashi, and Silvia Santini. 2019. USI_Laughs. https://sys.inf.usi.ch/archives/group/mcss/research/datasets/USI_Laughs. (2019).

[10]

Kevin El Haddad, Hüseyin Çakmak, Emer Gilmartin, Stéphane Dupont, and Thierry Dutoit. 2016. Towards a Listening Agent: A System Generating Audiovisual Laughs and Smiles to Show Interest. In Proceedings of the International Conference on Multimodal Interaction (ICMI).

Digital Library

[11]

Simon Flutura, Johannes Wagner, Florian Lingenfelser, Andreas Seiderer, and Elisabeth André. 2016. Laughter Detection in the Wild: Demonstrating a Tool for Mobile Social Signal Processing and Visualization. In Proceedings of the International Conference on Multimodal Interaction (ICMI).

Digital Library

[12]

Barbara L Fredrickson. 2001. The Role of Positive Emotions in Positive Psychology: The Broaden-and-build Theory of Positive Emotions. American Psychologist 56, 3 (2001).

[13]

Maurizio Garbarino, Matteo Lai, Dan Bender, Rosalind W Picard, and Simone Tognetti. 2014. Empatica E3 -- A Wearable Wireless Multi-sensor Device for Real-time Computerized Biofeedback and Data Acquisition. In Proceedings of the International Conference on Wireless Mobile Communication and Healthcare (Mobihealth).

[14]

Martin Gjoreski, Mitja Luštrek, Matjaž Gams, and Hristijan Gjoreski. 2017. Monitoring Stress with a Wrist Device Using Context. Journal of Biomedical Informatics 73 (2017).

Digital Library

[15]

Alberto Greco, Gaetano Valenza, Antonio Lanata, Enzo Pasquale Scilingo, and Luca Citi. 2016. cvxEDA: A Convex Optimization Approach to Electrodermal Activity Processing. IEEE Transactions on Biomedical Engineering 63, 4 (2016).

[16]

Gerhard Hagerer, Nicholas Cummins, Florian Eyben, and Björn Schuller. 2017. 'Did you laugh enough today?' -- Deep Neural Networks for Mobile and Wearable Laughter Trackers. In Proceedings of the International Speech Communication Association Conference (INTERSPEECH).

[17]

Wahida Handouzi, Choubeila Maaoui, Alain Pruski, and Abdelhak Moussaoui. 2014. Anxiety Recognition Using Relevant Features from BVP Signal: Application on Phobic Individuals. Modelling, Measurement, Control Journal, Modelling, Measurement, Control (2014).

[18]

Javier Hernandez, Ivan Riobo, Agata Rozga, Gregory D Abowd, and Rosalind W Picard. 2014. Using Electrodermal Activity to Recognize Ease of Engagement in Children During Social Interactions. In Proceedings of the International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp).

Digital Library

[19]

Hayley Hung, Gwenn Englebienne, and Jeroen Kools. 2013. Classifying Social Actions with a Single Accelerometer. In Proceedings of the 2013 International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp).

Digital Library

[20]

Alexei Ivanov and Giuseppe Riccardi. 2012. Kolmogorov-Smirnov Test for Feature Selection in Emotion Recognition from Speech. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]

Kyriaki Kalimeri and Charalampos Saitis. 2016. Exploring Multimodal Biosignal Features for Stress Detection During Indoor Mobility. In Proceedings of the International Conference on Multimodal Interaction (ICMI).

Digital Library

[22]

Michael Kipp. 2001. Anvil -- A Generic Annotation Tool for Multimodal Dialogue. In Proceeding of the European Conference on Speech Communication and Technology (EUROSPEECH).

[23]

Marco Leo, Marco Del Coco, Pierluigi Carcagni, Cosimo Distante, Massimo Bernava, Giovanni Pioggia, and Giuseppe Palestra. 2015. Automatic Emotion Recognition in Robot-children Interaction for ASD Treatment. In Proceedings of the IEEE International Conference on Computer Vision Workshops.

Digital Library

[24]

Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. 2018. Feature Selection: A Data Perspective. Computing Surveys 50, 6 (2018).

Digital Library

[25]

Daniel McDuff, Sarah Gontarek, and Rosalind Picard. 2014. Remote Measurement of Cognitive Stress via Heart Rate Variability. In Proceedings of the International Conference of Engineering in Medicine and Biology Society (EMBC).

[26]

Andreas C Müller, Sarah Guido, et al. 2016. Introduction to Machine Learning with Python: A Guide for Data Scientists. O'Reilly Media.

[27]

Radoslaw Niewiadomski, Maurizio Mancini, Giovanna Varni, Gualtiero Volpe, and Antonio Camurri. 2016. Automated Laughter Detection from Full-body Movements. IEEE Transactions on Human-Machine Systems 46, 1 (2016).

[28]

Christina Orphanidou, Timothy Bonnici, Peter Charlton, David Clifton, David Vallance, and Lionel Tarassenko. 2015. Signal-quality Indices for the Electrocardiogram and Photoplethysmogram: Derivation and Applications to Wireless Monitoring. IEEE Journal of Biomedical and Health Informatics 19, 3 (2015).

[29]

Erik Peper, Rick Harvey, I-Mei Lin, Hana Tylova, and Donald Moss. 2007. Is There More to Blood Volume Pulse than Heart Rate Variability, Respiratory Sinus Arrhythmia, and Cardiorespiratory Synchrony? Biofeedback 35, 2 (2007).

[30]

Erik Peper, Fred Shaffer, and I-Mei Lin. 2010. Garbage In; Garbage Out -- Identify Blood Volume Pulse (BVP) Artifacts Before Analyzing and Interpreting BVP, Blood Volume Pulse Amplitude, and Heart Rate/Respiratory Sinus Arrhythmia Data. Biofeedback 38, 1 (2010).

[31]

Giulia Perugia, Daniel Rodríguez-Martín, Marta Díaz Boladeras, Andreu Català Mallofré, Emilia Barakova, and Matthias Rauterberg. 2017. Electrodermal Activity: Explorations in the Psychophysiology of Engagement with Social Robots in Dementia. In Proceedings of International Symposium on the Robot and Human Interactive Communication (RO-MAN).

[32]

Monica Perusquia-Hernandez, Mazakasu Hirokawa, and Kenji Suzuki. 2017. A Wearable Device for Fast and Subtle Spontaneous Smile Recognition. IEEE Transactions on Affective Computing 8, 4 (2017).

[33]

Stavros Petridis, Brais Martinez, and Maja Pantic. 2013. The MAHNOB Laughter Database. Image and Vision Computing 31, 2 (2013).

Digital Library

[34]

Stavros Petridis and Maja Pantic. 2011. Audiovisual Discrimination Between Speech and Laughter: Why and When Visual Information Might Help. IEEE Transactions on Multimedia 13, 2 (2011).

Digital Library

[35]

Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017. A review of Affective Computing: From Unimodal Analysis to Multimodal Fusion. Information Fusion 37 (2017).

Digital Library

[36]

Boris Reuderink, Mannes Poel, Khiet Truong, Ronald Poppe, and Maja Pantic. 2008. Decision-level Fusion for Audio-visual Laughter Detection. In Proceedings of the International Workshop on Machine Learning for Multimodal Interaction.

Digital Library

[37]

Willibald Ruch. 1993. Exhilaration and Humor. Handbook of Emotions 1 (1993).

[38]

Willibald Ruch and Paul Ekman. 2001. The Expressive Pattern of Laughter. In Emotions, Qualia, and Consciousness. World Scientific.

[39]

J Ridley Stroop. 1935. Studies of Interference in Serial Verbal Reactions. Journal of Experimental Psychology 18, 6 (1935).

[40]

Shiho Tatsumi, Yasser Mohammad, Yoshimasa Ohmoto, and Toyoaki Nishida. 2014. Detection of Hidden Laughter for Human-agent Interaction. Procedia Computer Science 35 (2014).

[41]

Khiet P Truong and David A Van Leeuwen. 2007. Automatic Discrimination Between Laughter and Speech. Speech Communication 49, 2 (2007).

Digital Library

[42]

Jérôme Urbain, Elisabetta Bevacqua, Thierry Dutoit, Alexis Moinet, Radoslaw Niewiadomski, Catherine Pelachaud, Benjamin Picart, Joëlle Tilmanne, and Johannes Wagner. 2010. The AVLaughterCycle Database. In Proceedings of the International Conference on Language Resources and Evaluation (LREC).

[43]

Jérôme Urbain, Hüseyin Çakmak, Aurélie Charlier, Maxime Denti, Thierry Dutoit, and Stéphane Dupont. 2014. Arousal-Driven Synthesis of Laughter. IEEE Journal of Selected Topics in Signal Processing 8, 2 (2014).

[44]

Jérôme Urbain, Radoslaw Niewiadomski, Maurizio Mancini, Harry Griffin, Hüseyin Cakmak, Laurent Ach, and Gualtiero Volpe. 2013. Multimodal Analysis of Laughter for an Interactive System. In Proceedings of the International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN).

[45]

Melissa Weinberg, Thomas Hammond, and Robert Cummins. 2014. The Impact of Laughter Yoga on Subjective Well-being: A Pilot Study. European journal of humour research 1, 4 (2014).

[46]

Roberto Zangróniz, Arturo Martínez-Rodrigo, José Manuel Pastor, María T López, and Antonio Fernández-Caballero. 2017. Electrodermal Activity Sensor for Classification of Calm/Distress Condition. Sensors 17, 10 (2017).

[47]

Xiao Zhang, Yongqiang Lyu, Xiaomin Luo, Jingyu Zhang, Chun Yu, Hao Yin, and Yuanchun Shi. 2018. Touch Sense: Touch Screen Based Mental Stress Sense. Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) 2, 2 (2018).

Digital Library

[48]

Yuning Zhang, Maysam Haghdan, and Kevin S Xu. 2017. Unsupervised Motion Artifact Detection in Wrist-measured Electrodermal Activity Data. In Proceedings of the International Symposium on Wearable Computers (ISWC).

Digital Library

Cited By

Alchieri LAbdalazim NAlecci LGashi SGjoreski MSantini S(2024)Lateralization Effects in Electrodermal Activity Data Collected Using Wearable DevicesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435418:1(1-30)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643541
Quiros JCabrera-Quiros LOertel CHung H(2024)Impact of Annotation Modality on Label Quality and Model Performance in the Automatic Assessment of Laughter In-the-WildIEEE Transactions on Affective Computing10.1109/TAFFC.2023.326900315:2(519-534)Online publication date: Apr-2024
https://doi.org/10.1109/TAFFC.2023.3269003
Müller EBenke IMaedche A(2024)Heart Rate-Based Emotion Recognition and Adaptive Emotion Regulation Support with Wrist-Worn Wearables: A Systematic Literature ReviewInformation Systems and Neuroscience10.1007/978-3-031-58396-4_31(355-366)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-58396-4_31
Show More Cited By

Index Terms

Laughter Recognition Using Non-invasive Wearable Devices
1. Human-centered computing
  1. Ubiquitous and mobile computing
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Social Intelligence Modeling using Wearable Devices
IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces

Social Signal Processing techniques have given the opportunity to analyze in-depth human behavior in social face-to-face interactions. With recent advancements, it is henceforth possible to use these techniques to augment social interactions, especially ...
Cross-Corpus Experiments on Laughter and Emotion Detection in HRI with Elderly People
Social Robotics
Abstract
Social Signal Processing such as laughter or emotion detection is a very important issue, particularly in the field of human-robot interaction (HRI). At the moment, very few studies exist on elderly-people’s voices and social markers in real-life ...
Designing Speech and Multimodal Interactions for Mobile, Wearable, and Pervasive Applications
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Traditional interfaces are continuously being replaced by mobile, wearable, or pervasive interfaces. Yet when it comes to the input and output modalities enabling our interactions, we have yet to fully embrace some of the most natural forms of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

PervasiveHealth'19: Proceedings of the 13th EAI International Conference on Pervasive Computing Technologies for Healthcare

May 2019

475 pages

ISBN:9781450361262

DOI:10.1145/3329189

General Chairs:
Oscar Mayora
Fondazione Bruno Kessler, Italy
,
Stefano Forti
Fondazione Bruno Kessler, Italy
,
Program Chairs:
Jochen Meyer,
Lena Mamykina

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

EAI: The European Alliance for Innovation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PervasiveHealth'19

PervasiveHealth'19: The 13th International Conference on Pervasive Computing Technologies for Healthcare

May 20 - 23, 2019

Trento, Italy

Acceptance Rates

Overall Acceptance Rate 55 of 116 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
493
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)3

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alchieri LAbdalazim NAlecci LGashi SGjoreski MSantini S(2024)Lateralization Effects in Electrodermal Activity Data Collected Using Wearable DevicesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435418:1(1-30)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643541
Quiros JCabrera-Quiros LOertel CHung H(2024)Impact of Annotation Modality on Label Quality and Model Performance in the Automatic Assessment of Laughter In-the-WildIEEE Transactions on Affective Computing10.1109/TAFFC.2023.326900315:2(519-534)Online publication date: Apr-2024
https://doi.org/10.1109/TAFFC.2023.3269003
Müller EBenke IMaedche A(2024)Heart Rate-Based Emotion Recognition and Adaptive Emotion Regulation Support with Wrist-Worn Wearables: A Systematic Literature ReviewInformation Systems and Neuroscience10.1007/978-3-031-58396-4_31(355-366)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-58396-4_31
Gasparini FGrossi AGiltri MNishinari KBandini S(2023)Behavior and Task Classification Using Wearable Sensor Data: A Study across Different AgesSensors10.3390/s2306322523:6(3225)Online publication date: 17-Mar-2023
https://doi.org/10.3390/s23063225
Laporte MGjoreski MLangheinrich M(2023)LAUREATEProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108927:3(1-41)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610892
Alchieri LAlecci LAbdalazim NSantini S(2023)Recognition of Engagement from Electrodermal Activity Data Across Different ContextsAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610701(108-112)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610701
Yang KTag BWang CGu YSarsenbayeva ZDingler TWadley GGoncalves J(2023)Survey on Emotion Sensing Using Mobile DevicesIEEE Transactions on Affective Computing10.1109/TAFFC.2022.322048414:4(2678-2696)Online publication date: 1-Oct-2023
https://doi.org/10.1109/TAFFC.2022.3220484
Yang KWang CGu YSarsenbayeva ZTag BDingler TWadley GGoncalves J(2023)Behavioral and Physiological Signals-Based Deep Multimodal Approach for Mobile Emotion RecognitionIEEE Transactions on Affective Computing10.1109/TAFFC.2021.310086814:2(1082-1097)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TAFFC.2021.3100868
Gasparini FGrossi AGiltri MBandini S(2022)Personalized PPG Normalization Based on Subject Heartbeat in Resting State ConditionSignals10.3390/signals30200163:2(249-265)Online publication date: 18-Apr-2022
https://doi.org/10.3390/signals3020016
Hossain MKong YPosada-Quintero HChon K(2022)Comparison of Electrodermal Activity from Multiple Body Locations Based on Standard EDA Indices’ Quality and Robustness against Motion ArtifactSensors10.3390/s2209317722:9(3177)Online publication date: 21-Apr-2022
https://doi.org/10.3390/s22093177
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents