research-article

Emotion Recognition System via Facial Expressions and Speech Using Machine Learning and Deep Learning Techniques

Authors:

Aayushi Chaudhari,

Thanh Thi Nguyen,

Kalind SardaAuthors Info & Claims

SN Computer Science, Volume 4, Issue 4

https://doi.org/10.1007/s42979-022-01633-9

Published: 28 April 2023 Publication History

Abstract

Patients in hospitals frequently exhibit psychological issues such as sadness, pessimism, eccentricity, and anxiety. However, hospitals normally lack tools and facilities to continuously monitor the psychological health of patients. It is desirable to identify depression in patients so that it can be managed by instantly providing better therapy. This can be possible by advances in machine learning for image processing with notable applications in the domain of emotion recognition using facial expressions. In this paper, we have proposed two different methods, i.e. facial expression detection and voice analysis, to predict emotions. For facial expression recognition, we have used two approaches, one is the use of Gabor filters for feature extraction with support vector machine for classification and another is using convolutional neural network (CNN). For voice analysis, we extracted mel-frequency cepstral coefficients from speech data and, based on those features, predicted the emotions of the speech using a CNN model. Experimental results show that our proposed emotion recognition methods obtained high accuracy and thus could be potentially deployed to real-world applications.

References

[1]

Sonawane B and Sharma P Deep learning based approach of emotion detection and grading system Pattern Recognit Image Anal 2020 30 4 726-740

[2]

Kim DJ Facial expression recognition using ASM-based post-processing technique Pattern Recognit Image Anal 2016 26 3 576-581

[3]

Muhammad K, Khan S, Kumar N, Del Ser J, and Mirjalili S Vision-based personalized wireless capsule endoscopy for smart healthcare: taxonomy, literature review, opportunities and challenges Futur Gener Comput Syst 2020 113 266-280

[4]

Pisor AC, Gervais MM, Purzycki BG, and Ross CT Preferences and constraints: the value of economic games for studying human behaviour R Soc Open Sci 2020 7 6

[5]

Le DN, Nguyen GN, Van Chung L, and Dey N MMAS algorithm for features selection using 1D-DWT for video-based face recognition in the online video contextual advertisement user-oriented system J Glob Inf Manag (JGIM) 2017 25 4 103-124

[6]

Panning A, Al-Hamadi AK, Niese R, and Michaelis B Facial expression recognition based on Haar-like feature detection Pattern Recognit Image Anal 2008 18 3 447-452

[7]

Tarnowski P, Kołodziej M, Majkowski A, and Rak RJ Emotion recognition using facial expressions Proc Comput Sci 2017 108 1175-1184

[8]

Le DN, Nguyen GN, Bhateja V, and Satapathy SC Optimizing feature selection in video-based recognition using Max-Min Ant System for the online video contextual advertisement user-oriented system J Comput Sci 2017 21 361-370

[9]

Rozaliev VL and Orlova YA Motion and posture recognition for identifying human emotional reactions Pattern Recognit Image Anal 2015 25 4 710-721

[10]

Basu S, Chakraborty J, Bag A, Aftabuddin M. A review on emotion recognition using speech. In: 2017 international conference on inventive communication and computational technologies (ICICCT). IEEE; 2017. p. 109–114.

[11]

Liu M, Shan S, Wang R, Chen X. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2014. p. 1749–1756.

[12]

Jie S and Yongsheng Q Multi-view facial expression recognition with multi-view facial expression light weight network Pattern Recognit Image Anal 2020 30 4 805-814

[13]

Chen Y, Wang J, Chen S, Shi Z, Cai J. Facial motion prior networks for facial expression recognition. In: 2019 IEEE visual communications and image processing (VCIP). IEEE; 2019. p. 1–4.

[14]

Hibare R, Vibhute A. Feature extraction techniques in speech processing: a survey. Int J Comput Appl. 2014;107(5).

[15]

Meng Z, Liu P, Cai J, Han S, Tong Y. Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE; 2017. p. 558–565.

[16]

Meng D, Peng X, Wang K, Qiao Y. Frame attention networks for facial expression recognition in videos. In: 2019 IEEE international conference on image processing (ICIP). IEEE; 2019. p. 3866–3870.

[17]

Verma A, Dogra A, Malik K, Talwar M. Emotion recognition system for patients with behavioral disorders. In: Intelligent communication, control and devices. Singapore: Springer; 2018. p. 139–145.

[18]

Alugupally N, Samal A, Marx D, and Bhatia S Analysis of landmarks in recognition of face expressions Pattern Recognit Image Anal 2011 21 4 681-693

[19]

Wang X, Huang J, Zhu J, Yang M, Yang F. Facial expression recognition with deep learning. In: Proceedings of the 10th international conference on internet multimedia computing and service. 2018. p. 1–4.

[20]

Yang H, Ciftci U, Yin L. Facial expression recognition by deexpression residue learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. p. 2168–2177.

[21]

Kamachi M, Lyons M, Gyoba J. The Japanese Female Facial Expression (JAFFE) database. 1997. http://www.kasrl.org/jaffe.html.

[22]

Kosti R, Alvarez JM, Recasens A, and Lapedriza A Context based emotion recognition using emotic dataset IEEE Trans Pattern Anal Mach Intell 2019 42 11 2755-2766

[23]

Livingstone SR, Russo FA. The Ryerson audio-visual database of emotional speech and song (RAVDESS) [Data set]. In: PLoS ONE 2018;(1.0.0, Vol. 13, Number 5, p. e0196391). Zenodo.

[24]

Cao H, Cooper D, Keutmann M, Gur R, Nenkova A, and Verma R CREMA-D: crowd-sourced emotional multimodal actors dataset IEEE Trans Affect Comput 2014 5 377-390

[25]

Haq S, Jackson PJB. Multimodal emotion recognition. In: Wang W, editor. Machine audition: principles, algorithms and systems. Hershey: IGI Global Press; 2010. p. 398–423.

[26]

El Ayadi M, Kamel MS, and Karray F Survey on speech emotion recognition: features, classification schemes, and databases Pattern Recognit 2011 44 3 572-587

[27]

Koolagudi SG and Rao KS Emotion recognition from speech: a review Int J Speech Technol 2012 15 2 99-117

[28]

Palo HK, Chandra M, and Mohanty MN Emotion recognition using MLP and GMM for Oriya language Int J Comput Vis Robot 2017 7 4 426-442

[29]

Murthy HA and Yegnanarayana B Formant extraction from group delay function Speech Commun 1991 10 3 209-221

[30]

Choudhary A, Govil MC, Singh G, Awasthi LK. Workflow scheduling algorithms in cloud environment: a review, taxonomy, and challenges. In: 2016 4th international conference on parallel, distributed and grid computing (PDGC). IEEE; 2016. p. 617–624.

[31]

Albu F, Hagiescu D, Vladutu L, Puica MA. Neural network approaches for children’s emotion recognition in intelligent learning applications. In: Proceedings of the 7th international conference on education and new learning technologies (EDULEARN15). 2015. p. 3229–3239.

Cited By

Deshpande KSodhi MRaniyer NRao M(2024)A Time-Distributed CNN-LSTM with Attention Model for Speech Based Emotion RecognitionProceedings of the 2024 7th International Conference on Digital Medicine and Image Processing10.1145/3705927.3705939(67-71)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3705927.3705939

Recommendations

An efficient deep learning technique for facial emotion recognition
Abstract
Emotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion ...
Facial Expression Recognition Using Machine Learning and Deep Learning Techniques: A Systematic Review
Abstract
In the contemporary era, Facial Expression Recognition (FER) plays a pivotal role in numerous fields due to its vast application areas, such as e-learning, healthcare, marketing, and psychology, to name a few examples. Several research studies ...
Analysis of emotion recognition using facial expressions, speech and multimodal information
ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image SN Computer Science

SN Computer Science Volume 4, Issue 4

Apr 2023

1389 pages

EISSN:2661-8907

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 28 April 2023

Accepted: 22 December 2022

Received: 19 August 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Deshpande KSodhi MRaniyer NRao M(2024)A Time-Distributed CNN-LSTM with Attention Model for Speech Based Emotion RecognitionProceedings of the 2024 7th International Conference on Digital Medicine and Image Processing10.1145/3705927.3705939(67-71)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3705927.3705939

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents