research-article

Human emotion recognition based on facial expressions via deep learning on high-resolution images

Authors:

Mohammad BarrAuthors Info & Claims

Multimedia Tools and Applications, Volume 80, Issue 16

Pages 25241 - 25253

https://doi.org/10.1007/s11042-021-10918-9

Published: 01 July 2021 Publication History

Abstract

Detecting human emotion based on facial expression is considered a hard task for the computer vision community because of many challenges such as the difference of face shape from a person to another, difficulty of recognition of dynamic facial features, low quality of digital images, etc. In this paper, we propose a face-sensitive convolutional neural network (FS-CNN) for human emotion recognition. The proposed FS-CNN is used to detect faces on large scale images then analyzing face landmarks to predict expressions for emotion recognition. The FS-CNN is composed form two stages, patch cropping, and convolutional neural networks. The first stage is used to detect faces in high-resolution images and crop the face for further processing. The second stage is a convolutional neural network used to predict facial expression based on landmarks analytics, it was applied on pyramid images to process scale invariance. The proposed FS-CNN was trained and evaluated on the UMD Faces dataset. High performance was achieved with a mean average precision of about 95%.

References

[1]

Afif M, Ayachi R, Said Y, Pissaloux E, Atri M (2018) Indoor image recognition and classification via deep convolutional neural network. In: International conference on the Sciences of Electronics, Technologies of Information and Telecommunications, pp. 364–371. Cham: Springer

[2]

Afif M, Ayachi R, Said Y, Pissaloux E, Atri M (2020) An evaluation of RetinaNet on indoor object detection for blind and visually impaired persons assistance navigation. Neural Process Lett:1–15

[3]

Arshad H, Khan MA, Sharif MI, Yasmin M, Tavares JMRS, Zhang Y-D, Satapathy SC (2020) A multilevel paradigm for deep convolutional neural network features selection with an application to human gait recognition. Exp Syst e12541

[4]

Ayachi R, Afif M, Said Y, Atri M (2018) Strided convolution instead of max pooling for memory efficiency of convolutional neural networks. In: International conference on the Sciences of Electronics, Technologies of Information and Telecommunications (pp. 234–243). Cham: Springer

[5]

Ayachi R, Afif M, Said Y, Atri M (2019) Traffic signs detection for real-world application of an advanced driving assisting system using deep learning. Neural Process Lett:1–15

[6]

Ayachi R, Said v, Atri M (n.d.) To perform road signs recognition for autonomous vehicles using cascaded deep learning pipeline. Artif Intell Adv

[7]

Baber J, Bakhtyar M, Ahmed KU, Noor W, Devi V, Sammad A (2019) Facial expression recognition and analysis of interclass false positives using CNN. In: Future of Information and Communication Conference (pp. 46–54). Cham: Springer

[8]

Bansal A, Nanduri A, Castillo CD, Ranjan R, Chellappa R (2017) Umdfaces: An annotated face dataset for training deep networks. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), pp. 464–473. IEEE

[9]

Bargal SA, Barsoum E, Ferrer CC, Zhang C (2016) Emotion recognition in the wild from videos using images. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction (pp. 433–436)

[10]

Bhowmik MK, Saha K, Majumder S, Majumder G, Saha A, Sarma AN, Bhattacharjee D, Basu DK, Nasipuri M (2011) Thermal infrared face recognition—a biometric identification technique for robust security system. Reviews refinements and new ideas in face recognition 7

[11]

Bodla N, Singh B, Chellappa R, Davis LS (2017) Soft-NMS--improving object detection with one line of code. In: Proceedings of the IEEE international conference on computer vision (pp. 55615569)

[12]

Dandıl E and Özdemir R Real-time facial emotion classification using deep learning Data Sci Appl 2019 2 1 13-17

[13]

Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial-expression databases from movies

[14]

Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision (pp. 1440-1448)

[15]

Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W et al. (2013) Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing (pp. 117–124). Berlin: Springer

[16]

He K, Zhang X, Ren S, and Sun J Spatial pyramid pooling in deep convolutional networks for visual recognition IEEE Trans Pattern Anal Mach Intell 2015 37 9 1904-1916

[17]

Hsu R-L, Abdel-Mottaleb M, and Jain AK Face detection in color images IEEE Trans Pattern Anal Mach Intell 2002 24 5 696-706

[18]

Jaiswal S, Nandi GC (2019) Robust real-time emotion detection system using CNN architecture. Neural Comput Appl 1–10

[19]

Jiao L, Zhang F, Liu F, Yang S, Li L, Feng Z, and Qu R A survey of deep learning-based object detection IEEE Access 2019 7 128837-128868

[20]

Jose E, Greeshma M, Mithun Haridas TP, Supriya MH (2019) Face recognition based surveillance system using facenet and mtcnn on jetson tx2. In: 2019 5th International Conference on Advanced Computing & Communication Systems (ICACCS), pp. 608–613. IEEE

[21]

Jumani SZ, Ali F, Guriro S, Kandhro IA, Khan A, and Zaidi A Facial expression recognition with histogram of oriented gradients using CNN Indian J Sci Technol 2019 12 24

[22]

Kavitha SN, Shahila K, Kumar SCP (2018) Biometrics Secured Voting System with Finger Print, Face and Iris Verification. In: 2018 Second International Conference on Computing Methodologies and Communication (ICCMC), pp. 743–746. IEEE

[23]

Khan MA, Javed K, Khan SA, Saba T, Habib U, Khan JA, Abbasi AA (2020) Human action recognition using fusion of multiview and deep features: an application to video surveillance. Multimedia Tools Appl 1–27

[24]

Khan MA, Zhang Y-D, Khan SA, Attique M, Rehman A, Seo S (2020) A resource conscious human action recognition framework using 26-layered deep convolutional neural network. Multimedia Tools Appl 1–23

[25]

Kollias D, Zafeiriou S (2019) Exploiting multi-cnn features in cnn-rnn based dimensional emotion recognition on the omg in-the-wild dataset. arXiv preprint arXiv:1910.01417

[26]

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems (pp. 1097–1105)

[27]

Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC 2016 Ssd: Single shot multibox detector. In: European conference on computer vision (pp. 21–37). Cham: Springer

[28]

Liu Z, Luo P, Wang X, Tang X (2018) Large-scale celebfaces attributes (celeba) dataset. Retrieved August 15: 2018

[29]

Mehendale N Facial emotion recognition using convolutional neural networks (FERC) SN Appl Sci 2020 2 3 1-8

[30]

Mehmood A, Khan MA, Sharif M, Khan SA, Shaheen M, Saba T, Riaz N, Ashraf I (2020) Prosperous human gait recognition: an end-to-end system based on pre-trained CNN features selection. Multimedia Tools Appl

[31]

Ranjan R, Patel VM, and Chellappa R Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition IEEE Trans Pattern Anal Mach Intell 2017 41 1 121-135

[32]

Rashid M, Khan MA, Alhaisoni M, Wang S-H, Naqvi SR, Rehman A, and Saba T A sustainable deep learning framework for object recognition using multi-layers deep features fusion and selection Sustainability 2020 12 12 5037

[33]

Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems (pp. 91-99)

[34]

UMD Faces Dataset (n.d.) available at : http://umdfaces.io/

[35]

Viola P and Jones MJ Robust real-time face detection Int J Comput Vis 2004 57 2 137-154

[36]

Wu W, Qian C, Yang S, Wang Q, Cai Y, Zhou Q (2018) Look at boundary: A boundary-aware face alignment algorithm. In: 2018 Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2129–2138. IEEE

[37]

Yitzhak N, Gurevich T, Inbar N, Lecker M, Atias D, Avramovich H, and Aviezer H Recognition of emotion from subtle and non-stereotypical dynamic facial expressions in Huntington's disease Cortex 2020 126 343-354

Cited By

Younis EMohsen SHoussein EIbrahim O(2024)Machine learning for human emotion recognition: a comprehensive reviewNeural Computing and Applications10.1007/s00521-024-09426-236:16(8901-8947)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s00521-024-09426-2
Qian ZGuo PWang YXiao F(2023)Ethical and moral decision-making for self-driving cars based on deep reinforcement learningJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-22455345:4(5523-5540)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/JIFS-224553
Maithri MRaghavendra UGudigar ASamanth JPrabal Datta Barua Murugappan MChakole YAcharya U(2022)Automated emotion recognitionComputer Methods and Programs in Biomedicine10.1016/j.cmpb.2022.106646215:COnline publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.cmpb.2022.106646
Show More Cited By

Recommendations

Human-Computer Interaction Using Emotion Recognition from Facial Expression
EMS '11: Proceedings of the 2011 UKSim 5th European Symposium on Computer Modeling and Simulation

This paper describes emotion recognition system based on facial expression. A fully automatic facial expression recognition system is based on three steps: face detection, facial characteristic extraction and facial expression classification. We have ...
Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method
Abstract
Development of automated systems to recognize human emotions can enhance the quality of delivery of public health service to a great extent. Due to this reason, extensive researches have been started in the recent years to recognize human ...
Human centric facial expression recognition
HCI '18: Proceedings of the 32nd International BCS Human Computer Interaction Conference

Facial expression recognition (FER) is an area of active research, both in computer science and in behavioural science. Across these domains there is evidence to suggest that humans and machines find it easier to recognise certain emotions, for example ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 80, Issue 16

Jul 2021

1585 pages

ISSN:1380-7501

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 July 2021

Accepted: 01 April 2021

Revision received: 28 December 2020

Received: 26 July 2020

Author Tags

Qualifiers

Research-article

Funding Sources

Northern Borders University

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Younis EMohsen SHoussein EIbrahim O(2024)Machine learning for human emotion recognition: a comprehensive reviewNeural Computing and Applications10.1007/s00521-024-09426-236:16(8901-8947)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s00521-024-09426-2
Qian ZGuo PWang YXiao F(2023)Ethical and moral decision-making for self-driving cars based on deep reinforcement learningJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-22455345:4(5523-5540)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/JIFS-224553
Maithri MRaghavendra UGudigar ASamanth JPrabal Datta Barua Murugappan MChakole YAcharya U(2022)Automated emotion recognitionComputer Methods and Programs in Biomedicine10.1016/j.cmpb.2022.106646215:COnline publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.cmpb.2022.106646
Muthamilselvan TBrindha KSenthilkumar SSaransh Chatterjee JHu Y(2022)Optimized face-emotion learning using convolutional neural network and binary whale optimizationMultimedia Tools and Applications10.1007/s11042-022-14124-z82:13(19945-19968)Online publication date: 24-Nov-2022
https://dl.acm.org/doi/10.1007/s11042-022-14124-z

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents