research-article

Facial expression recognition based on improved depthwise separable convolutional network

Authors:

ZhongHua LiuAuthors Info & Claims

Multimedia Tools and Applications, Volume 82, Issue 12

Pages 18635 - 18652

https://doi.org/10.1007/s11042-022-14066-6

Published: 23 November 2022 Publication History

Abstract

A single network model can’t extract more complex and rich effective features. Meanwhile, the network structure is usually huge, and there are many parameters and consume more space resources, etc. Therefore, the combination of multiple network models to extract complementary features has attracted extensive attention. In order to solve the problems existing in the prior art that the network model can’t extract high spatial depth features, redundant network structure parameters, and weak generalization ability, this paper adopts two models of Xception module and inverted residual structure to build the neural network. Based on this, a face expression recognition method based on improved depthwise separable convolutional network is proposed in the paper. Firstly, Gaussian filtering is performed by Canny operator to remove noise, and combined with two original pixel feature maps to form a three-channel image. Secondly, the inverted residual structure of MobileNetV2 model is introduced into the network structure. Finally, the extracted features are classified by Softmax classifier, and the entire network model uses ReLU6 as the nonlinear activation function. The experimental results show that the recognition rate is 70.76% in Fer2013 dataset (facial expression recognition 2013) and 97.92% in CK+ dataset (extended Cohn Kanade). It can be seen that this method not only effectively mines the deeper and more abstract features of the image, but also prevents network over-fitting and improves the generalization ability.

References

[1]

Abasi SA, Tehran M, and Fairchild MD Colour metrics for image edge detection Color Res Appl 2020 45 4 632-643

[2]

Amirkhani D and Bastanfard A An objective method to evaluate exemplar-based inpainted images quality using jaccard index Multimedia Tools Appl 2021 80 17 26199-26212

[3]

Bastanfard A, Bastanfard O, Takahashi H, and Nakajima M Toward anthropometrics simulation of face rejuvenation and skin cosmetic Comput Anim Virtual Worlds 2004 15 3–4 347-352

[4]

Bastanfard A, Takahashi H, Nakajima M (2004) Toward e-appearance of human face and hair by age, expression and rejuvenation. In: 2004 International conference on cyberworlds, pp 306–311. IEEE, DOI, (to appear in print)

[5]

Bastanfard A, Amirkhani D, Mohammadi M (2022) Toward image super-resolution based on local regression and nonlocal means. Multimedia Tools Appl, pp 1–20.

[6]

Chen L, Peng L, Yao G, Liu C, Zhang X (2019) A modified inception-resnet network with discriminant weighting loss for handwritten chinese character recognition. In: 2019 International conference on document analysis and recognition (ICDAR), Sydney, NSW, Australia, pp 1220–1225, DOI, (to appear in print)

[7]

Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1251–1258

[8]

Darwin C (2015) The expression of the emotions in man and animals. In: The expression of the emotions in man and animals. University of Chicago press, Chicago

[9]

Ekman P and Friesen WV Constants across cultures in the face and emotion J personality Social Psychol 1971 17 2 124

[10]

Gao J, Cai Y, and He Z Tp-fer: a three-channel facial expression recognition method based on optimized convolutional neural network Comput Appl Res 2021 38 7 2213-2219

[11]

Ge H, Zhu Z, Dai Y, Wang B, and Wu X Facial expression recognition based on deep learning Comput Methods Programs Biomed 2022 215 106621

[12]

Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H et al (2013) Challenges in representation learning: A report on three machine learning contests. In: International conference on neural information processing, pp 117–124. Springer, DOI, (to appear in print)

[13]

Han S, Hu J, Li W, Zhao S, Chen M, Xu P, and Luo Y From structure to concepts: the two stages of facial expression recognition Neuropsychologia 2021 150 107700

[14]

Hassan AK and Mohammed SN A novel facial emotion recognition scheme based on graph mining Defence Technol 2020 16 5 1062-1072

[15]

He J, He Z, Cai J, et al. A new multi-angle facial expression recognition method Comput Appl Res 2018 35 1 282-286

[16]

Jun B and Kim D Robust face detection using local gradient patterns and evidence accumulation Pattern Recognit 2012 45 9 3304-3316

[17]

Karthick S, Selvakumarasamy S, Arun C, Agrawal P (2021) Automatic attendance monitoring system using facial recognition through feature-based methods (pca, lda). Mater Today: Proc, vol 2.

[18]

Li H and Xu H Deep reinforcement learning for robust emotional classification in facial expression recognition Knowledge-Based Syst 2020 204 106172

[19]

Li Y, Zeng J, Shan S, and Chen X Occlusion aware facial expression recognition using cnn with attention mechanism IEEE Trans Image Process 2018 28 5 2439-2450

[20]

Lin M, Chen Q, Yan S (2013) Network in network. Comput Sci.

[21]

Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 Ieee computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE, DOI, (to appear in print)

[22]

McIlhagga W The canny edge detector revisited Int J Comput Vision 2011 91 3 251-261

[23]

McIlhagga W The canny edge detector revisited Int J Comput Vision 2011 91 3 251-261

[24]

Mehrabian A Communication without words Psychol Today 1968 2 4 53-56

[25]

Meng Q, Hu X, Kang J, and Wu Y On the effectiveness of facial expression recognition for evaluation of urban sound perception Sci Total Environ 2020 710 135484

[26]

Minoofam S. A. H, Bastanfard A, Keyvanpour M. R (2021) Trcla: a transfer learning approach to reduce negative transfer for cellular learning automata. IEEE Trans Neural Networks Learn Syst.

[27]

Navabifar F, Yusof R, and Emadi M Using rotated asymmetric haar-like features for non-frontal face detection Adv Sci Lett 2013 19 12 3520-3524

[28]

Navabifar F, Yusof R, and Emadi M Using rotated asymmetric haar-like features for non-frontal face detection Adv Sci Lett 2013 19 12 3520-3524

[29]

Niu X (2014) Support vector selection algorithm based on knn algorithm and 10 fold cross validation method. In: Journal of central China normal university (natural science edition), pp 335–338

[30]

Ojala T, Pietikäinen M, and Harwood D A comparative study of texture measures with classification based on featured distributions Pattern Recognit 1996 29 1 51-59

[31]

Opschoor JA, Petersen PC, and Schwab C Deep relu networks and high-order finite element methods Anal Appl 2020 18 5 715-770

[32]

Petersen P and Voigtlaender F Optimal approximation of piecewise smooth functions using deep relu neural networks Neural Netw 2018 108 296-330

[33]

Ramis S, Buades J M, Perales F J, Manresa-Yee C (2022) A novel approach to cross dataset studies in facial expression recognition. Multimedia Tools Appl, pp 1–38.

[34]

Rizzo L and Longo L Self-reported data for mental workload modelling in human-computer interaction and third-level education Data Brief 2020 30 105433

[35]

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4510–4520

[36]

Saurav S, Gidde P, Saini R, and Singh S Dual integrated convolutional neural network for real-time facial expression recognition in the wild Visual Comput 2022 38 3 1083-1096

[37]

Sharifara A, Rahim M S M, Anisi Y (2014) A general ref37 of human face detection including a study of neural networks and haar feature-based cascade classifier in face detection. In: 2014 International symposium on biometrics and security technologies (ISBAST), pp 73–78. IEEE, DOI, (to appear in print)

[38]

Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2818–2826

[39]

Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: 31st AAAI conference on artificial intelligence, pp 4278–4284

[40]

Tian Y-I, Kanade T, and Cohn JF Recognizing action units for facial expression analysis IEEE Trans Pattern Anal Mach Intell 2001 23 2 97-115

[41]

Wang Y, Chen R, Lijun YU, et al. (2019) Denoising from remote sensing satellite image based on two-dimensional EMD and adaptive Gauss filtering[J]. Bull Surv Mapp. (02):22–27.

[42]

Wang X, Liu S, Li Q et al (2021) Classification of surrounding rock of svm tunnel based on k fold cross validation. Min Metall Eng (6) 126–128133

[43]

Wei J, Lu G, Yan J, Liu H (2022) Micro-expression recognition using local binary pattern from five intersecting planes. Multimedia Tools Appl, pp 20643–20668.

[44]

Xiao L, Hu X, Chen Y, Xue Y, Chen B, Gu D, Tang B (2022) Multi-head self-attention based gated graph convolutional networks for aspect-based sentiment classification. Multimedia Tools Appl, pp 19051–19070.

[45]

Xu L and Gajic Z Improved network for face recognition based on feature super resolution method Int J Autom Comput 2021 18 6 915-925

[46]

Zhang T, Mao L (2019) The method of cutting image of vehicle face based on haar feature and improved cascade classifier. In: Journal of physics: conference series, vol 1335, p 012018. IOP publishing, DOI, (to appear in print)

Cited By

Kumari VKumar PKumari B(2023)A Novel Optimized Deep Network for Ear Detection and Occlusion AnalysisWireless Personal Communications: An International Journal10.1007/s11277-023-10519-9131:3(1721-1743)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1007/s11277-023-10519-9
Kaya YGürsoy E(2023)RETRACTED ARTICLE: A MobileNet-based CNN model with a novel fine-tuning mechanism for COVID-19 infection detectionSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-022-07798-y27:9(5521-5535)Online publication date: 4-Jan-2023
https://dl.acm.org/doi/10.1007/s00500-022-07798-y

Recommendations

Image ratio features for facial expression recognition application
Special issue on game theory

Video-based facial expression recognition is a challenging problem in computer vision and human-computer interaction. To target this problem, texture features have been extracted and widely used, because they can capture image intensity changes raised ...
Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware
Abstract
Since the distinction between two expressions is fairly vague, usually a subtle change in one part of the human face is enough to change a facial expression. Most of the existing facial expression recognition algorithms are not robust enough ...
FERGCN: facial expression recognition based on graph convolution network
Abstract
Due to the problems of occlusion, pose change, illumination change, and image blur in the wild facial expression dataset, it is a challenging computer vision problem to recognize facial expressions in a complex environment. To solve this problem, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 82, Issue 12

May 2023

1535 pages

ISSN:1380-7501

Issue’s Table of Contents

© The Author(s) 2022.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 23 November 2022

Accepted: 10 October 2022

Revision received: 29 August 2022

Received: 03 June 2022

Author Tags

Qualifiers

Research-article

Funding Sources

the National Natural Science Foundation of China
the National Key Research and Development Program of China
the Research Program of Foundation and Advanced Technology of Henan in China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumari VKumar PKumari B(2023)A Novel Optimized Deep Network for Ear Detection and Occlusion AnalysisWireless Personal Communications: An International Journal10.1007/s11277-023-10519-9131:3(1721-1743)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1007/s11277-023-10519-9
Kaya YGürsoy E(2023)RETRACTED ARTICLE: A MobileNet-based CNN model with a novel fine-tuning mechanism for COVID-19 infection detectionSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-022-07798-y27:9(5521-5535)Online publication date: 4-Jan-2023
https://dl.acm.org/doi/10.1007/s00500-022-07798-y

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents