Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Virtual facial expression recognition using deep CNN with ensemble learning

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

In the current era, virtual environments and virtual characters have become popular. In the near future, recognition of virtual facial expressions plays an important role in virtual assistants, online video games, security systems, entertainment, psychological study, video conferencing, virtual reality, and online classes. The objective of this work is to recognize the facial emotions of virtual characters. Facial expression recognition (FER) from virtual characters is a difficult task due to its intra-class variation and inter-class similarity. The performances of existing FER systems are limited in this aspect. To address these challenges, we designed and developed a multi-block deep convolutional neural networks (DCNN) model to recognize the facial emotions from virtual, stylized and human characters. In multi-block DCNN, we defined four blocks with various computational elements to extract the discriminative features from facial images. To increase stability and to make better predictions two more models were proposed using ensemble learning which are bagging ensemble with SVM (DCNN-SVM), and the ensemble of three different classifiers with a voting technique (DCNN-VC). Image data augmentation was applied to expand the dataset to improve model performance and generalization. The accuracy of the proposed DCNN model was studied by tuning hyperparameters. Performances of the three proposed models were examined in contrast with pre-trained models such as VGGNet-19, ResNet50 with a voting technique for emotion recognition. The proposed models are evaluated and achieved the best accuracy when compared with other models on five publicly available facial emotion datasets that include UIBVFED, FERG, CK+, JAFFE, and TFEID.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

The datasets used in the current study are available in the below links. UIBVFED: http://ugivia.uib.es/uibvfed/. FERG: http://grail.cs.washington.edu/projects/deepexpr/ferg-2d-db.html. CK+: https://www.pitt.edu/~emotion/ck-spread.htm. JAFFE: https://zenodo.org/record/3451524#.X3NMx2gzbIU. TFEID:http://bml.ym.edu.tw/tfeid/modules/wfdownloads/.

References

  • Kim J, Kim B, Roy PP, Jeong D (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. In: IEEE Access 7:41273–41285.

  • Mandal M, Verma M, Mathur S, Vipparthi S, Murala S, Deveerasetty K (2019) Radap: regional adaptive affinitive patterns with logical operators for facial expression recognition. IET Image Process 13:850–861

    Article  Google Scholar 

  • Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real time face detection and facial expression recognition: Development and applications to human computer interaction. Proc IEEE Conf Comput Vis Pattern Recog Workshop 5:53–53.

  • Teow MYW (2017) Understanding convolutional neural networks using a minimal model for handwritten digit recognition(2017). In: 2017 IEEE 2nd international conference on automatic control and intelligent systems (I2CACIS), Kota Kinabalu, pp 167–172.

  • Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceeding - 3rd IEEE Int Conf Autom Face Gesture Recognition, FG 1998, pp 200–205 

  • Minaee S, Abdolrashidi A (2019) Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv preprint http://arxiv.org/abs/1902.01019

  • Xie S, Hu H, Wu Y (2019) Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recognit 92:177–191

    Article  Google Scholar 

  • Connie T, Al-Shabi M, Cheah WP, Goh M (2017) Facial expression recognition using a hybrid CNN–SIFT aggregator. In: Proceedings of the MIWAI, Cham, Switzerland Springer, vol 10607. pp 139–149

  • Fan Y, Li V, Lam JCK (2020) Facial expression recognition with deeply-supervised attention network. In: IEEE transactions on affective computing, vol 3045, pp 1–1

  • Alsmirat MA, Al-Alem F, Al-Ayyoub M, Jararweh Y, Gupta B (2019) Impact of digital fingerprint image quality on the fingerprint recognition accuracy. Multimedia Tools and Applications 78(3):3649–3688

    Article  Google Scholar 

  • Aneja D, Colburn A, Faigin G, Shapiro L, Mones B (2016) Modeling stylized character expressions via deep learning. Asian conference on computer vision. Springer, Cham, pp 136–153

    Google Scholar 

  • Ashir AM, Eleyan A (2017) Facial expression recognition based on image pyramid and single-branch decision tree. Signal, Image Video Process, 11:1017–1024

    Google Scholar 

  • Bendjillali RI, Beladgham M, Merit K, Taleb-Ahmed A (2019) Improved facial expression recognition based on DWT feature for deep CNN. Electronics 8:324

    Article  Google Scholar 

  • Benitez-Garcia G, Nakamura T, Kaneko M (2017) Facial expression recognition based on local Fourier coefficients and facial Fourier descriptors. J Signal Inf Process 08:132–151

    Google Scholar 

  • Chen L-F, Yen Y-S (2007) Taiwanese Facial Expression Image Database. Institute of Brain Science, National Yang-Ming University, Taipei, Taiwan, Brain Mapping Laboratory

    Google Scholar 

  • Reddy Chirra VR, Uyyala SR, Kishore Kolli VK (2019) Deep CNN: A machine learning approach for driver drowsiness detection based on eye state. Rev d’Intelligence Artif 33:461–466 

    Article  Google Scholar 

  • Cockburn J, Bartlett M, Tanaka J, Movellan J, Pierce M, Schultz R (2008) SmileMaze: a tutoring system in real-time facial expression perception and production in children with autism spectrum disorder. In: Proceedings of the workshop facial bodily expressions control adaptation games

    Google Scholar 

  • Ekman P, Friesen WV, O’Sullivan M, Chan AYC, Diacoyanni-Tarlatzis I, Heider KG, Krause R, LeCompte WA, Pitcairn T, Bitti PER (1972) Universals and cultural differences in facial expressions of emotion. J Pers Soc Psychol 53(4):712–717

    Article  Google Scholar 

  • Ekman P, Friesen W (1978) The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Santa Clara, CA, USA

  • Farajzadeh N, Pan G, Wu Z (2014) Facial Expression recognition based on meta probability codes. Pattern Anal Appl 17:763–781

    Article  MathSciNet  Google Scholar 

  • Feutry C, Piantanida P, Bengio Y, Duhamel P (2018) Learning anonymized representations with adversarial neural networks. arXiv 1–20

  • Gogić I, Manhart M, Pandžić IS, Ahlberg J (2020) Fast facial expression recognition using local binary features and shallow neural networks. Vis Comput 36:97–112 

    Article  Google Scholar 

  • González-Lozoya S, de la Calleja J, Pellegrin L, Escalante HJ, Medina M, Benitez-Ruiz A (2020) Recognition of facial expressions based on CNN features. Multimedia Tools Appl 79:13987–14007

    Article  Google Scholar 

  • Goyani M, Patel N (2017) Multi-level Haar wavelet based facial expression recognition using logistic regression. Indian J Sci Technol 10:1–9

    Article  Google Scholar 

  • Han S, Meng Z, Khan AS, Tong Y (2016)  Incremental boosting convolutional neural network for facial action unit recognition. In: Advances in neural information processing systems, pp 109–117

  • He K, Zhang  X,  Ren S and Sun J, (2016) Deep Residual Learning for Image Recognition, In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp 770–778

  • Kim H-C, Pang S, Je H-M, Kim D, Bang S (2002) Support vector machine ensemble with bagging, vol. 2388, pp 397–407

  • Mayya V, Pai RM, Manohara Pai MM (2016) Automatic Facial Expression Recognition Using DCNN. Procedia Comput Sci 93:453–461

    Article  Google Scholar 

  • Lango M, Stefanowski J (2017) Multi-class and feature selection extensions of roughly balanced bagging for imbalanced data. J Intell Inf Syst 50(1):97–127

    Article  Google Scholar 

  • Lee SH, Plataniotis  KN,  Ro YM (2014) Intra-Class Variation Reduction Using Training Expression Images for Sparse Representation Based Facial Expression Recognition. In: IEEE Transactions on Affective Computing, vol. 5, pp 340–351

  • Li K, Jin Y, Akram MW, et al (2020) Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis Comput 36:391–404 

    Article  Google Scholar 

  • Li Y, Zeng J, Shan S, Chen X (2019) Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans Image Process 28(5):2439–2450

    Article  MathSciNet  Google Scholar 

  • Li Y, Shi H, Chen L, Jiang F (2019) Convolutional approach also benefits traditional face pattern recognition algorithm [208!] International Journal of Software Science and Computational Intelligence, vol. 11, pp 1–16

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010), The extended cohn-kanade dataset (CK+): a complete expression dataset for action unit and emotion-specified expression. In: Proceedings of the third international workshop on CVPR for human communicative behavior analysis, San Francisco, USA, pp 94–101

  • Mahesh Babu D, VenkataRamiReddy Ch, Srinivasulu Reddy U (2019) An automatic driver drowsiness detection system using DWT and RBFNN. Int J Recent Technol Eng 7(5S4):41–44

    Google Scholar 

  • Mehrabian G (2007) Nonverbal communication. Aldine, New Brunswick, NJ, USA

    Google Scholar 

  • Oliver MM, Alcover EA (2020) UIBVFed: Virtual facial expression dataset. PLoS One 15:1–10

  • Ozcan T, Basturk A (2020) Static facial expression recognition using convolutional neural networks based on transfer learning and hyperparameter optimization. Multimedia Tools and Applications 79:26587–26604

    Article  Google Scholar 

  • Perez-Gomez V, Rios-Figueroa HV, Rechy-Ramirez EJ, Mezura-Montes E, Marin-Hernandez A (2020) Feature selection on 2D and 3D geometric features to improve facial expression recognition. Sensors  20:1–20

  • Pons G, Masi D (2018) Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans Affect Comput 9:343–350

    Article  Google Scholar 

  • Pu X, Fan, Ke& Chen, Xiong&Ji, Luping & Zhou, Zhihu. (2015) Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168:1173–1180

  • Purnama J, Sari R (2019) Unobtrusive academic emotion recognition based on facial expression using rgb-d camera using adaptive-network-based fuzzy inference system (ANFIS). Int J Softw Sci Comput Intell 11:1–15

    Article  Google Scholar 

  • Ramireddy C V., Kishore KVK (2013) Facial expression classification using Kernel based PCA with fused DCT and GWT features. 2013 IEEE Int Conf Comput Intell Comput Res IEEE ICCIC, vol. 2013, pp 2–7

    Google Scholar 

  • VenkataRamiReddy Ch, Kishore KVK, Bhattacharyya D, Kim TH (2014) Multi-feature fusion based facial expression classification using DLBP and DCT. Int J Softw Eng Appl 8:55–68

    Google Scholar 

  • Reddy CVR, Reddy US, Kishore KVK (2019) Facial emotion recognition using NLPCA and SVM. Trait du Signal 36:13–22

    Article  Google Scholar 

  • Sadeghi H, Raie AA (2019) Human vision inspired feature extraction for facial expression recognition. Multimed Tools Appl 78:30335–30353

    Article  Google Scholar 

  • Sikkandar H, Thiyagarajan R (2020) Deep learning based facial expression recognition using improved Cat Swarm Optimization. J Ambient Intell Human Comput.

  • Soleymani M, Pantic M (2013) Emotionally Aware TV. Proc TVUX-2013 Work Explor Enhancing User Exp TV ACM CHI 2013

    Google Scholar 

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. CoRR

  • Verma G, Verma H (2020) Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions. Rev Socionetwork Strateg 14:171–180

    Article  Google Scholar 

  • Viola P, Jones M (2004) Robust real-time face detection. Int J Comput Vis 57:137–154

    Google Scholar 

  • Wang Q, Jia K, Liu P (2016) Design and Implementation of Remote Facial Expression Recognition Surveillance System Based on PCA and KNN Algorithms. Proc - 2015 Int Conf Intell Inf Hiding Multimed Signal Process IIH-MSP 2015, pp 314–317

    Google Scholar 

  • Whitehill J, Serpell Z, Lin YC, et al (2014) The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Trans Affect Comput 5:86–98

    Article  Google Scholar 

  • Xie S, Hu H (2019) Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans Multimedia 21:211–220

    Article  Google Scholar 

  • Xie S, Hu H, Yin Z (2017) Facial expression recognition using intraclass variation reduced features and manifold regularisation dictionary pair learning. IET Comput Vis 12(4):458–465

    Article  Google Scholar 

  • Yang B, Cao J, Ni R, Zhang Y (2018) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Article  Google Scholar 

  • Zhang H, Huang B, GuohuiTian, (2020) Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture. Pattern Recogn Lett 131:128–134

  • Zhao H, Liu Q, Yang Y (2018) Transfer Learning with Ensemble of Multiple Feature Representations. Proc - 2018 IEEE/ACIS 16th Int Conf Softw Eng Res Manag Appl SERA 2018 54–61

Download references

Funding

Not Applicable.

Author information

Authors and Affiliations

  1. Machine Learning and Data Analytics Lab, Department of Computer Applications, National Institute of Technology, Tiruchirappalli, 620015, India

    Venkata Rami Reddy Chirra & Srinivasulu Reddy Uyyala

  2. Centre of Excellence in Artificial Intelligence, National Institute of Technology, Tiruchirappalli, 620015, India

    Srinivasulu Reddy Uyyala

  3. Department of Computer Science and Engineering, VFSTR, Guntur, 522213, India

    Venkata Rami Reddy Chirra & Venkata Krishna Kishore Kolli

Authors
  1. Venkata Rami Reddy Chirra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Srinivasulu Reddy Uyyala
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Venkata Krishna Kishore Kolli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Venkata Rami Reddy Chirra.

Ethics declarations

Conflict of interest

The authors doesn’t have any conflicts of interest.

Code availability

Not Applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chirra, V.R.R., Uyyala, S.R. & Kolli, V.K.K. Virtual facial expression recognition using deep CNN with ensemble learning. J Ambient Intell Human Comput 12, 10581–10599 (2021). https://doi.org/10.1007/s12652-020-02866-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02866-3

Keywords

Search

Navigation