Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Facial Emotion Intensity: A Fusion Way

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Many scientific works have been conducted for developing the Emotion intensity recognition system. But developing a system that is capable to estimate small to peak intensity levels with less complexity is still challenging. Therefore, we propose an effective facial emotion intensity classifier by fusion of the pre-trained deep architecture and fuzzy inference system. The pre-trained architecture VGG16 is used for basic emotion classification and it predicts emotion class with the class index value. By class index value, images are sent to the corresponding Fuzzy inference system for estimating the intensity level of detected emotion. This fusion model effectively identifies the facial emotions (happy, sad, surprise, and angry) and also predict the 13 categories of emotion intensity. This fusion model got 83% accuracy on a combined dataset (FER 2013, CK + and KDEF). The performance and findings of this proposed work are further compared with state-of-the-art models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Chiu I, Piguet O, Diehl-Schmid J, Riedl L, Beck J, Leyhe T, Holsboer-Trachsler E, Kressig RW, Berres M, Monsch AU, Sollberger M. Facial emotion recognition performance differentiates between behavioral variant frontotemporal dementia and major depressive disorder. J Clin Psychiatry. 2018. https://doi.org/10.4088/JCP.16M11342.

    Article  Google Scholar 

  2. Chu HC, Tsai WWJ, Liao MJ, Chen YM. Facial emotion recognition with transition detection for students with high-functioning autism in adaptive e-learning. Soft Comput. 2018;22:2973–99. https://doi.org/10.1007/S00500-017-2549-Z.

    Article  Google Scholar 

  3. Oh KJ, Lee D, Ko B, Choi HJ. A chatbot for psychiatric counseling in mental healthcare service based on emotional dialogue analysis and sentence generation. In: Proc—18th IEEE Int Conf Mob Data Manag MDM 2017. Institute of Electrical and Electronics Engineers Inc.; 2017;371–6. https://doi.org/10.1109/MDM.2017.64.

  4. Wang L, Geng X. Behavioral biometrics for human identification: intelligent applications. Med Inf Sci Ref. 2010;505:44–56.

  5. Saste ST, Jagdale SM. Emotion recognition from speech using MFCC and DWT for security system. In: Proc Int Conf Electron Commun Aerosp Technol ICECA 2017. Institute of Electrical and Electronics Engineers Inc.; 2017;2017-January: 701–4. https://doi.org/10.1109/ICECA.2017.8203631.

  6. Seli P, Wammes JD, risko EF, Smilek D. On the relation between motivation and retention in educational contexts: The role of intentional and unintentional mind wandering. Psychon Bull Rev. 2016;23:1280–7. https://doi.org/10.3758/S13423-015-0979-0.

    Article  Google Scholar 

  7. Zhao J, Mao X, Chen L. Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomed Signal Process Control. 2019;47:312–23. https://doi.org/10.1016/J.BSPC.2018.08.035.

    Article  Google Scholar 

  8. Borth D, Ji R, Chen T, Breuel T, Chang SF. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: MM 2013—Proc 2013 ACM Multimed Conf. Association for Computing Machinery; 2013; p. 223–32. https://doi.org/10.1145/2502081.2502282.

  9. Sajjanhar A, Wu Z, Wen Q. Deep learning models for facial expression recognition. In: 2018 Int Conf Digit Image Comput Tech Appl DICTA 2018. Institute of Electrical and Electronics Engineers Inc.; 2019; p. 1–6. https://doi.org/10.1109/DICTA.2018.8615843.

  10. Soleymani M, Asghari-Esfeden S, Fu Y, Pantic M. Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans Affect Comput. 2016;7:17–28. https://doi.org/10.1109/TAFFC.2015.2436926.

    Article  Google Scholar 

  11. Schindler K, Van Gool L, de Gelder B. Recognizing emotions expressed by body pose: a biologically inspired neural model. Neural Netw. 2008;21:1238–46. https://doi.org/10.1016/J.NEUNET.2008.05.003.

    Article  Google Scholar 

  12. Kosti R, Alvarez JM, Recasens A, Lapedriza A. Context based emotion recognition using EMOTIC dataset. IEEE Trans Pattern Anal Mach Intell IEEE. 2020;42:2755–66. https://doi.org/10.1109/TPAMI.2019.2916866.

    Article  Google Scholar 

  13. Bhattacharya, S. A Survey on: facial expression recognition using various deep learning techniques. In: Advanced computational paradigms and hybrid intelligent computing. Springer, Singapore. 2022; 1373:619–31. https://doi.org/10.1007/978-981-16-4369-9_59.

  14. Rassadin A, Gruzdev A, Savchenko A. Group-Level emotion recognition using transfer learning from face identification. In: ICMI 2017—Proc 19th ACM Int Conf Multimodal Interact. 2017; 2017-Janua:544–8. https://doi.org/10.1145/3136755.3143007.

  15. Kishan Kondaveeti H, Vishal Goud M. Emotion detection using deep facial features. In: Proc IEEE Int Conf Advent Trends Multidiscip Res Innov ICATMRI 2020. 2020; p. 1–8. https://doi.org/10.1109/ICATMRI51801.2020.9398439.

  16. Islam MN, Loo CK. Geometric feature-based facial emotion recognition using two-stage fuzzy reasoning model. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics). 2014;8835:344–51. https://doi.org/10.1007/978-3-319-12640-1_42.

    Article  Google Scholar 

  17. Bahreini K, van der Vegt W, Westera W. A fuzzy logic approach to reliable real-time recognition of facial emotions. Multimed Tools Appl. 2019;78:18943–66. https://doi.org/10.1007/s11042-019-7250-z.

    Article  Google Scholar 

  18. Esau N, Wetzel E, Kleinjohann L, Kleinjohann B. Real-time facial expression recognition using a fuzzy emotion model. In: IEEE Int Conf Fuzzy Syst. 2007; p. 1–6. https://doi.org/10.1109/FUZZY.2007.4295451.

  19. Amal VS, Suresh S, Deepa G. Real-time emotion recognition from facial expressions using convolutional neural network with Fer2013 dataset. In: Ubiquitous intelligent systems. Springer, Singapore. 2022; 243:541–51. https://doi.org/10.1007/978-981-16-3675-2_41.

  20. Nicolai A, Choi A. Facial emotion recognition using fuzzy systems. In: Proc—2015 IEEE Int Conf Syst Man, Cybern SMC 2015. 2016; p. 2216–21. https://doi.org/10.1109/SMC.2015.387.

  21. Farahani FS, Sheikhan M, Farrokhi A. A fuzzy approach for facial emotion recognition. In: 13th Iran Conf Fuzzy Syst IFSC 2013. 2013; p. 1–4. https://doi.org/10.1109/IFSC.2013.6675597.

  22. Chakraborty A, Konar A, Chakraborty UK, Chatterjee A. Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Trans Syst Man Cybern Part A Syst Hum. 2009;39:726–43. https://doi.org/10.1109/TSMCA.2009.2014645.

    Article  Google Scholar 

  23. Witzig P, Kennedy J, Segalin C. Smile intensity detection in multiparty interaction using deep learning. In: 2019 8th Int Conf Affect Comput Intell Interact Work Demos, ACIIW 2019. 2019; p. 168–74. https://doi.org/10.1109/ACIIW.2019.8925261.

  24. Vinola C, Vimala DK. Smile intensity recognition in real time videos: fuzzy system approach. Multimed Tools Appl. 2019;78:15033–52. https://doi.org/10.1007/s11042-018-6890-8.

    Article  Google Scholar 

  25. Savran A, Sankur B, Taha BM. Regression-based intensity estimation of facial action units. Image Vis Comput. 2012;30:774–84. https://doi.org/10.1016/J.IMAVIS.2011.11.008.

    Article  Google Scholar 

  26. Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J. Toward practical smile detection. IEEE Trans Pattern Anal Mach Intell. 2009;31:2106–11. https://doi.org/10.1109/TPAMI.2009.42.

    Article  Google Scholar 

  27. Ozdemir MA, Elagoz B, Alaybeyoglu A, Sadighzadeh R, Akan A. Real time emotion recognition from facial expressions using CNN architecture. In: TIPTEKNO 2019—Tip Teknol Kongresi. 2019; p. 529–32. https://doi.org/10.1109/TIPTEKNO.2019.8895215.

  28. Ahmed TU, Hossain S, Hossain MS, Ul Islam R, Andersson K. Facial expression recognition using convolutional neural network with data augmentation. In: 2019 Jt 8th Int Conf Informatics, Electron Vision, ICIEV 2019 3rd Int Conf Imaging, Vis Pattern Recognition, icIVPR 2019 with Int Conf Act Behav Comput ABC 2019. 2019; p. 336–41. https://doi.org/10.1109/ICIEV.2019.8858529.

  29. Kim JH, Kim BG, Roy PP, Jeong DM. Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access. 2019;7:41273–85.

    Article  Google Scholar 

  30. Song Z. Facial expression emotion recognition model integrating philosophy and machine learning theory. Front Psychol. 2021;12:759485. https://doi.org/10.3389/fpsyg.2021.759485.

    Article  Google Scholar 

  31. Park SJ, Kim BG, Chilamkurti N. A robust facial expression recognition algorithm based on multi-rate feature fusion scheme. Sensors. 2021;21(21):1–26.

    Article  Google Scholar 

  32. Chu J, Guo Z, Leng L. Object detection based on multi-layer convolution feature fusion and online hard example mining. IEEE Access. 2018;6:19959–67.

    Article  Google Scholar 

  33. Zhang Y, Chu J, Leng L, Miao J. Mask-refined R-CNN: a network for refining object details in instance segmentation. Sensors (Switzerland). 2020;20(4):1010.

    Article  Google Scholar 

  34. Dhall A, Goecke R, Gedeon T. Automatic group happiness intensity analysis. IEEE Trans Affect Comput IEEE. 2015;6:13–26. https://doi.org/10.1109/TAFFC.2015.2397456.

    Article  Google Scholar 

  35. Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv: 2014:1409–1556.

  36. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee DH, Zhou Y, Ramaiah C, Feng F, Li R, Wang X, Athanasakis D, Shawe-Taylor J, Milakov M, Park J, Ionescu R, Popescu M, Grozea C, Bergstra J, Xie J, Romaszko L, Xu B, Chuang Z, Bengio Y. Challenges in representation learning: a report on three machine learning contests. In: Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics). 2013;8228 LNCS:117–24. https://doi.org/10.1007/978-3-642-42051-1_16.

  37. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Comput Soc Conf Comput Vis Pattern Recognit—Work CVPRW 2010. 2010; p. 94–101. https://doi.org/10.1109/CVPRW.2010.5543262.

  38. Lundqvist D, Flykt A, Öhman A. The Karolinska directed emotional faces—KDEF. Department of Clinical Neuroscience, Psychology Section, Karolinska Institutet; 1998.

  39. Vaidya S. Corrective re-annotation of FER-CK+-KDEF 2020. Available at: https://www.kaggle.com/sudarshanvaidya/corrective-reannotation-of-fer-ck-kdef. Accessed 3 May 2021.

  40. Liew CF, Yairi T. Facial expression recognition and analysis: a comparison study of feature descriptors. IPSJ Trans Comput Vis Appl. 2015;7:104–20. https://doi.org/10.2197/ipsjtcva.7.104.

    Article  Google Scholar 

  41. Li J, Lam EY. Facial expression recognition using deep neural networks. In: IST 2015—2015 IEEE Int Conf Imaging Syst Tech Proc. IEEE; 2015; p. 1–6. https://doi.org/10.1109/IST.2015.7294547.

  42. Liu X, Kumar BVKV, You J, Jia P. Adaptive deep metric learning for identity-aware facial expression recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2017; p. 522–31. https://doi.org/10.1109/CVPRW.2017.79.

  43. Alshamsi H, Kepuska V, Meng H. Real time automated facial expression recognition app development on smart phones. In: 2017 8th IEEE Annu Inf Technol Electron Mob Commun Conf IEMCON 2017. 2017; p. 384–92. https://doi.org/10.1109/IEMCON.2017.8117150.

  44. Zeng N, Zhang H, Song B, Liu W, Li Y, Dobaie AM. Facial expression recognition via learning deep sparse autoencoders. Neurocomputing [Internet]. 2018;273:643–9. https://doi.org/10.1016/j.neucom.2017.08.043.

    Article  Google Scholar 

  45. Kim BK, Dong SY, Roh J, Kim G, Lee SY. Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach. In: IEEE Comput Soc Conf Comput Vis Pattern Recognit Work. 2016; p. 1499–508. https://doi.org/10.1109/CVPRW.2016.187.

  46. Pramerdorfer C, Kampel M. Facial Expression Recognition using Convolutional Neural Networks: State of the Art. 2016; arXiv: http://arxiv.org/abs/1612.02903.

  47. Georgescu MI, Ionescu RT, Popescu M. Local learning with deep and handcrafted features for facial expression recognition. IEEE Access IEEE. 2019;7:64827–36. https://doi.org/10.1109/ACCESS.2019.2917266.

    Article  Google Scholar 

  48. Girard JM, Cohn JF, De La Torre F. Estimating smile intensity: A better way. Pattern Recognit Lett. 2015;66:13–21. https://doi.org/10.1016/j.patrec.2014.10.004.

    Article  Google Scholar 

  49. Leng L, Li M, Kim C, Bi X. Dual-source discrimination power analysis for multi-instance contactless palmprint recognition. Multimed Tools Appl [Internet]. 2017;76(1):333–54. https://doi.org/10.1007/s11042-015-3058-7.

    Article  Google Scholar 

  50. Leng L, Zhang J. PalmHash code vs. palmPhasor code. Neurocomputing [Internet]. 2013;108:1–12. https://doi.org/10.1016/j.neucom.2012.08.028.

    Article  Google Scholar 

  51. Pitaloka DA, Wulandari A, Basaruddin T, Liliana DY. Enhancing CNN with preprocessing stage in automatic emotion recognition. Proc Comput Sci. 2017;116:523–9. https://doi.org/10.1016/j.procs.2017.10.038.

    Article  Google Scholar 

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Mathematics, Statistics and Computer Science, G. B. Pant University of Agriculture and Technology, Pantnagar, 263145, India

    Ankita Pandey & Arun Kumar

Authors
  1. Ankita Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arun Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ankita Pandey.

Ethics declarations

Conflict of Interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pandey, A., Kumar, A. Facial Emotion Intensity: A Fusion Way. SN COMPUT. SCI. 3, 162 (2022). https://doi.org/10.1007/s42979-022-01049-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01049-5

Keywords

Search

Navigation