Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Emotion recognition framework using multiple modalities for an effective human–computer interaction

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Human emotions are subjective reactions to objects or events that are related to diverse physiological, behavioral and intellectual changes. The research community is gaining more interest in emotion recognition due to its vast applications including human–computer interaction, virtual reality, self-driving, digital content entertainment, human behavior monitoring, and medicine. Electroencephalogram (EEG) signals that are collected from the brain are playing a massive part in the advancement of brain–computer interface systems. The current techniques that are using EEG signals for emotion recognition are lacking in subject-independent or cross-subject emotion analysis. Additionally, there is a lack of multimodal approaches that combine EEG data with other modalities. In view of the stated deficiencies, this study presents an efficient multimodal strategy for cross-subject emotion recognition utilizing EEG and facial gestures. The proposed method fuses the spectral and statistical features extracted from the EEG data with a histogram of oriented gradients and local binary patterns features extracted from the facial images. Following on, support vector machines, k-nearest neighbor, and ensemble are employed for emotion classification. Additionally, the class misbalance problem is solved using the up-sampling approach. The accuracy of the suggested method is assessed on the dataset of emotion analysis using physiological signals with tenfold cross-validation. The findings of the research study are promising, with the highest accuracy of 97.25% for valence and 96.1% for arousal, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The datasets used in this research are publicly available, and datasets can be made available by contacting the corresponding author.

References

  1. Erol BA, Majumdar A, Benavidez P, Rad P, Choo K-KR, Jamshidi M (2019) Toward artificial emotional intelligence for cooperative social human–machine interaction. IEEE Trans Comput Soc Syst 7:234–246

    Article  Google Scholar 

  2. Picard RW, Klein J (2002) Computers that recognise and respond to user emotion: theoretical and practical implications. Interact Comput 14:141–169

    Article  Google Scholar 

  3. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23:1175–1191

    Article  Google Scholar 

  4. Siddharth S, Jung T-P, Sejnowski TJ (2019) Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Trans Affect Comput 13(1):96–107

    Article  Google Scholar 

  5. Kalsum T, Anwar SM, Majid M, Khan B, Ali SM (2018) Emotion recognition from facial expressions using hybrid feature descriptors. IET Image Proc 12:1004–1012

    Article  Google Scholar 

  6. Wang X-W, Nie D, Lu B-L (2014) Emotional state classification from EEG data using machine learning approach. Neurocomputing 129:94–106

    Article  Google Scholar 

  7. Liu Y-J, Yu M, Zhao G, Song J, Ge Y, Shi Y (2017) Real-time movie-induced discrete emotion recognition from EEG signals. IEEE Trans Affect Comput 9:550–562

    Article  Google Scholar 

  8. Raheel A, Anwar SM, Majid M (2019) Emotion recognition in response to traditional and tactile enhanced multimedia using electroencephalography. Multimed Tools Appl 78:13971–13985

    Article  Google Scholar 

  9. Qayyum H, Majid M, ul Haq E, Anwar SM (2019) Generation of personalized video summaries by detecting viewer’s emotion using electroencephalography. J Vis Commun Image Represent 65:102672

    Article  Google Scholar 

  10. Mehreen A, Anwar SM, Haseeb M, Majid M, Ullah MO (2019) A hybrid scheme for drowsiness detection using wearable sensors. IEEE Sens J 19:5119–5126

    Article  Google Scholar 

  11. Raheel A, Majid M, Anwar SM, Bagci U (2019) Emotion classification in response to tactile enhanced multimedia using frequency domain features of brain signals. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 1201–1204

  12. Yoon HJ, Chung SY (2013) EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm. Comput Biol Med 43:2230–2237

    Article  Google Scholar 

  13. Ackermann P, Kohlschein C, Bitsch JÁ, Wehrle K, Jeschke S (2016) EEG-based automatic emotion recognition: Feature extraction, selection and classification methods. In: 2016 IEEE 18th International Conference on E-Health Networking, Applications and Services (Healthcom), pp 1–6

  14. Menezes MLR, Samara A, Galway L, Sant’Anna A, Verikas A, Alonso-Fernandez F et al (2017) Towards emotion recognition for virtual environments: an evaluation of eeg features on benchmark dataset. Pers Ubiquitous Comput 21:1003–1013

    Article  Google Scholar 

  15. Tomarken AJ, Davidson RJ, Wheeler RE, Kinney L (1992) Psychometric properties of resting anterior EEG asymmetry: temporal stability and internal consistency. Psychophysiology 29:576–592

    Article  Google Scholar 

  16. Nakisa B, Rastgoo MN, Tjondronegoro D, Chandran V (2018) Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors. Expert Syst Appl 93:143–155

    Article  Google Scholar 

  17. Rho S, Yeo S (2013) Bridging the semantic gap in multimedia emotion/mood recognition for ubiquitous computing environment. J Supercomput 65(1):274–286

    Article  Google Scholar 

  18. Duan R-N, Zhu J-Y, Lu B-L (2013) Differential entropy feature for EEG-based emotion classification. In: 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), pp 81–84

  19. George FP, Shaikat IM, Ferdawoos PS, Parvez MZ, Uddin J (2019) Recognition of emotional states using EEG signals based on time-frequency analysis and SVM classifier. Int J Electr Comput Eng (2088–8708), 9

  20. Vaid S, Singh P, Kaur C (2015) Classification of human emotions using multiwavelet transform based features and random forest technique. Indian J Sci Technol 8:1–7

    Article  Google Scholar 

  21. Bono V, Biswas D, Das S, Maharatna K (2016) Classifying human emotional states using wireless EEG based ERP and functional connectivity measures. In: 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pp 200–203

  22. Soundarya S (2019) An EEG based emotion recognition and classification using machine learning techniques, I. J Emerg Technol Innov Eng 5:744–750

    Google Scholar 

  23. Chen J, Zhang P, Mao Z, Huang Y, Jiang D, Zhang Y (2019) Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks. IEEE Access 7:44317–44328

    Article  Google Scholar 

  24. Jeevan RK, SP VMR, Kumar PS, Srivikas M (2019) EEG-based emotion recognition using LSTM-RNN machine learning algorithm. In: 2019 1st International Conference on Innovations in Information and Communication Technology (ICIICT), pp 1–4

  25. Thammasan N, Fukui K-I, Numao M (2016) Application of deep belief networks in eeg-based dynamic music-emotion recognition. In: 2016 International Joint Conference on Neural Networks (IJCNN), pp 881–888

  26. Prieto LAB, Oplatková ZK (2018) Emotion recognition using autoencoders and convolutional neural networks. In: Mendel, pp 113–120

  27. Li M, Xu H, Liu X, Lu S (2018) Emotion recognition from multichannel EEG signals using K-nearest neighbor classification. Technol Health Care 26:509–519

    Article  Google Scholar 

  28. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T et al (2011) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3:18–31

    Article  Google Scholar 

  29. Alazrai R, Homoud R, Alwanni H, Daoud MI (2018) EEG-based emotion recognition using quadratic time-frequency distribution. Sensors 18:2739

    Article  Google Scholar 

  30. Cimtay Y, Ekmekcioglu E (2020) Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors 20:2034

    Article  Google Scholar 

  31. Pan SJ, Tsang IW, Kwok JT, Yang Q (2010) Domain adaptation via transfer component analysis. IEEE Trans Neural Netw 22:199–210

    Article  Google Scholar 

  32. Chai X, Wang Q, Zhao Y, Liu X, Bai O, Li Y (2016) Unsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition. Comput Biol Med 79:205–214

    Article  Google Scholar 

  33. Zhang W, Wang F, Jiang Y, Xu Z, Wu S, Zhang Y (2019) Cross-subject EEG-based emotion recognition with deep domain confusion. In: International Conference on Intelligent Robotics and Applications, pp 558–570

  34. Yang F, Zhao X, Jiang W, Gao P, Liu G (2019) Multi-method fusion of cross-subject emotion recognition based on high-dimensional EEG features. Front Comput Neurosci 13:53

    Article  Google Scholar 

  35. Pandey P, Seeja K (2019) Subject independent emotion recognition from EEG using VMD and deep learning. J King Saud Univ-Comput Inf Sci. https://doi.org/10.1016/j.jksuci.2019.11.003

    Article  Google Scholar 

  36. Keelawat P, Thammasan N, Kijsirikul B, Numao M (2019) Subject-independent emotion recognition during music listening based on EEG using Deep convolutional neural networks. In: 2019 IEEE 15th International Colloquium on Signal Processing & Its Applications (CSPA), pp 21–26

  37. Gupta V, Chopda MD, Pachori RB (2018) Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sens J 19:2266–2274

    Article  Google Scholar 

  38. Yin Z, Wang Y, Liu L, Zhang W, Zhang J (2017) Cross-subject EEG feature selection for emotion recognition using transfer recursive feature elimination. Front Neurorobot 11:19

    Article  Google Scholar 

  39. Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22:1345–1359

    Article  Google Scholar 

  40. Kang H, Nam Y, Choi S (2009) Composite common spatial pattern for subject-to-subject transfer. IEEE Signal Process Lett 16:683–686

    Article  Google Scholar 

  41. Lan Z, Sourina O, Wang L, Scherer R, Müller-Putz GR (2018) Domain adaptation techniques for EEG-based emotion recognition: a comparative study on two public datasets. IEEE Trans Cognit Dev Syst 11:85–94

    Article  Google Scholar 

  42. Alyasseri ZAA, Alomari OA, Al-Betar MA, Awadallah MA, Abdulkareem KH, Mohammed MA, Kadry S, Rajinikanth V, Rho S (2022) EEG channel selection using multiobjective cuckoo search for person identification as protection system in healthcare applications. Comput Intell Neurosc 2022. https://doi.org/10.1155/2022/5974634

    Article  Google Scholar 

  43. Mehmood I, Sajjad M, Rho S, Baik SW (2016) Divide-and-conquer based summarization framework for extracting affective video content. Neurocomputing 174:393–403

    Article  Google Scholar 

  44. Wang Z-M, Hu S-Y, Song H (2019) Channel selection method for eeg emotion recognition using normalized mutual information. IEEE Access 7:143303–143311

    Article  Google Scholar 

  45. Davidson RJ, Jackson DC, Kalin NH (2000) Emotion, plasticity, context, and regulation: perspectives from affective neuroscience. Psychol Bull 126:890

    Article  Google Scholar 

  46. Özerdem MS, Polat H (2017) Emotion recognition based on EEG features in movie clips with channel selection. Brain Inform 4:241–252

    Article  Google Scholar 

  47. Khateeb M, Anwar SM, Alnowami M (2021) Multi-domain feature fusion for emotion classification using DEAP dataset. IEEE Access 9:12134–12142

    Article  Google Scholar 

  48. Bukhari M, Bajwa KB, Gillani S, Maqsood M, Durrani MY, Mehmood I et al (2020) An efficient gait recognition method for known and unknown covariate conditions. IEEE Access 9:6465–6477

    Article  Google Scholar 

  49. Maqsood M, Yasmin S, Mehmood I, Bukhari M, Kim M (2021) An efficient DA-net architecture for lung nodule segmentation. Mathematics 9:1457

    Article  Google Scholar 

  50. Sharma R, Pachori RB, Sircar P (2020) Automated emotion recognition based on higher order statistics and deep learning algorithm. Biomed Signal Process Control 58:101867

    Article  Google Scholar 

  51. Wang F, Wu S, Zhang W, Xu Z, Zhang Y, Wu C et al (2020) Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia 146:107506

    Article  Google Scholar 

  52. Bhatti AM, Majid M, Anwar SM, Khan B (2016) Human emotion recognition and analysis in response to audio music using brain signals. Comput Hum Behav 65:267–275

    Article  Google Scholar 

  53. Cheng J, Chen M, Li C, Liu Y, Song R, Liu A et al (2020) Emotion recognition from multi-channel eeg via deep forest. IEEE J Biomed Health Inform 25:453–464

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant and Korea Forestry Promotion Institute grant funded by the Korea government (MSIT and KFS) (No.2020-0-00994 and No.2021338B10-2223-CD02).

Author information

Authors and Affiliations

  1. Department of Computer Science, COMSATS University Islamabad, Attock Campus, Attock, Pakistan

    Anam Moin & Farhan Aadil

  2. R&D Department, National University of Computer and Emerging Sciences, Islamabad, 44000, Pakistan

    Zeeshan Ali

  3. Department of Computer Science and Engineering, Seoul National University of Science and Technology, Seoul, 01811, Republic of Korea

    Dongwann Kang

Authors
  1. Anam Moin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Farhan Aadil
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zeeshan Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dongwann Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dongwann Kang.

Ethics declarations

Conflict of interest

All authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moin, A., Aadil, F., Ali, Z. et al. Emotion recognition framework using multiple modalities for an effective human–computer interaction. J Supercomput 79, 9320–9349 (2023). https://doi.org/10.1007/s11227-022-05026-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-05026-w

Keywords