research-article

Emotion Recognition Based on Physiological Signals Using Convolution Neural Networks

Authors:

Tongshuai Song,

Jingjie YanAuthors Info & Claims

ICMLC '20: Proceedings of the 2020 12th International Conference on Machine Learning and Computing

Pages 161 - 165

https://doi.org/10.1145/3383972.3384003

Published: 26 May 2020 Publication History

Abstract

Physiological signal which consists of electrocardiogram (EEG) and peripheral signal is becoming increasingly important in affective computing, because of its intimacy with nerves. In this paper, two models are proposed on the basis of Convolution Neural Network (CNN) to process EEG and peripheral signals respectively. Taking the extraction of traditional features into account, the first model is based on two-dimensional Convolutional Neural Network (2D-CNN) using original EEG data, where its kernel is one-dimensional to extract same kinds of features for every channel. In the second model, we apply one-dimensional Convolution Neural Network (1D-CNN) to every channel of peripheral signals and then concatenate results for classification. Experiments have been done to evaluate our models on the MAHNOB-HCI database. As a result, in the three-category model, the classification accuracies in arousal dimension of the two models using CNN are 61.5%, 58.01% and 58%, 56.28% in valence. Compared with the classical methods, the accuracy using CNN is increased by 9.1% in arousal for EEG and 11.81% for peripheral signals, achieving state-of-the-art performance.

References

[1]

Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., ... & Yang, X. (2018). A review of emotion recognition using physiological signals. Sensors, 18(7), 2074.

[2]

Tao, F., Liu, G., & Zhao, Q. (2018, April). An ensemble framework of voice-based emotion recognition system for films and tv programs. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6209--6213). IEEE.

Digital Library

[3]

Tarnowski, P., Kołodziej, M., Majkowski, A., & Rak, R. J. (2017). Emotion recognition using facial expressions. Procedia Computer Science, 108, 1175--1184.

[4]

Majumder, N., Poria, S., Gelbukh, A., & Cambria, E. (2017). Deep learning-based document modeling for personality detection from text. IEEE Intelligent Systems, 32(2), 74--79.

Digital Library

[5]

Noroozi, F., Kaminska, D., Corneanu, C., Sapinski, T., Escalera, S., & Anbarjafari, G. (2018). Survey on emotional body gesture recognition. IEEE transactions on affective computing.

Digital Library

[6]

Zheng, W. L., Zhu, J. Y., & Lu, B. L. (2017). Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing.

Digital Library

[7]

Keren, G., Kirschstein, T., Marchi, E., Ringeval, F., & Schuller, B. (2017, July). End-to-end learning for dimensional emotion recognition from physiological signals. In 2017 IEEE International Conference on Multimedia and Expo (ICME) (pp. 985--990). IEEE.

[8]

Smitha, K. G., & Vinod, A. P. (2013, January). Hardware efficient FPGA implementation of emotion recognizer for autistic children. In 2013 IEEE International Conference on Electronics, Computing and Communication Technologies (pp. 1--4). IEEE.

[9]

Liu, W., Zhang, L., Tao, D., & Cheng, J. (2018). Reinforcement online learning for emotion prediction by using physiological signals. Pattern Recognition Letters, 107, 123--130.

[10]

E Jenkins, L. (2017). Does personality effect emotion facial recognition? A comparison between Ekman's Emotion Hexagon Test and a newly created measure. Madridge Journal of Neuroscience, 1(1), 38--46.

[11]

Mohammad, S. M. (2016). Sentiment analysis: Detecting valence, emotions, and other affectual states from text. In Emotion measurement (pp. 201--237). Woodhead Publishing.

[12]

Tsujimoto, T., Takahashi, Y., Takeuchi, S., & Maeda, Y. (2016, July). RNN with Russell's circumplex model for emotion estimation and emotional gesture generation. In 2016 IEEE Congress on Evolutionary Computation (CEC) (pp. 1427--1431). IEEE.

[13]

Hatcher, W. G., & Yu, W. (2018). A survey of deep learning: platforms, applications and emerging research trends. IEEE Access, 6, 24411--24432.

[14]

Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., ... & Patras, I. (2011). Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, 3(1), 18--31.

[15]

Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2011). A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing, 3(1), 42--55.

Digital Library

[16]

Katsigiannis, S., & Ramzan, N. (2017). DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE journal of biomedical and health informatics, 22(1), 98--107.

[17]

Duan, R. N., Zhu, J. Y., & Lu, B. L. (2013, November). Differential entropy feature for EEG-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER) (pp. 81--84). IEEE.

[18]

Zheng, W. L., & Lu, B. L. (2015). Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 7(3), 162--175.

Digital Library

[19]

Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B. W., & Zafeiriou, S. (2017). End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of Selected Topics in Signal Processing, 11(8), 1301--1309.

[20]

Zhou, Q. (2018). Multi-layer affective computing model based on emotional psychology. Electronic Commerce Research, 18(1), 109--124.

[21]

Kim, J. (2007). Bimodal emotion recognition using speech and physiological changes. In Robust speech recognition and understanding. IntechOpen.

[22]

Kim, J., & André, E. (2008). Emotion recognition based on physiological changes in music listening. IEEE transactions on pattern analysis and machine intelligence, 30(12), 2067--2083.

Digital Library

[23]

Khalili, Z., & Moradi, M. H. (2009, June). Emotion recognition system using brain and peripheral signals: using correlation dimension to improve the results of EEG. In 2009 International Joint Conference on Neural Networks (pp. 1571--1575). IEEE.

[24]

Monkaresi, H., Hussain, M. S., & Calvo, R. A. (2012, October). Classification of affects using head movement, skin color features and physiological signals. In 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 2664--2669). IEEE.

[25]

Chen, J., Hu, B., Xu, L., Moore, P., & Su, Y. (2015, November). Feature-level fusion of multimodal physiological signals for emotion recognition. In 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (pp. 395--399). IEEE.

Digital Library

[26]

Koelstra, S., & Patras, I. (2013). Fusion of facial expressions and EEG for implicit affective tagging. Image and Vision Computing, 31(2), 164--174.

Digital Library

[27]

Wiem, M. B. H., & Lachiri, Z. (2017). Emotion classification in arousal valence model using MAHNOB-HCI database. Int. J. Adv. Comput. Sci. Appl. IJACSA, 8(3).

[28]

Tripathi, S., Acharya, S., Sharma, R. D., Mittal, S., & Bhattacharya, S. (2017, February). Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset. In Twenty-Ninth IAAI Conference.

[29]

Alhagry, S., Fahmy, A. A., & El-Khoribi, R. A. (2017). Emotion recognition based on EEG using LSTM recurrent neural network. Emotion, 8(10), 355--358.

[30]

Song, T., Zheng, W., Song, P., & Cui, Z. (2018). EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing.

Cited By

Wu THuang YPurwanto ECraig P(2024)EmoMA-Net: A Novel Model for Emotion Recognition Using Hybrid Multimodal Neural Networks in Adaptive Educational SystemsProceedings of the 2024 7th International Conference on Big Data and Education10.1145/3704289.3704303(65-71)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3704289.3704303
Liu HLou TZhang YWu YXiao YJensen CZhang D(2024)EEG-Based Multimodal Emotion Recognition: A Machine Learning PerspectiveIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.336913073(1-29)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3369130
Alfaras MKaran O(2024)A Review of Advancements in Driver Emotion Detection: Deep Learning Approaches and Dataset Analysis2024 4th International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET)10.1109/IRASET60544.2024.10549432(1-9)Online publication date: 16-May-2024
https://doi.org/10.1109/IRASET60544.2024.10549432
Show More Cited By

Index Terms

Emotion Recognition Based on Physiological Signals Using Convolution Neural Networks
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI theory, concepts and models

Recommendations

Emotion Recognition Using Physiological Signals
MIDI '15: Proceedings of the Mulitimedia, Interaction, Design and Innnovation

In this paper the problem of emotion recognition using physiological signals is presented. Firstly the problems with acquisition of physiological signals related to specific human emotions are described. It is not a trivial problem to elicit real ...
Recognition of emotions using multimodal physiological signals and an ensemble deep learning model

An ensemble of deep classifiers is built for recognizing emotions using multimodal physiological signals.The higher-level abstractions of physiological features of each modality are separately extracted by deep hidden neurons in member stacked-...
Multimodal emotion recognition based on manifold learning and convolution neural network
Abstract
Multimodal emotion recognition task based on physiological signals is becoming a research hotspot. Traditional methods need to design and extract a series of features from single-channel or multi-channel physiological signals on the basis of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMLC '20: Proceedings of the 2020 12th International Conference on Machine Learning and Computing

February 2020

607 pages

ISBN:9781450376426

DOI:10.1145/3383972

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Shenzhen University: Shenzhen University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 May 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Key Research and Development Program of Jiangsu Province
the National Natural Science Foundation of China
China Postdoctoral Science Foundation

Conference

ICMLC 2020

ICMLC 2020: 2020 12th International Conference on Machine Learning and Computing

February 15 - 17, 2020

Shenzhen, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
477
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)2

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu THuang YPurwanto ECraig P(2024)EmoMA-Net: A Novel Model for Emotion Recognition Using Hybrid Multimodal Neural Networks in Adaptive Educational SystemsProceedings of the 2024 7th International Conference on Big Data and Education10.1145/3704289.3704303(65-71)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3704289.3704303
Liu HLou TZhang YWu YXiao YJensen CZhang D(2024)EEG-Based Multimodal Emotion Recognition: A Machine Learning PerspectiveIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.336913073(1-29)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3369130
Alfaras MKaran O(2024)A Review of Advancements in Driver Emotion Detection: Deep Learning Approaches and Dataset Analysis2024 4th International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET)10.1109/IRASET60544.2024.10549432(1-9)Online publication date: 16-May-2024
https://doi.org/10.1109/IRASET60544.2024.10549432
Zhu QLan TLiang JXiang DHe H(2024)Feature Alignment and Reconstruction Constraints for Multimodal Sentiment Analysis2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650976(1-6)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650976
Banik SKumar HGanapathy NSwaminathan R(2024)Exploring Central-Peripheral Nervous System Interaction Through Multimodal Biosignals: A Systematic ReviewIEEE Access10.1109/ACCESS.2024.339403612(60347-60368)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3394036
Liu YJia ZWang HEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)EmotionKD: A Cross-Modal Knowledge Distillation Framework for Emotion Recognition Based on Physiological SignalsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612277(6122-6131)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612277
Ying GHao C(2023)A novel one-dimensional convolutional neural network-based method for emotion recognition of electric power industry workersEnergy Reports10.1016/j.egyr.2023.04.2979(763-771)Online publication date: Sep-2023
https://doi.org/10.1016/j.egyr.2023.04.297
Singh JSharma D(2023)Automated detection of mental disorders using physiological signals and machine learning: A systematic review and scientometric analysisMultimedia Tools and Applications10.1007/s11042-023-17504-183:29(73329-73361)Online publication date: 10-Nov-2023
https://doi.org/10.1007/s11042-023-17504-1
Zaman KSun ZShah SShoaib MPei LHussain A(2022)Driver Emotions Recognition Based on Improved Faster R-CNN and Neural Architectural Search NetworkSymmetry10.3390/sym1404068714:4(687)Online publication date: 26-Mar-2022
https://doi.org/10.3390/sym14040687
Xefteris VTsanousa AGeorgakopoulou NDiplaris SVrochidis SKompatsiaris I(2022)Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion RecognitionSensors10.3390/s2221819822:21(8198)Online publication date: 26-Oct-2022
https://doi.org/10.3390/s22218198
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten