research-article

Multi-Label Emotion Detection via Emotion-Specified Feature Extraction and Emotion Correlation Learning

Authors:

Fuji RenAuthors Info & Claims

IEEE Transactions on Affective Computing, Volume 14, Issue 1

Pages 475 - 486

https://doi.org/10.1109/TAFFC.2020.3034215

Published: 01 January 2023 Publication History

Abstract

Textual emotion detection is an attractive task while previous studies mainly focused on polarity or single-emotion classification. However, human expressions are complex, and multiple emotions often co-occur with non-negligible emotion correlations. In this paper, a Multi-label Emotion Detection Architecture (MEDA) is proposed to detect all associated emotions expressed in a given piece of text. MEDA is mainly composed of two modules: Multi-Channel Emotion-Specified Feature Extractor (MC-ESFE) and Emotion Correlation Learner (ECorL). MEDA captures underlying emotion-specified features through MC-ESFE module, which is composed of multiple channel-wise ESFE networks. Each channel in MC-ESFE is devoted to the feature extraction of a specified emotion from sentence-level to context-level through a hierarchical structure. With underlying features, emotion correlation learning is implemented through an emotion sequence predictor in ECorL. Furthermore, we define a new loss function: multi-label focal loss. With this loss function, the model can focus more on misclassified positive-negative emotion pairs and improve the overall performance by balancing the prediction of positive and negative emotions. The evaluation of proposed MEDA architecture is carried out on emotional corpus: RenCECps and NLPCC2018 datasets. The experimental results indicate that the proposed method can achieve better performance than state-of-the-art methods in this task.

References

[1]

W. Liang, H. Xie, Y. Rao, R. Y. K. Lau, and F. L. Wang, “Universal affective model for readers’ emotion classification over short texts,” Expert Syst. Appl., vol. 114, pp. 322–333, Dec. 2018.

[2]

F. Ren and K. Matsumoto, “Semi-Automatic creation of youth slang corpus and its application to affective computing,” IEEE Trans. Affect. Comput., vol. vol.7, no. 2, pp. 176–189, Second Quarter 2016.

[3]

H. Y. Shum, X. He, and D. Li, “From eliza to xiaoice: Challenges and opportunities with social chatbots,” Front. Inf. Technol. Electron. Eng., vol. 19, no. 1, pp. 10–26, Jan. 2018.

[4]

R. Jayakrishnan, G. N. Gopal, and M. S. Santhikrishna, “Multi-class emotion detection and annotation in malayalam novels,” in Proc. Int. Conf. Comput. Commun. Inform., 2018, pp. 1–5.

[5]

T. Kowatsch, M. Nißen, C. H. I. Shih and D. Rüegger, “Text-based healthcare chatbots supporting patient and health professional teams: Preliminary results of a randomized controlled trial on childhood obesity,” in Proc. Persuasive Embodied Agents Behav. Chang., 2017, pp. 1–10.

[6]

C. Chen, R. Zhuo, and J. Ren, “Gated recurrent neural network with sentimental relations for sentiment classification,” Inf. Sci., vol. 502, pp. 268–278, Oct. 2019.

Digital Library

[7]

D. Sznycer and A. W. Lukaszewski, “The emotion–valuation constellation: Multiple emotions are governed by a common grammar of social valuation,” Evol. Hum. Behav., vol. 40, no. 4, pp. 395–404, Jul. 2019.

[8]

X. Kang, F. Ren, and Y. Wu, “Exploring latent semantic information for textual emotion recognition in blog articles,” IEEE/CAA J. Automatica Sinica, vol. 5, no. 1, pp. 204–216, Jan. 2018.

[9]

A. Bandhakavi, N. Wiratunga, and D. Padmanabhan, “Lexicon based feature extraction for emotion text classification,” Pattern Recognit. Lett., vol. 93, no. 1, pp. 133–142, Jul. 2017.

[10]

S. M. Liu and J. H. Chen, “A multi-label classification based approach for sentiment classification,” Expert Syst. Appl., vol. 42, no. 3, pp. 1083–1093, Feb. 2015.

Digital Library

[11]

N. Colneriĉ and J. Demsar, “Emotion recognition on twitter: Comparative study and training a unison model,” IEEE Trans. Affective Comput., vol. 11, no. 3, pp. 433–446, Third Quarter 2018.

[12]

D. Pan and J. Nie, “Mutux at semeval-2018 task 1: Exploring impacts of context information on emotion detection,” in Proc. 12th Int. Workshop Semantic Eval., 2018, pp. 345–349.

[13]

H. Huihui and R. Xia, “Joint binary neural network for Multi-label learning with applications to emotion classification,” in Proc. Int. Conf. Nature Lang. Process. Chin. Compu., 2018, pp. 250–259.

[14]

P. D. Turney, “Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews,” in Proc. Assoc. Comput. Linguist., 2002, pp. 417–424.

[15]

F. Ren and Y. Wu, “Predicting user-topic opinions in twitter with social and topical context,” IEEE Trans. Affective Comput., vol. 4, no. 4, pp. 412–424, Fourth Quarter 2013.

[16]

B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up? Sentiment classification using machine learning techniques,” in Proc. Conf. Empir. Methods Natural Lang. Process., 2012, pp. 79–86.

[17]

S. Mohammad and S. Kiritchenko. “Understanding emotions: A dataset of tweets to study interactions between affect categories,” in Proc. 11th Int. Conf. Lang. Resour. Eval., 2018, pp. 198–209.

[18]

N. H. Frijda, “The laws of emotion,” Amer. Psychol., vol. 43, no. 5, pp. 349–358, May 1988.

[19]

P. Ekman, “An argument for basic emotions,” Cogn. Emotion, vol. 6, no. 3-4, pp. 169–200, 1992.

[20]

Q. Changqin and F. Ren. “Sentence emotion analysis and recognition based on emotion words using Ren-cecps,” Int. J. Adv. Intell., vol. 2, no. 1, pp. 105–117, Jul. 2010.

[21]

J. Li and F. Ren, “Creating a chinese emotion lexicon based on corpus Ren-cecps,” in Proc. IEEE Int. Conf. Cloud Comp. Intell. Syst., 2011, pp. 80–84.

[22]

C. Strapparava and A. Valitutti, “Wordnet affect: An affective extension of wordnet,” in Proc. 4th Int. Conf. Lang. Resour. Eval., 2004, vol. 4, pp. 1083–1086.

[23]

S. Mohammad, “Portable features for classifying emotional text,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol., 2012, pp. 587–591.

[24]

Q. Liu and S. Li, “Word semantic similarity computation based on hownet,” Comput. Linguist. Chin. Lang. Process., vol. 7, no. 2, pp. 59–76, 2002.

[25]

S. M. Mohammad and P. D. Turney, “Crowdsourcing a word–emotion association lexicon,” Comput. Intell., vol. 29, no. 3, pp. 436–465, Sep. 2012.

[26]

F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with LSTM,” in Proc. 9th Int. Conf. Artif. Neural Netw., 1999, pp. 850–855.

[27]

S. Poria, E. Cambria, D. Hazarika, and N. Majumder, “Context-dependent sentiment analysis in user-generated videos,” in Proc. 55th Annu. Meeting Assoc. Comput. Linguist., 2017, pp. 873–883.

[28]

D. Tang, B. Qin, and T. Liu, “Aspect level sentiment classification with deep memory network,” in Proc. Conf. Empir. Methods Natural Lang. Process., 2016, pp. 214–224.

[29]

Y. Kim, “Convolutional neural networks for sentence classification,” in Proc. Conf. Empir. Methods Natural Lang. Process., 2014, pp. 1746–1751.

[30]

T. Rao, X. Li, H. Zhang, and M. Xu, “Multi-level region-based convolutional neural network for image emotion classification,” Neurocomputing, vol. 333, no. 14, pp. 429–439, Mar. 2019.

Digital Library

[31]

M. S. Akhtar, D. Ghosal, and A. Ekbal, “A Multi-task ensemble framework for emotion, sentiment and intensity prediction,” 2018,.

[32]

M. Chae, T. H. Kim, Y. H. Shin, J. W. Kim, and S. Y. Lee, “End-to-end multimodal emotion and gender recognition with dynamic weights of joint loss,” 2018,.

[33]

P. Xu, A. Madotto, C. S. Wu, J. H. Park, and P. Fung, “Emo2vec: Learning generalized emotion representation by multi-task training,” in Proc. 9th Workshop Comput. Approaches Subjectivity, Sentiment Social Media Anal., 2018, pp. 292–298.

[34]

A. Illendula and A. Sheth, “Multimodal emotion classification,” in Proc. World Wide Web Conf., 2019, pp. 439–449.

[35]

L. Buitinck, J. V. Amerongen, and E. Tan, “Multi-emotion detection in user-generated reviews,” in Proc. Eur. Conf. Inf. Retrieval, 2015, pp. 43–48.

[36]

Y. Kim, H. Lee, and K. Jung, “AttnConvnet at semeval-2018 task 1: Attention-based convolutional neural networks for multi-label emotion classification,” in Proc. 12th Int. Workshop Semantic Eval., 2018, pp. 141–145.

[37]

J. Yu, L. Marujo, J. Jiang, and P. Karuturi, “Improving multi-label emotion classification via sentiment classification with dual attention transfer network,” in Proc. Conf. Empir. Methods Natural Lang. Process., 2018, pp. 1097–1102.

[38]

D. Zhou, Y. Yang, and Y. He, “Relevant emotion ranking from text constrained with emotion relationships,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Lang. Technol., 2018, vol. 1, pp. 561–571.

[39]

M. L. Zhang and Z. H. Zhou, “Multi-label neural networks with applications to functional genomics and text categorization,” IEEE Trans. Knowl. Data Eng., vol. 18, no. 10, pp. 1338–1351, Oct. 2006.

Digital Library

[40]

P. Yang, X. Sun, W. Li, S. Ma, W. Wu, and H. Wang, “SGM: Sequence generation model for multi-label classification,” in Proc. 27th Int. Conf. Comput. Linguistics, 2018, pp. 3915–3926.

[41]

B. Zhao, X. Li, X. Lu, and Z. Wang, “A CNN–RNN architecture for multi-label weather recognition,” Neurocomputing, vol. 322, no. 17, pp. 47–57, Dec. 2018.

[42]

S. Lian, J. Liu, R. Lu, and X. Luo, “Captured multi-label relations via joint deep supervised autoencoder,” Appl. Soft. Comput., vol. 74, pp. 709–728, Jan. 2019.

[43]

C. K. Yeh, W. C. Wu, W. J. Ko, and Y. C. F. Wang, “Learning deep latent space for multi-label classification,” in Proc. 31th AAAI Conf. Artif. Intell., 2017, pp. 2838–2844.

[44]

K. Wang, M. Yang and W. Yang, “Deep correlation structure preserved label space embedding for Multi-label classification,” in Proc. Asian Conf. Mach. Learn., 2018, vol. 95, pp. 1–16.

[45]

D. A. Phan, Y. Matsumoto, and H. Shindo, “Autoencoder for semisupervised multiple emotion detection of conversation transcripts,” IEEE Trans. Affect. Comput., to be published,.

[46]

Z. F. He, M. Yang, Y. Gao, H. D. Liu, and Y. Yin, “Joint multi-label classification and label correlations with missing labels and feature selection,” Knowl. Based Syst., vol. 163, pp. 145–158, Jan. 2019.

[47]

M. Rei and A. Søgaard, “Jointly learning to label sentences and tokens,” in Proc. 33th AAAI Conf. Artif. Intell., 2019, pp. 6916–6923.

[48]

J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186.

[49]

J. Chung, C. Gulcehre, K. H. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” in NIPS Workshop Deep Learn., 2014,.

[50]

Z. Yang, D. Yang, C. Dyer, X. He, and A. Smola, “Hierarchical attention networks for document classification,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Lang. Technol., 2016, pp. 1480–1489.

[51]

T. Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2980–2988.

[52]

M. L. Zhang and Z. H. Zhou, “Multilabel neural networks with applications to functional genomics and text categorization,” IEEE Trans. Knowl. Data Eng., vol. 18, no. 10, pp. 1338–1351, Oct. 2006.

Digital Library

[53]

Z. Wang, S. Li, F. Wu, Q. Sun, and G. Zhou, “Overview of NLPCC 2018 shared task 1: Emotion detection in code-switching text,” in Proc. CCF Int. Conf. Natural Lang. Process. Chin. Comput., 2018, pp. 429–433.

[54]

S. Li, Z. Zhao, R. Hu, W. Li, T. Liu, and X. Du, “Analogical reasoning on chinese morphological and semantic relations,” in Proc. 56th Annu. Meeting Assoc. Comput. Linguist., 2018, pp. 138–143.

[55]

O. Luaces, J. Díez, J. Barranquero, and J. D. Coz, “Binary relevance efficacy for multilabel classification,” Program. Artif. Intell., vol. 1, no. 4, pp. 303–313, 2012.

[56]

J. Read, B. Pfahringer, G. Holmes, and E. Frank, “Classifier chains for multi-label classification,” Mach. Learn., vol. 85, no. 3, pp. 333–359, 2011.

Digital Library

[57]

R. Johnson and T. Zhang, “Deep pyramid convolutional neural networks for text categorization,” in Proc. 55th Annu. Meeting Assoc. Comput. Linguist., 2017, vol. 1, pp. 562–570.

[58]

J. Yu, L. Marujo, J. Jiang, P. Karuturi, and W. Brendel, “Improving multi-label emotion classification via sentiment classification with dual attention transfer network,” in Proc. Assoc. Comput. Linguistics, 2018, pp. 1097–1102.

[59]

W. Liao, Y. Wang, Y. Yin, X. Zhang, and P. Ma, “Improved sequence generation model for multi-label classification via CNN and initialized fully connection,” Neurocomputing, vol. 382, no. 21, pp. 188–195, Mar. 2020.

Digital Library

[60]

D. M. Powers, “Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation,” J. Mach. Learn. Technol., vol. 2, no. 1, pp. 37–63, Dec. 2011.

Cited By

Deng ZLi TDeng DLiu KLuo ZZhang P(2024)Feature Selection for Handling Label Ambiguity Using Weighted Label-Fuzzy Relevancy and RedundancyIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2024.339961732:8(4436-4447)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TFUZZ.2024.3399617
Zhu QZheng CZhang ZShao WZhang D(2024)Dynamic Confidence-Aware Multi-Modal Emotion RecognitionIEEE Transactions on Affective Computing10.1109/TAFFC.2023.334092415:3(1358-1370)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TAFFC.2023.3340924
Zhou YKang XRen F(2024)Prompt Consistency for Multi-Label Textual Emotion DetectionIEEE Transactions on Affective Computing10.1109/TAFFC.2023.325488315:1(121-129)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TAFFC.2023.3254883

Recommendations

Vector based 3D Emotion Expression for Emotion Robot
ICMRE'19: Proceedings of the 5th International Conference on Mechatronics and Robotics Engineering

This paper suggests the novel 3D emotion express method based on the vector for the emotional robot. The emotion function such as face emotion, voice emotion, and gesture emotion can be expressed in several ways. To express fully emotion like the human ...
Do all facial emojis communicate emotion? The impact of facial emojis on perceived sender emotion and text processing
Abstract
Facial emojis can express a variety of positive and negative emotions, and are commonly used in digital, written communication. However, little is known about how emojis impact text processing and how different emoji-text combinations give rise ...
Highlights
- We investigated how facial emojis impact text comprehension using behavioral ratings and EEG.
- In ratings, negative emojis communicate specific mental states, while positive emojis communicate general positive emotion.
- In ERPs, the ...
Customer emotion detection by emotion expression analysis on adverbs

The growing interests of affective computing (AC) and its implementation in e-business demand researchers to explore the applications of AC techniques in detecting customer emotions. Recent studies have shown that language is powerful in conveying ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Affective Computing

IEEE Transactions on Affective Computing Volume 14, Issue 1

Jan.-March 2023

863 pages

ISSN:1949-3045

Issue’s Table of Contents

1949-3045 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Computer Society Press

Washington, DC, United States

Publication History

Published: 01 January 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Deng ZLi TDeng DLiu KLuo ZZhang P(2024)Feature Selection for Handling Label Ambiguity Using Weighted Label-Fuzzy Relevancy and RedundancyIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2024.339961732:8(4436-4447)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TFUZZ.2024.3399617
Zhu QZheng CZhang ZShao WZhang D(2024)Dynamic Confidence-Aware Multi-Modal Emotion RecognitionIEEE Transactions on Affective Computing10.1109/TAFFC.2023.334092415:3(1358-1370)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TAFFC.2023.3340924
Zhou YKang XRen F(2024)Prompt Consistency for Multi-Label Textual Emotion DetectionIEEE Transactions on Affective Computing10.1109/TAFFC.2023.325488315:1(121-129)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TAFFC.2023.3254883

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents