Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Multimodal dual emotion with fusion of visual sentiment for rumor detection

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, widely spreading rumors have brought devastating impacts on society, making rumor detection a significant challenge. Exsisting researches improves that image palys an important role in rumor detection. But they only pay attention to semantic features of images’ content, while neglecting visual emotions. We propose a Multimodal Dual Emotion feature for rumor detection, which consists of publish visual emotion, publish textual emotion and social emotion. Our experiments proves that the image emotion improves the rumor detection efficiency. To the best of our knowledge, this is the first study which uses visual emotion in rumor detection. The proposed Multimodal Dual Emotion feature serves as a model plugin that can be seamlessly integrated with various rumor detection models, leading to performance enhancements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Notes

  1. https://blog.flickr.net

  2. https://www.reddit.com/

  3. https://www.alexa.com/topsites

  4. https://github.com/NVIDIA/sentiment-discovery

  5. https://www.nltk.org/api/nltk.sentiment.html

  6. https://en.wikipedia.org/wiki/List_of_emoticons

References

  1. Ajao O, Bhowmik D, Zargari S (2019) Sentiment aware fake news detection on online social networks. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE pp 2507–2511

  2. Alam F, Cresci S, Chakraborty T et al. (2021a) A survey on multimodal disinformation detection. arXiv:2103.12541

  3. Alam F, Cresci S, Chakraborty T et al. (2021b) A survey on multimodal disinformation detection. arXiv preprint arXiv:2103.12541arXiv:2103.12541

  4. Alonso MA, Vilares D, Gómez-Rodríguez C et al (2021) Sentiment analysis for fake news detection. Electronics 10(11):1348

    Article  Google Scholar 

  5. Bian T, Xiao X, Xu T et al. (2020) Rumor detection on social media with bi-directional graph convolutional networks. In: Proceedings of the AAAI conference on artificial intelligence, pp 549–556

  6. Campos V, Jou B, Giro-i Nieto X (2017) From pixels to sentiment: Fine-tuning cnns for visual sentiment prediction. Image Vis Comput 65:15–22

    Article  Google Scholar 

  7. Cao J, Guo J, Li X et al. (2018) Automatic rumor detection on microblogs: A survey. arXiv:1807.03505

  8. Cao J, Qi P, Sheng Q et al. (2020) Exploring the role of visual content in fake news detection. Disinformation, Misinformation, and Fake News in Social Media pp 141–161

  9. Castillo C, Mendoza M, Poblete B (2011) Information credibility on twitter. In: Proceedings of the 20th international conference on World wide web, pp 675–684

  10. Cheema GS, Hakimov S, Müller-Budack E et al. (2021) On the role of images for analyzing claims in social media. In: CEUR Workshop Proceedings

  11. Chen J, Wu Z, Yang Z et al (2022) Multimodal fusion network with contrary latent topic memory for rumor detection. IEEE MultiMed 29(1):104–113

    Article  Google Scholar 

  12. Chen X, Zhou F, Trajcevski G et al. (2022b) Multi-view learning with distinguishable feature fusion for rumor detection. Knowl-Based Syst 108085

  13. Chen Y, Li D, Zhang P et al (2022) Cross-modal ambiguity learning for multimodal fake news detection. Proceedings of the ACM Web Conference 2022:2897–2905

    Google Scholar 

  14. Cheng M, Nazarian S, Bogdan P (2020) Vroc: Variational autoencoder-aided multi-task rumor classifier based on text. Proceedings of the web conference 2020:2892–2898

    Google Scholar 

  15. Choudhry A, Khatri I, Jain M et al. (2022) An emotion-aware multitask approach to fake news and rumor detection using transfer learning. IEEE Trans Comput Soc Syst

  16. Deng J, Dong W, Socher R et al. (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, Ieee, pp 248–255

  17. Domm P (2013) False rumor of explosion at white house causes stocks to briefly plunge; ap confirms its twitter feed was hacked. CNBC COM 23:2062

    Google Scholar 

  18. Dosovitskiy A, Beyer L, Kolesnikov A et al. (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929

  19. Fisher D, Heymann D (2020) Q &a: The novel coronavirus outbreak causing covid-19. BMC Med 18(1):1–3

    Article  Google Scholar 

  20. Friggeri A, Adamic L, Eckles D et al. (2014) Rumor cascades. In: proceedings of the international AAAI conference on web and social media, pp 101–110

  21. Gandhi A, Adhvaryu K, Poria S et al. (2022) Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Information Fusion

  22. Ghanem B, Ponzetto SP, Rosso P et al. (2021) Fakeflow: Fake news detection by modeling the flow of affective information. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp 679–689

  23. Giachanou A, Rosso P, Crestani F (2019) Leveraging emotional signals for credibility detection. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 877–880

  24. Giachanou A, Rosso P, Crestani F (2021) The impact of emotional signals on credibility assessment. J Assoc Inf Sci Technol 72(9):1117–1132

    Article  PubMed  PubMed Central  Google Scholar 

  25. Guo H, Cao J, Zhang Y et al. (2018) Rumor detection with hierarchical social attention network. In: Proceedings of the 27th ACM international conference on information and knowledge management, pp 943–951

  26. He L, Qi H, Zaretzki R (2015) Image color transfer to evoke different emotions based on color combinations. Signal,Image & Video Processing

  27. Hou JR, Kankham S (2022) More than feelings? how facebook reaction icons affect online users’ behavioral intentions toward online health rumor posts. Int Res 32(6):1978–2002

    Google Scholar 

  28. Jiang J, Wen S, Yu S et al (2016) Rumor source identification in social networks with time-varying topology. IEEE Trans Dependable Secure Comput 15(1):166–179

    Article  Google Scholar 

  29. Jin Z, Cao J, Guo H et al. (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on Multimedia, pp 795–816

  30. Jing J, Wu H, Sun J et al. (2023) Multimodal fake news detection via progressive fusion networks. Inf Process Manag 60(1):103,120

  31. Kauffmann E, Peral J, Gil D et al (2020) A framework for big data analytics in commercial social networks: A case study on sentiment analysis and fake review detection for marketing decision-making. Ind Mark Manag 90:523–537

    Article  Google Scholar 

  32. Khattar D, Goud JS, Gupta M et al. (2019) Mvae: Multimodal variational autoencoder for fake news detection. In: The world wide web conference, pp 2915–2921

  33. Kim HR, Kang H, Lee IK (2016) Image recoloring with valence-arousal emotion model. In: Computer Graphics Forum, Wiley Online Library, pp 209–216

  34. Kim HR, Kim YS, Kim SJ et al (2018) Building emotional machines: Recognizing image emotions through deep neural networks. IEEE Trans Multimed 20(11):2980–2992

    Article  MathSciNet  Google Scholar 

  35. Kondamudi MR, Sahoo SR, Chouhan L et al. (2023) A comprehensive survey of fake news in social networks: Attributes, features, and detection approaches. J King Saud Univ-Comput Inf Sci 35(6):101,571

  36. Kumari R, Ashok N, Ghosal T et al. (2022) What the fake? probing misinformation detection standing on the shoulder of novelty and emotion. Inf Process Manag 59(1):102,740

  37. Lazer DM, Baum MA, Benkler Y et al (2018) The science of fake news. Science 359(6380):1094–1096

    Article  ADS  CAS  PubMed  Google Scholar 

  38. Li Q, Zhang Q, Si L (2019) Rumor detection by exploiting user credibility information, attention and multi-task learning. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 1173–1179

  39. Liu Y, Ott M, Goyal N et al. (2019) Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692

  40. Machajdik J, Hanbury A (2010) Affective image classification using features inspired by psychology and art theory. In: Proceedings of the 18th ACM international conference on Multimedia, pp 83–92

  41. Meeker M, Wu L (2018) Internet trends 2018

  42. Mohammad S, Kiritchenko S (2018) Understanding emotions: A dataset of tweets to study interactions between affect categories. In: LREC

  43. Mohammad SM, Turney PD (2013) Crowdsourcing a word-emotion association lexicon. Comput Intell 29(3):436–465

    Article  MathSciNet  Google Scholar 

  44. Nakamura K, Levy S, Wang WY (2020) Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In: Proceedings of the 12th Language Resources and Evaluation Conference, pp 6149–6157

  45. Nan Q, Cao J, Zhu Y et al. (2021) Mdfend: Multi-domain fake news detection. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp 3343–3347

  46. Palani B, Elango S, Viswanathan KV (2022) Cb-fake: A multimodal deep learning framework for automatic fake news detection using capsule neural network and bert. Multimedia Tools and Applications 81(4):5587–5620

    Article  PubMed  Google Scholar 

  47. Pröllochs N, Bär D, Feuerriegel S (2021a) Emotions explain differences in the diffusion of true vs. false social media rumors. Sci Reports 11(1):22,721

  48. Pröllochs N, Bär D, Feuerriegel S (2021) Emotions in online rumor diffusion. EPJ Data Sci 10(1):51

    Article  Google Scholar 

  49. Rani N, Das P, Bhardwaj AK (2022) Rumor, misinformation among web: a contemporary review of rumor detection techniques during different web waves. Concurr Comput: Pract Exper 34(1):e6479

    Article  Google Scholar 

  50. Sailunaz K, Dhaliwal M, Rokne J et al (2018) Emotion detection from text and speech: a survey. Soc Netw Anal Min 8(1):1–26

    Article  Google Scholar 

  51. Shi A, Qu Z, Jia Q et al. (2020) Rumor detection of covid-19 pandemic on online social networks. In: 2020 IEEE/ACM Symposium on Edge Computing (SEC), IEEE, pp 376–381

  52. Tony R (2020) Facebook will remove misinformation about coronavirus (2020)

  53. Wang Y, Ma F, Jin Z et al. (2018) Eann: Event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining, pp 849–857

  54. Warriner AB, Kuperman V, Brysbaert M (2013) Norms of valence, arousal, and dominance for 13,915 english lemmas. Behav Res Methods 45(4):1191–1207

    Article  PubMed  Google Scholar 

  55. Wei Z, Zhang J, Lin Z et al. (2020) Learning visual emotion representations from web data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13,106–13,115

  56. Wu L, Liu P, Zhang Y (2023) See how you read? multi-reading habits fusion reasoning for multi-modal fake news detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 13,736–13,744

  57. Yang J, SMShe D (2017) Joint image emotion classification and distribution learning via deep convolutional neural. In: International Joint Conference on Artificial Intelligence, IJCAI, pp 3266–3272

  58. You Q, Luo J, Jin H et al. (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Twenty-ninth AAAI conference on artificial intelligence

  59. Zhang K, Zhu Y, Zhang W et al (2021) Cross-modal image sentiment analysis via deep correlation of textual semantic. Knowl-Based Syst 216(106):803

    Google Scholar 

  60. Zhang X, Cao J, Li X et al (2021) Mining dual emotion for fake news detection. Proceedings of the Web Conference 2021:3465–3476

    Google Scholar 

  61. Zhang X, Zhu H, Huang Y et al (2022) Sharing or not: Psychological motivations of brand rumors spread and the stop solutions. Front Psychol 13(830):002

    Google Scholar 

  62. Zhao S, Yao H, Gao Y et al (2016) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9(4):526–540

    Article  Google Scholar 

  63. Zhao S, Jia Z, Chen H et al. (2019) Pdanet: Polarity-consistent deep attention network for fine-grained visual emotion regression. In: Proceedings of the 27th ACM international conference on multimedia, pp 192–201

  64. Zheng P, Huang Z, Dou Y et al (2023) Rumor detection on social media through mining the social circles with high homogeneity. Inf Sci 642(119):083

    Google Scholar 

  65. Zhou B, Zhao H, Puig X et al (2019) Semantic understanding of scenes through the ade20k dataset. Int J Comput Vis 127(3):302–321

    Article  Google Scholar 

  66. Zubiaga A, Aker A, Bontcheva K et al (2018) Detection and resolution of rumours in social media: A survey. ACM Comput Surv (CSUR) 51(2):1–36

    Article  Google Scholar 

Download references

Acknowledgements

Funding from the Natural Science Foundation of Chongqing (CSTB2022NSCQ-MSX1415), Chongqing Municipal Education Commission of Science and Technology Research Project (KJZD-K202114401) and 2022 Postgraduate Research Capability Improvement Program Funding are gratefully acknowledged.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Beijing Technology and Business University, Beijing, 100048, China

    Ge Wang, Li Tan & Ziliang Shang

  2. Chongqing Institute of Microelectronics Industry Technology, University of Electronic Science and Technology of China, Xiyong Micro-electricity Park, Pingba District, Chongqing, 400000, China

    Li Tan

  3. Chongqing Academy of Educational Sciences, Chongqing, 400015, China

    He Liu

Authors
  1. Ge Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziliang Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. He Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Tan.

Ethics declarations

Conflict of Interests

There is no interest dispute with this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, G., Tan, L., Shang, Z. et al. Multimodal dual emotion with fusion of visual sentiment for rumor detection. Multimed Tools Appl 83, 29805–29826 (2024). https://doi.org/10.1007/s11042-023-16732-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16732-9

Keywords

Search

Navigation