research-article

BCMF: : A bidirectional cross-modal fusion model for fake news detection

Authors:

Gang LiAuthors Info & Claims

Volume 59, Issue 5

https://doi.org/10.1016/j.ipm.2022.103063

Published: 01 September 2022 Publication History

Highlights

•

We propose a novel model, namely BCMF, for fake news detection.

•

BCMF leverages both contextualized visual embeddings and bi-directional fusions.

•

We propose a bi-directional cross-modal aggregation mechanism to deeply fuse the visual and textual information.

•

The model outperforms most of the state-of-the-art methods on four datasets.

•

The research sheds light on the role of bidirectional cross-modal fusion.

Abstract

In recent years, fake news detection has been a significant task attracting much attention. However, most current approaches utilize the features from a single modality, such as text or image, while the comprehensive fusion between features of different modalities has been ignored. To deal with the above problem, we propose a novel model named Bidirectional Cross-Modal Fusion (BCMF), which comprehensively integrates the textual and visual representations in a bidirectional manner. Specifically, the proposed model is decomposed into four submodules, i.e., the input embedding, the image2text fusion, the text2image fusion, and the prediction module. We conduct intensive experiments on four real-world datasets, i.e., Weibo, Twitter, Politi, and Gossip. The results show 2.2, 2.5, 4.9, and 3.1 percentage points of improvements in classification accuracy compared to the state-of-the-art methods on Weibo, Twitter, Politi, and Gossip, respectively. The experimental results suggest that the proposed model could better capture integrated information of different modalities and has high generalizability among different datasets. Further experiments suggest that the bidirectional fusions, the number of multi-attention heads, and the aggregating function could impact the performance of the cross-modal fake news detection. The research sheds light on the role of bidirectional cross-modal fusion in leveraging multi-modal information to improve the effect of fake news detection.

References

[1]

O. Ajao, D. Bhowmik, S. Zargari, Sentiment aware fake news detection on online social networks, in: Proceedings of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Piscataway, 2019, pp. 2507–2511.

[2]

A. Barrientos-Baez, A.M. Martinez-Sala, V.P. Altamirano, D.C. Dominguez, Fake news: The COVID-19 pandemic and its chronology in the tourism sector, Historia Y Comunicacion Social 26 (2021) 135–148,.

[3]

C. Boididou, S. Papadopoulos, D.T. Dang-Nguyen, G. Boato, M. Riegler, G. Boato, A. Petlund, Y. Kompatsiaris, The VMU participation @ verifying multimedia use 2016, in: Proceedings of the working notes proceedings of the mediaeval 2016 workshop, 2016.

[4]

A. Choudhary, A. Arora, Linguistic feature based learning model for fake news detection and classification, Expert systems with applications 169 (1) (2021),.

[5]

S.K.W. Chu, R. Xie, Y. Wang, Cross-Language Fake News Detection, Data and Information Management 5 (1) (2021) 100–109,.

[6]

L. Cui, S. Wang, D Lee, SAME: sentiment-aware multi-modal embedding for detecting fake news, in: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM '19), ACM, New York, 2019, pp. 41–48,.

Digital Library

[7]

J. Devlin, M.W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, ACL, 2019, pp. 4171–4186,. Minneapolis.

[8]

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021) An image is worth 16×16 words: transformers for image recognition at scale. In Proceedings of the 9th international conference on learning representations (pp. 1–22). Vienna: OpenReview.net.10.48550/arXiv:2010.11929.

[9]

S. Girgis, E. Amer, M. Gadallah, Deep learning algorithms for detecting fake news in online text, in: Proceedings of 2018 13th international conference on computer engineering and systems (ICCES), IEEE, Piscataway, 2018, pp. 93–97,.

[10]

MH. Goldani, R. Safabakhsh, S. Momtazi, Convolutional neural network with margin loss for fake news detection, Information Processing & Management 58 (1) (2021),.

[11]

S. Hakak, M. Alazab, S. Khan, T.R. Gadekallu, P.K.R. Maddikunta, W.Z. Khan, An ensemble machine learning approach through effective feature extraction to classify fake news, Future Generation Computer Systems 117 (2021) 47–58,.

[12]

A. Hurriyetoglu, E. Yoruk, O. Mutlu, F. Durusan, C. Yoltar, D. Yuret, B. Gurel, Cross-context news corpus for protest event-related knowledge base construction, Data Intelligence 3 (2) (2021) 308–335,.

[13]

Z. Jin, J. Cao, Y. Zhang, J. Zhou, Q. Tian, Novel visual and statistical image features for microblogs news verification, IEEE Transactions on Multimedia 19 (3) (2017) 598–608,.

Digital Library

[14]

D. Khattar, J.S. Goud, M. Gupta, V. Varma, MVAE: Multimodal variational autoencoder for fake news detection, in: Proceedings of the world wide web conference (WWW '19), ACM, New York, 2019, pp. 2915–2921,.

Digital Library

[15]

Y. Kim, Convolutional neural networks for sentence classification, in: Proceedings of the 2014 conference on empirical methods in natural language processing, 2014, pp. 1746–1751,. Stroudsburg: ACL.

[16]

D. Kingma, J Ba, Adam: a method for stochastic optimization, in: Proceedings of the 3rd international conference for learning representations, San Diego, 2014, pp. 1–15,.

[17]

R. Kumari, N. Ashok, T. Ghosal, A. Ekbal, Misinformation detection using multitask learning with mutual learning for novelty detection and emotion recognition, Information Processing & Management 58 (5) (2021),.

Digital Library

[18]

R. Kumari, N. Ashok, T. Ghosal, A. Ekbal, What the fake? Probing misinformation detection standing on the shoulder of novelty and emotion, Information Processing & Management 58 (5) (2021),.

Digital Library

[19]

R. Kumari, A. Ekbal, AMFB: Attention based multimodal factorized bilinear pooling for multimodal fake news detection, Expert Systems with Applications 184 (2021),.

Digital Library

[20]

Y. Li, Z. Fan, X. Yuan, X. Zhang, Recognizing fake information through a developed feature scheme: A user study of health misinformation on social media in China, Information Processing & Management 59 (2021),.

Digital Library

[21]

B. Liu, F.X. Han, D. Niu, L.L. Kong, K.F. Lai, Y. Xu, Story forest: Extracting events and telling stories from breaking news, ACM Transactions on Knowledge Discovery from Data (TKDD) 14 (3) (2020) 1–28,.

Digital Library

[22]

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, …., V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach, arXiv preprint, 2019.

[23]

J. Ma, W. Gao, P. Mitra, S. Kwon, B.J. Jansen, K.F. Wong, M. Cha, Detecting rumors from microblogs with recurrent neural networks, in: Proceedings of the 25th international joint conference on artificial intelligence(IJCAI-16), AAAI Press, Palo Alto, 2016, pp. 3818–3824.

[24]

J. Ma, W. Gao, Z. Wei, Y. Lu, K.F. Wong, Detect rumors using time series of social context information on microblogging websites, in: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, ACM, New York, 2015, pp. 1751–1754,.

Digital Library

[25]

Y. Matalon, O. Magdaci, A. Almozlino, D Yamin, Using sentiment analysis to predict opinion inversion in tweets of political communication, Scientific reports 11 (1) (2021) 1–9,.

[26]

M. Potthast, J. Kiesel, K. Reinartz, J. Bevendorff, B. Stein, A stylometric inquiry into hyperpartisan and fake news, in: Proceedings of the 56th annual meeting of the association for computational linguistics, 1, Long Papers, 2018.

[27]

P. Qi, J. Cao, T. Yang, J. Guo, J. Li, Exploiting multi-domain visual information for fake news detection, in: 2019 IEEE international conference on data mining (ICDM), IEEE, Piscataway, 2019, pp. 518–527,.

[28]

S.S. Qian, J.G. Wang, J. Hu, Q. Fang, C.S. Xu, Hierarchical multi-modal contextual attention network for fake news detection, in: Proceedings of the SIGIR'21: The 44th International ACM SIGIR conference on research and development in information retrieval, ACM, New York, 2021, pp. 153–162,.

Digital Library

[29]

P. Rao, M. Taboada, Gender bias in the news: A scalable topic modelling and visualization framework, Frontiers in Artificial Intelligence 4 (2021),.

[30]

N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using siamese BERT-networks, arXiv preprint, 2019.

[31]

O. Repp, H. Ramampiaro, Extracting news events from microblogs, Journal of Statistics and Management Systems 21 (4) (2018) 695–723,.

[32]

N. Ruchansky, S. Seo, Y. Liu, Csi: A hybrid deep model for fake news detection, in: Proceedings of the 2017 ACM on conference on information and knowledge management(CIKM ’17), ACM, New York, 2017, pp. 797–806.

[33]

M. Samadi, M. Mousavian, S. Momtazi, Deep contextualized text representation and learning for fake news detection, Information Processing & Management 58 (1) (2021),.

Digital Library

[34]

K. Shu, D. Mahudeswaran, S Wang, D.W. Lee, H. Liu, FakeNewsNet: A data repository with news content, social context and spatiotemporal information for studying fake news on social media, Big Data 8 (3) (2020) 171–188,.

[35]

K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: Proceedings of the 3th international conference on learning representations, 2015, pp. 1–14,.

[36]

B. Singh, D.K. Sharma, Predicting image credibility in fake news over social media using multi-modal approach, Neural computing and applications, 2021, pp. 1–15,.

Digital Library

[37]

S. Singhal, A. Kabra, M. Sharma, R.R. Shah, T. Chakraborty, P. Kumaraguru, SpotFake+: A multimodal framework for fake news detection via transfer learning (student abstract, in: Proceedings of the thirty-fourth AAAI conference on artificial intelligence (AAAI-20), AAAI, New York, 2020, pp. 13915–13916,.

[38]

S. Singhal, R.R. Shah, T. Chakraborty, P. Kumaraguru, S.I. Satoh, SpotFake: A multi-modal framework for fake news detection, in: Proceedings of the 2019 IEEE 5th International Conference on Multimedia Big Data (BigMM), IEEE, Piscataway, 2019, pp. 39–47,.

[39]

C. Song, N. Ning, Y. Zhang, B. Wu, A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks, Information Processing & Management 58 (1) (2021),.

[40]

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. Jégou, Training data-efficient image transformers & distillation through attention, in: Proceedings of the 38th international conference on machine learning, Virtual Event: PMLR, 2021, pp. 10347–10357,.

[41]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Proceeding of the advances in neural information processing systems, 30, MIT Press, Cambridge, 2017, pp. 5998–6008,.

[42]

Y. Wang, F. Ma, Z.W. Jin, Y. Yuan, G.X. Xun, EANN: Event adversarial neural networks for multi-modal fake news detection, in: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, ACM, New York, 2018, pp. 849–857,.

Digital Library

[43]

J. Wehrmann, C. Kolling, R.C. Barros, Adaptive cross-modal embeddings for image-text alignment, in: Proceedings of the AAAI conference on artificial intelligence, 34, 2020, pp. 12313–12320,.

[44]

Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, Q.V. Le, XLNet: Generalized Autoregressive Pretraining for Language Understanding, ACM, New York, 2019, pp. 5753–5763.

[45]

F. Yu, Q. Liu, S. Wu, L. Wang, T. Tan, A convolutional approach for misinformation identification, in: Proceedings of the 26th international joint conference on artificial intelligence(IJCAI-17), AAAI Press, Palo Alto, 2017, pp. 3901–3907,.

[46]

X. Zhang, J. Cao, X. Li, Q. Sheng, L. Zhong, K. Shu, Mining dual emotion for fake news detection, in: Proceedings of the web conference 2021 (WWW '21), ACM, New York, 2021, pp. 3465–3476,.

Digital Library

[47]

X.C. Zhang, A.A. Ghorbani, An overview of online fake news: Characterization, detection, and discussion, Information Processing & Management 57 (2) (2020),.

Digital Library

[48]

X.Y. Zhou, J.D. Wu, R Zafarani, Safe: Similarity-aware multi-modal fake news detection, in: Proceedings of the 24th pacific-asia conference on knowledge discovery and data mining (PAKDD 2020), Springer, Berlin, Germany, 2020, pp. 354–367,. Lecture Notes in Computer Science12085.

Digital Library

[49]

A. Zrnec, M. Pozenel, D Lavbic, Users’ ability to perceive misinformation: An information quality assessment approach, Information Processing & Management 59 (1) (2022),.

Digital Library

Cited By

Chen ZHui SZhuang FLiao LJia MLi JHuang H(2024)A syntactic evidence network model for fact verificationNeural Networks10.1016/j.neunet.2024.106424178:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106424
Luvembe ALi WLi SLiu FWu X(2024)CAF-ODNNInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10365361:3Online publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103653
Ouyang QLin NZhou YYang AZhou D(2024)Enhancing Few-Shot Multi-modal Fake News Detection Through Adaptive FusionWeb and Big Data10.1007/978-981-97-7241-4_27(432-447)Online publication date: 31-Aug-2024
https://dl.acm.org/doi/10.1007/978-981-97-7241-4_27

Index Terms

BCMF: A bidirectional cross-modal fusion model for fake news detection
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information retrieval
  2. Information systems applications

Index terms have been assigned to the content through auto-classification.

Recommendations

EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection
KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

As news reading on social media becomes more and more popular, fake news becomes a major issue concerning the public and government. The fake news can take advantage of multimedia content to mislead readers and get dissemination, which can cause ...
TRIMOON: Two-Round Inconsistency-based Multi-modal fusion Network for fake news detection
Abstract
Compared to ordinary news, fake news is characterized by faster dissemination and lower production cost and therefore causes a great social harm. For these reasons, the challenge to efficiently and accurately detect fake news has attracted a lot ...
Highlights
- A two-round inconsistency-based multi-modal fusion network for fake news detection.
- Effectively suppresses fusion noise when images and text are inconsistent.
- A multi-modal fusion mechanism for strengthening the leading role of ...
A Multimodal Fake News Detection Model with Self-supervised Unimodal Label Generation
Advanced Intelligent Computing Technology and Applications
Abstract
Fake news detection has become a hot topic. Most multimodal fake news detection models only focus on the semantic correlation between single modalities and often ignore the semantic differences between single modalities, which limited the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Information Processing and Management: an International Journal

Information Processing and Management: an International Journal Volume 59, Issue 5

Sep 2022

730 pages

ISSN:0306-4573

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 September 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen ZHui SZhuang FLiao LJia MLi JHuang H(2024)A syntactic evidence network model for fact verificationNeural Networks10.1016/j.neunet.2024.106424178:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106424
Luvembe ALi WLi SLiu FWu X(2024)CAF-ODNNInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10365361:3Online publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103653
Ouyang QLin NZhou YYang AZhou D(2024)Enhancing Few-Shot Multi-modal Fake News Detection Through Adaptive FusionWeb and Big Data10.1007/978-981-97-7241-4_27(432-447)Online publication date: 31-Aug-2024
https://dl.acm.org/doi/10.1007/978-981-97-7241-4_27
Papadopoulos SKoutlis CPapadopoulos SPetrantonakis P(2023)Synthetic Misinformers: Generating and Combating Multimodal MisinformationProceedings of the 2nd ACM International Workshop on Multimedia AI against Disinformation10.1145/3592572.3592842(36-44)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592572.3592842

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents