Nothing Special   »   [go: up one dir, main page]
Skip to main content
SpringerLink
Log in

Generative AI for Explainable Automated Fact Checking on the FactEx: A New Benchmark Dataset

  • Conference paper
  • First Online:
Disinformation in Open Online Media (MISDOOM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14397))

  • 873 Accesses

Abstract

The immense volume of online information has made verifying claims’ credibility more complex, increasing interest in automatic fact-checking models that classify evidence into binary or multi-class verdicts. However, there are few studies on predicting textual verdicts to explain claims’ credibility. This field focuses on generating a textual verdict to explain a given claim based on a given news article. This paper presents our three-fold contribution to this field. Firstly, we collected the FactEx, an English dataset of facts with explanations from various fact-checking websites on different topics. Secondly, we employed seq2seq models and LLMs (namely T5, BERT2BERT, and BLOOM) to develop an automated fact-checking system. Lastly, we used ChatGPT to generate verdicts to check its performance and compared the results against other models. In addition, we explored the impact of dataset size on the model performance by conducting a series of experiments on seven different dataset sizes. The findings indicate that our fine-tuned T5-based model outperformed other generative LLMs and Seq2Seq Models with a ROUGE-1 score of about 26.75, making it the selected baseline for this task. Our study recommends examining the semantic similarity of the generative models for automatic fact-checking applications while also highlighting the importance of evaluating such models using additional techniques, such as crowd-based tools, to ensure the accuracy and reliability of the generated verdicts.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 49.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/althabiti/FactEx.

  2. 2.

    https://huggingface.co/althabiti/VerdictGen_t5-based.

  3. 3.

    https://fullfact.org/europe/vote-leave-facts-leaflet-exports/

  4. 4.

    https://fullfact.org/immigration/nigel-farage-census-london-manchester/

References

  1. Shu, K., Sliva, A., Wang, S., Tang, J., Liu, H.: Fake news detection on social media: a data mining perspective. ACM SIGKDD Explor. Newsl. 19(1), 22–36 (2017)

    Article  Google Scholar 

  2. Tandoc, E.C., Jr., Lim, Z.W., Ling, R.: Defining ‘fake news’ a typology of scholarly definitions. Digit. Journalism 6(2), 137–153 (2018)

    Article  Google Scholar 

  3. Jahng, M.R., Lee, H., Rochadiat, A.: Public relations practitioners’ management of fake news: exploring key elements and acts of information authentication. Public Relat. Rev. 46(2), 101907 (2020)

    Article  Google Scholar 

  4. Tejedor, S., Portalés-Oliva, M., Carniel-Bugs, R., Cervi, L.: Journalism students and information consumption in the era of fake news. Media Commun. 9(1), 338–350 (2021)

    Article  Google Scholar 

  5. Andorfer, A.: Spreading like wildfire: solutions for abating the fake news problem on social media via technology controls and government regulation. Hastings LJ 69, 1409 (2017)

    Google Scholar 

  6. Rapti, M., Tsakalidis, G., Petridou, S., Vergidis, K.: Fake news incidents through the lens of the DCAM disinformation blueprint. Information 13(7), 306 (2022)

    Article  Google Scholar 

  7. Rocha, Y.M., de Moura, G.A., Desidério, G.A., de Oliveira, C.H., Louren-ço, F.D., de Figueiredo Nicolete, L.D.: The impact of fake news on social media and its influence on health during the COVID-19 pandemic: a systematic review. J. Public Health (Bangkok) 31(7), 1–10 (2021)

    Google Scholar 

  8. Khan, S.A. Alkawaz, M.H. Zangana, H.M.: The use and abuse of social media for spreading fake news. In: 2019 IEEE International Conference on Automatic Control and Intelligent Systems (I2CACIS), pp. 145–148. IEEE (2019)

    Google Scholar 

  9. Zlatkova, D., Nakov, P., Koychev, I.: Fact-checking meets fauxtography: verifying claims about images. arXiv preprint arXiv:1908.11722 (2019)

  10. Guo, Z., Schlichtkrull, M., Vlachos, A.: A survey on automated fact-checking. Trans. Assoc. Comput. Linguist. 10, 178–206 (2022)

    Article  Google Scholar 

  11. Wang, W.Y.: liar, liar pants on fire: a new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648 (2017)

  12. Zeng, X., Abumansour, A.S., Zubiaga, A.: Automated fact-checking: a survey. Lang. Linguist. Compass 15(10), e12438 (2021)

    Article  Google Scholar 

  13. Naderi, N., Hirst, G.: Automated fact-checking of claims in argumentative parliamentary debates. In: Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pp. 60–65 (2018)

    Google Scholar 

  14. Alanazi, S.S., Khan, M.B.: Arabic fake news detection in social media using readers’ comments: text mining techniques in action. Int. J. Comput. Sci. Netw. Secur. 20(9), 29–35 (2020)

    Google Scholar 

  15. Althabiti, S., Alsalka, M., Atwell, E.: SCUoL at CheckThat! 2021: an AraBERT model for check-worthiness of Arabic tweets. In: CEUR Workshop Proceedings (2021)

    Google Scholar 

  16. Köhler, J.: Overview of the CLEF-2022 CheckThat! lab task 3 on fake news detection. In: Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, Bologna, Italy (2022)

    Google Scholar 

  17. Nakov, P., et al.: Overview of the CLEF-2022 CheckThat! lab on fighting the COVID-19 infodemic and fake news detection. In: Proceedings of the 13th International Conference of the CLEF Association: Information Access Evaluation meets Multilinguality, Multimodality, and Visualization, Bologna, Italy (2022)

    Google Scholar 

  18. Nakov, P., et al.: The CLEF-2022 CheckThat! lab on fighting the COVID-19 infodemic and fake news detection. In: European Conference on Information Retrieval, Springer, pp. 416–428 (2022). https://doi.org/10.1007/978-3-030-99739-7_52

  19. Althabiti, S., Alsalka, M.A., Atwell, E.: SCUoL at CheckThat! 2022: fake news detection using transformer-based models. In: CEUR Workshop Proceedings, CEUR Workshop Proceedings, pp. 428–433 (2022)

    Google Scholar 

  20. Zhou, X., Zafarani, R.: A survey of fake news: fundamental theories, detection methods, and opportunities. ACM Comput. Surv. (2020). https://doi.org/10.1145/3395046

    Article  Google Scholar 

  21. Althabiti, S., Alsalka, M.A., Atwell, E.: Detecting Arabic fake news on social media using sarcasm and hate speech in comments. Int. J. Islamic Appl. Comput. Sci. Technol. (IJASAT) 10(4), 28–36 (2022)

    Google Scholar 

  22. Alhindi, T., Petridis, S., Muresan, S.: Where is your evidence: improving fact-checking by justification modeling. In: Proceedings of the first work-shop on fact extraction and verification (FEVER), pp. 85–90 (2018)

    Google Scholar 

  23. Atanasova, P., Simonsen, J.G., Lioma, C., Augenstein, I.: Generating fact checking explanations. arXiv preprint arXiv:2004.05773 (2020)

  24. Kotonya, N., Toni, F.: Explainable automated fact-checking: a survey. arXiv preprint arXiv:2011.03870 (2020)

  25. Kotonya, N., Toni, F.: Explainable automated fact-checking for public health claims. arXiv preprint arXiv:2010.09926 (2020)

  26. Stammbach, D., Ash, E.: e-FEVER: explanations and summaries for automated fact checking. In: Proceedings of the 2020 Truth and Trust Online (TTO 2020), pp. 32–43 (2020)

    Google Scholar 

  27. Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355 (2018)

  28. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)

    MathSciNet  Google Scholar 

  29. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  30. Falaki, A.A.: How to train a seq2seq summarization model using ‘BERT’ as both encoder and decoder!! (BERT2BERT) (2022). https://pub.towardsai.net/how-to-train-a-seq2seq-summarization-model-using-bert-as-both-encoder-and-decoder-bert2bert-2a5fb36559b8

  31. Floridi, L., Chiriatti, M.: GPT-3: Its nature, scope, limits, and consequences. Minds Mach. (Dordr) 30, 681–694 (2020). https://doi.org/10.1007/s11023-020-09548-1

    Article  Google Scholar 

  32. Hassani, H., Silva, E.S.: The role of ChatGPT in data science: how Ai-assisted conversational interfaces are revolutionizing the field. Big Data Cogn. Comput. 7(2), 62 (2023)

    Article  Google Scholar 

  33. Zong, M., Krishnamachari, B.: a survey on GPT-3. arXiv preprint arXiv:2212.00857 (2022)

  34. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318 (2002)

    Google Scholar 

  35. Lin, C.-Y.: ROUGE: a package for automatic evaluation of summaries. In: Text summarization branches out, pp. 74–81 (2004)

    Google Scholar 

  36. Wazery, Y.M., Saleh, M.E., Alharbi, A., Ali, A.A.: Abstractive Arabic text summarization based on deep learning. Comput. Intell. Neurosci. 2022, 1–14 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Leeds, Leeds, LS2 9JT, UK

    Saud Althabiti, Mohammad Ammar Alsalka & Eric Atwell

  2. King Abdulaziz University, Jeddah, 22254, Saudi Arabia

    Saud Althabiti

Authors
  1. Saud Althabiti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Ammar Alsalka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric Atwell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saud Althabiti .

Editor information

Editors and Affiliations

  1. Centrum Wiskunde & Informatica, Amsterdam, The Netherlands

    Davide Ceolin

  2. University of Groningen, Groningen, The Netherlands

    Tommaso Caselli

  3. University of Amsterdam, Amsterdam, The Netherlands

    Marina Tulin

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Althabiti, S., Alsalka, M.A., Atwell, E. (2023). Generative AI for Explainable Automated Fact Checking on the FactEx: A New Benchmark Dataset. In: Ceolin, D., Caselli, T., Tulin, M. (eds) Disinformation in Open Online Media. MISDOOM 2023. Lecture Notes in Computer Science, vol 14397. Springer, Cham. https://doi.org/10.1007/978-3-031-47896-3_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47896-3_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47895-6

  • Online ISBN: 978-3-031-47896-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation