Abstract
In recent years, the advancement of Artificial Intelligence (AI) technology has brought both convenience and panic. One of the most notable AI systems in recent years was ChatGPT in 2022. In 2023, GPT-4 was released as the latest version. Scholars are increasingly investigating the potential of ChatGPT/GPT-4 for text generation and summarization. Inspired by the principle of “Faithfulness, Expressiveness, and Elegance” in translation, this study investigates the writing and summarizing capabilities of GPT-4, one of the latest AI chatbots. For this purpose, we collected 60 articles from top financial and technology journals, extracted the abstract part, and fed it into GPT-4 to generate abstracts. Three evaluation metrics were created for evaluation: the Text Relevance Score, the AI Detector Score, and the Plagiarism Detector Score. Our findings indicate that abstracts generated by GPT-4 closely resemble the original abstracts without being detected by the plagiarism detector Turnitin in most cases. This implies that GPT-4 can produce logical and reasonable abstracts of articles on its own. Also, we conducted a cross-temporal analysis of GPT-4’s effectiveness and observed continuous and significant improvement. Nevertheless, with the advancement of AI detectors, the abstracts generated by GPT-4 can broadly be recognized as AI-generated. Furthermore, this paper also discusses ethical concerns and future research directions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Haenlein, M., Kaplan, A.: A brief history of artificial intelligence: on the past, present, and future of artificial intelligence. Calif. Manage. Rev. 61, 5–14 (2019)
Weizenbaum, J.: ELIZA—a computer program for the study of natural language communication between man and machine. Commun. ACM 9, 36–45 (1966)
Swinbanks, D., Anderson, C.: Japan stubs its toes on fifth-generation computer. Nature 356, 273–274 (1992)
Campbell, M., Hoane, A.J., Hsu, F.: Deep blue. Artif. Intell. 134, 57–83 (2002)
Breazeal, C.: Toward sociable robots. Robot. Auton. Syst. 42, 167–175 (2003)
Ferrucci, D.A.: Introduction to “this is watson.” IBM J. Res. Dev. 56, 1:1–1:15 (2012)
Silver, D., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Radford, A., Narasimhan, K.: Improving language understanding by generative pre-training (2018)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
Brown, T., et al.: Language models are few-shot learners. arXiv (Cornell University) (2020)
Ouyang, L., et al.: Training language models to follow instructions with human feedback. arXiv (Cornell University) (2022)
Natalie: ChatGPT—Release Notes. https://help.openai.com/en/articles/6825453-chatgpt-release-notes. Accessed 28 Aug 2023
Baruffati, A.: Chat GPT statistics 2023: trends and the future perspectives GITNUX. https://blog.gitnux.com/chat-gpt-statistics/. Accessed 27 Sept 2023
van Dis, E.A.M., Bollen, J., Zuidema, W., van Rooij, R., Bockting, C.L.: ChatGPT: five priorities for research. Nature 614, 224–226 (2023)
Stokel-Walker, C.: ChatGPT listed as author on research papers: many scientists disapprove. Nature 613 (2023)
Thorp, H.H.: ChatGPT is fun, but not an author. Science 379, 313 (2023)
Gao, C.A., et al.: Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv (2022)
Dowling, M., Lucey, B.: ChatGPT for (finance) research: the Bananarama conjecture. Financ. Res. Lett. 53, 103662 (2023)
Cascella, M., Montomoli, J., Bellini, V., Bignami, E.: Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J. Med. Syst. 47 (2023)
OpenAI: GPT-4 technical report. arXiv (Cornell University). (2023)
Nori, H., King, N., McKinney, S.M., Carignan, D., Horvitz, E.: Capabilities of GPT-4 on medical challenge problems. arXiv (2023)
Bubeck, S., et al.: Sparks of artificial general intelligence: early experiments with GPT-4. arXiv (Cornell University) (2023)
Fu, Y.: Tianyan Lun yi liyan [Preface to the evolution and ethics]. In: Luo, X. (ed.) Fanyi Lunji [An Anthology of Chinese Translation Theories]. Commercial Press, Beijing (1898)
Liu, Y., et al: Summary of ChatGPT-related research and perspective towards the future of large language models. arXiv (Cornell University) (2023)
Zeng, A., et al.: GLM-130B: an open bilingual pre-trained model. arXiv (Cornell University) (2022)
Thoppilan, R., et al.: LaMDA: language models for dialog applications. arXiv (2022)
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., et al.: PaLM: scaling language modeling with pathways. arXiv (Cornell University) (2022)
Katz, D.M., Bommarito, M.J., Gao, S., Arredondo, P.: GPT-4 passes the bar exam. SSRN Electron. J. (2023)
Sanderson, K.: GPT-4 is here: what scientists think. Nature (2023)
Lee, P., Bubeck, S., Petro, J.: Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N. Engl. J. Med. 388, 1233–1239 (2023)
Ufuk, F., Peker, H., Sagtas, E., Yagci, A.B.: Distinguishing GPT-4-generated radiology abstracts from original abstracts: performance of blinded human observers and AI content detector. medRxiv (2023)
Li, X., Zhu, X., Ma, Z., Liu, X., Shah, S.: Are ChatGPT and GPT-4 general-purpose solvers for financial text analytics? An examination on several typical tasks. arXiv (Cornell University) (2023)
Mack, C.A.: How to Write a Good Scientific Paper. Washington, Usa Spie Press, Bellingham (2018)
Khalil, M., Er, E.: Will ChatGPT get you caught? Rethinking of plagiarism detection. arXiv (Cornell University) (2023)
Dwivedi, Y.K., et al.: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int. J. Inf. Manage. 71, 102642 (2023)
Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., Smith, A.: Generating scholarly content with ChatGPT: ethical challenges for medical publishing. Lancet Digit. Health 5 (2023)
Rahimi, F., Talebi Bezmin Abadi, A.: ChatGPT and publication ethics. Arch. Med. Res. 54 (2023)
Acknowledgments
This research was funded by The Science and Technology Development Fund, Macau SAR (File no. 0091/2020/A2).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, B., Chen, Q., Lin, J., Li, S., Yen, J. (2024). Assessing GPT-4 Generated Abstracts: Text Relevance and Detectors Based on Faithfulness, Expressiveness, and Elegance Principle. In: Tan, Y., Shi, Y. (eds) Data Mining and Big Data. DMBD 2023. Communications in Computer and Information Science, vol 2017. Springer, Singapore. https://doi.org/10.1007/978-981-97-0837-6_12
Download citation
DOI: https://doi.org/10.1007/978-981-97-0837-6_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-0836-9
Online ISBN: 978-981-97-0837-6
eBook Packages: Computer ScienceComputer Science (R0)