Abstract
Discourse coherence is an important characteristic of well-written texts and coherent speech. It is observed at several levels of discourse analysis: lexical, syntactic, semantic, and pragmatic. Recent work on discourse coherence uses deep neural network architectures to model coherence. However, most of these architectures are not linguistically explainable. In this paper, we propose a fine-tuned Large Language Model (LLM) and three interpretable approaches for modeling discourse coherence, that target different levels of discourse analysis and coherence information, capturing contextual information, semantic relatedness between adjacent sentences and paragraphs, and syntactic patterns of coherent texts. We want to determine whether these explainable approaches lead to competitive results compared to the proposed fine-tuned LLM. These architectures are evaluated on the multi-domain Grammarly Corpus of Discourse Coherence (GCDC) and compared to state-of-the-art (SOTA) and recent models. The results of our experiments show that the syntactic patterns combined with the semantic relatedness are a good indicator of the overall coherence and highlight the importance of the number of training examples for the model’s ability to use the information provided by the syntactic patterns to make accurate predictions. Furthermore, the contextual information captured by the transformer-based model achieves good results that significantly outperform all other models, showing that the use of a fine-tuned LLM is now typically the best performing approach, despite being less interpretable than other methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Barzilay, R., Lapata, M.: Modeling local coherence: an entity-based approach. Comput. Linguist. 34(1), 1–34 (2008)
Berrar, D.: Cross-validation. In: Encyclopedia of Bioinformatics and Computational Biology, pp. 542–545. Academic Press, Oxford (2019)
Bird, S., Klein, E., Loper, E.: Natural language processing with Python: analyzing text with the natural language toolkit, chap. 5, pp. 179–219. O’Reilly Media (2009)
Brownlee, J.: Hyperparameter optimization with random search and grid search. Machine Learning Mastery (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186. ACL (2019)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Jeon, S., Strube, M.: Entity-based neural local coherence modeling. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7787–7805 (2022)
Laban, P., Dai, L., Bandarkar, L., Hearst, M.A.: Can transformer models measure coherence in text: re-thinking the shuffle test. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on NLP (V2: Short Papers), pp. 1058–1064 (2021)
Lai, A., Tetreault, J.: Discourse coherence in the wild: a dataset, evaluation and methods. In: Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pp. 214–223 (2018)
Li, J., Jurafsky, D.: Neural net models of open-domain discourse coherence. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 198–209 (2017)
Mann, W.C., Thompson, S.A.: Rhetorical structure theory: a theory of text organization. University of Southern California, Information Sciences Institute (1987)
Mesgar, M., Strube, M.: A neural local coherence model for text quality assessment. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4328–4339 (2018)
Nguyen, D.T., Joty, S.: A neural local coherence model. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1320–1330 (2017)
Oufaida, H., Blache, P., Nouali, O.: A coherence model for sentence ordering. In: Métais, E., Meziane, F., Vadera, S., Sugumaran, V., Saraee, M. (eds.) NLDB 2019. LNCS, vol. 11608, pp. 261–273. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23281-8_21
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. ACL (2014)
Pitler, E., Nenkova, A.: Revisiting readability: a unified framework for predicting text quality. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 186–195 (2008)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Telenyk, S., Pogorilyy, S., Kramov, A.: Evaluation of the coherence of polish texts using neural network models. Appl. Sci. 11(7), 3210 (2021)
Wang, Y., Lee, H.Y., Chen, Y.N.: Tree transformer: integrating tree structures into self-attention. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1061–1070. ACL (2019)
Xue, L., et al.: mT5: a massively multilingual pre-trained text-to-text transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 483–498. ACL (2021)
Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Neural Information Processing Systems (2019)
Acknowledgements
The authors would like to thank Alice Lai and Joel Tetreault for making their dataset and code available, and Sungho Jeon and Michael Strube for sharing their code. Thanks also to the anonymous reviewers for their comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors declare that they have no known competing financial interests or personal relationships that could have potentially influenced the results presented in this paper.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Azrou, L., Oufaida, H., Blache, P., Hamdine, I. (2024). Using Neural Coherence Models to Assess Discourse Coherence. In: Nöth, E., Horák, A., Sojka, P. (eds) Text, Speech, and Dialogue. TSD 2024. Lecture Notes in Computer Science(), vol 15048. Springer, Cham. https://doi.org/10.1007/978-3-031-70563-2_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-70563-2_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70562-5
Online ISBN: 978-3-031-70563-2
eBook Packages: Computer ScienceComputer Science (R0)