Abstract
The standard approach for abstractive text summarization is to use an encoder-decoder architecture. The encoder is responsible for capturing the general meaning from the source text, and the decoder is in charge of generating the final text summary. While this approach can compose summaries that resemble human writing, some may contain unrelated or unfaithful information. This problem is called “hallucination” and it represents a serious issue in legal texts as legal practitioners rely on these summaries when looking for precedents, used to support legal arguments. Another concern is that legal documents tend to be very long and may not be fed entirely to the encoder. We propose our method called LegalSumm for addressing these issues by creating different “views” over the source text, training summarization models to generate independent versions of summaries, and applying entailment module to judge how faithful these candidate summaries are with respect to the source text. We show that the proposed approach can select candidate summaries that improve ROUGE scores in all metrics evaluated.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Beltagy I, Peters ME, Cohan A (2020) Longformer: The long-document transformer arXiv preprint arXiv:200405150
Cao Z, Wei F, Li W, Li S (2018) Faithful to the original: fact aware neural abstractive summarization. In: thirty-second AAAI conference on artificial intelligence
Carbonell J, Goldstein J (1998) The use of mmr, diversity-based reranking for reordering documents and producing summaries. In: proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval, pp 335–336
Chandrasekaran MK, Yasunaga M, Radev D, Freitag D, Kan MY (2019) Overview and results: Cl-scisumm shared task 2019. In: in proceedings of joint workshop on bibliometric-enhanced information retrieval and NLP for digital libraries (BIRNDL 2019)
Child R, Gray S, Radford A, Sutskever I (2019) Generating long sequences with sparse transformers. arXiv preprint arXiv:190410509
Compton P, Jansen R (1990) Knowledge in context: A strategy for expert system maintenance. In: proceedings of the second Australian joint conference on artificial intelligence, Springer-Verlag, Berlin, Heidelberg, AI ’88, p 292–306
Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805
Falke T, Ribeiro LF, Utama PA, Dagan I, Gurevych I (2019) Ranking generated summaries by correctness: an interesting but challenging application for natural language inference. In: proceedings of the 57th annual meeting of the association for computational linguistics, pp 2214–2220
Fan A, Grangier D, Auli M (2018) Controllable abstractive summarization. In: proceedings of the 2nd workshop on neural machine translation and generation, 45–54
Feijo D, Moreira V (2018) Rulingbr: A summarization dataset for legal texts. In: computational processing of the portuguese language (PROPOR 2018), Springer International Publishing, pp 255–264
Feijo D, Moreira V (2019) Summarizing legal rulings: comparative experiments. in: proceedings of the international conference on recent advances in natural language processing (RANLP 2019), pp 313–322
Galgani F, Compton P, Hoffmann A (2012). Combining different summarization techniques for legal text. In: proceedings of the workshop on innovative hybrid approaches to the processing of textual data, Association for Computational Linguistics, pp 115–123
Gelbart D, Smith J (1991) Beyond boolean search: Flexicon, a legal text-based intelligent system. In: proceedings of the 3rd international conference on Artificial intelligence and law, pp 225–234
Goldhan D, Eckart T, Quasthoff U (2012) Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In: proceedings of the 8th international language resources and evaluation (LREC’12)
Goodrich B, Rao V, Liu PJ, Saleh M (2019) Assessing the factual accuracy of generated text. In: proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 166–175
Grover C, Hachey B, Hughson I, Korycinski C (2003a) Automatic summarisation of legal documents. In: proceedings of the 9th international conference on artificial intelligence and law, association for computing machinery, ICAIL ’03, p 243–251
Grover C, Hachey B, Korycinski C (2003b). Summarising legal texts: Sentential tense and argumentative roles. In: proceedings of the HLT-NAACL 03 on text summarization workshop-volume 5, association for computational linguistics, pp 33–40
Guimarães JAC (2011) Elaboração de ementas jurisprudenciais: elementos teórico-metodológicos. Série Monografias do CEJ 9
Kitaev N, Kaiser Ł, Levskaya A (2020) Reformer: the efficient transformer arXiv preprint arXiv:200104451
Klein G, Kim Y, Deng Y, Senellart J, Rush A (2017). OpenNMT: Open-source toolkit for neural machine translation. In: proceedings of ACL 2017, system demonstrations, association for computational linguistics, Vancouver, Canada, pp 67–72
Kryściński W, McCann B, Xiong C, Socher R (2020) Evaluating the factual consistency of abstractive text summarization. In: proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 9332–9346
Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L (2020) BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: proceedings of the 58th annual meeting of the association for computational linguistics, association for computational linguistics, Online, pp 7871–7880, https://doi.org/10.18653/v1/2020.acl-main.703, https://aclanthology.org/2020.acl-main.703
Lin CY (2004) Rouge: a package for automatic evaluation of summaries. In: text summarization branches out, pp 74–81
Liu Y (2019) Fine-tune BERT for extractive summarization. arXiv preprint arXiv:190310318
Liu Y, Lapata M (2019) Fine-tune BERT for extractive summarization. arXiv preprint arXiv:190310318
Luijtgaarden N (2019) Automatic summarization of legal text. Utrecht University Master’s thesis
Matsumaru K, Takase S, Okazaki N (2020) Text summarization with pretrained encoders. In: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 3721–3731
Maynez J, Narayan S, Bohnet B, McDonald R (2020) On faithfulness and factuality in abstractive summarization. In: proceedings of the 58th annual meeting of the association for computational linguistics, pp 1906–1919
Moens MF, Uyttendaele C (1997) Automatic text structuring and categorization as a first step in summarizing legal cases. Inf Proces Manag 33(6):727–737
Mudrakarta PK, Taly A, Sundararajan M, Dhamdhere K (2018) Did the model understand the question? In: proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 1896–1906
Pandya V (2019) Automatic text summarization of legal cases: A hybrid approach. 5th international conference on advances in computer science and information technology (ACSTY-2019)
Paulus R, Xiong C, Socher R (2017) A deep reinforced model for abstractive summarization. arXiv preprint arXiv:170504304
Pires T, Schlinger E, Garrette D (2019) How multilingual is multilingual BERT? In: proceedings of the 57th annual meeting of the association for computational linguistics, pp 4996–5001
Roy A, Saffar M, Vaswani A, Grangier D (2021) Efficient content-based sparse attention with routing transformers. Transac Assoc Comput Linguistics 9:53–68
See A, Liu PJ, Manning CD (2017) Get to the point: Summarization with pointer-generator networks. In: proceedings of the 55th annual meeting of the association for computational linguistics 1:1073–1083
Tay Y, Bahri D, Yang L, Metzler D, Juan DC (2020) Sparse sinkhorn attention. In: international conference on machine learning, PMLR, pp 9438–9447
Turtle H (1995) Text retrieval in the legal world. Artif Intell and Law 3(1):5–54
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: proceedings of the 31st international conference on neural information processing systems, pp 6000–6010
Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, et al. (2019) Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:191003771
Yousfi-Monod M, Farzindar A, Lapalme G (2010) Supervised machine learning for summarizing legal documents. In: Canadian conference on artificial intelligence, Springer, pp 51–62
Zhang J, Zhao Y, Saleh M, Liu P (2020) Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In: international conference on machine learning, PMLR, pp 11328–11339
Zhang X, Wei F, Zhou M (2019) HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In: proceedings of the 57th annual meeting of the association for computational linguistics, pp 5059–5069
Zhao Z, Cohen SB, Webber B (2020) Reducing quantity hallucinations in abstractive summarization. arXiv preprint arXiv:200913312
Zhong L, Zhong Z, Zhao Z, Wang S, Ashley KD, Grabmair M (2019) Automatic summarization of legal decisions using iterative masking of predictive sentences. In: proceedings of the seventeenth international conference on artificial intelligence and law, ICAIL ’19, p 163–172
Acknowledgements
This work was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. The authors thank the two anonymous reviewers whose suggestions helped improve and clarify this manuscript.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Feijo, D.d., Moreira, V.P. Improving abstractive summarization of legal rulings through textual entailment. Artif Intell Law 31, 91–113 (2023). https://doi.org/10.1007/s10506-021-09305-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10506-021-09305-4