Computer Science and Information Systems 2024 Volume 21, Issue 3, Pages: 899-921
https://doi.org/10.2298/CSIS230912017M
Full text (
261 KB)
Reaching quality and efficiency with a parameter-efficient controllable sentence simplification approach
Menta Antonio (E.T.S.I. Informática (UNED) C. de Juan del Rosal, Madrid, Spain), amenta1@alumno.uned.es
Garcia-Serrano Ana (E.T.S.I. Informática (UNED) C. de Juan del Rosal, Madrid, Spain), agarcia@lsi.uned.es
The task of Automatic Text Simplification (ATS) aims to transform texts to improve their readability and comprehensibility. Current solutions are based on Large Language Models (LLM). These models have high performance but require powerful computing resources and large amounts of data to be fine-tuned when working in specific and technical domains. This prevents most researchers from adapting the models to their area of study. The main contributions of this research are as follows: (1) proposing an accurate solution when powerful resources are not available, using the transfer learning capabilities across different domains with a set of linguistic features using a reduced size pre-trained language model (T5-small) and making it accessible to a broader range of researchers and individuals; (2) the evaluation of our model on two well-known datasets, Turkcorpus and ASSET, and the analysis of the influence of control tokens on the SimpleText corpus, focusing on the domains of Computer Science and Medicine. Finally, a detailed discussion comparing our approach with state-of-the-art models for sentence simplification is included.
Keywords: Text Simplification, Transfer Learning, Language Models
Show references
Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework (2019)
Alarcon, R., Moreno, L., Martínez, P., Macías, J.A.: Easier system. evaluating a spanish lexical simplification proposal with people with cognitive impairments. International Journal of Human-Computer Interaction 0(0), 1-15 (2022)
Althunayyan, S., Azmi, A.: Automated text simplification: A survey. ACM Computing Surveys 54, Article no. 43 (03 2021)
Alva-Manchego, F., Bingel, J., Paetzold, G., Scarton, C., Specia, L.: Learning how to simplify from explicit labeling of complex-simplified text pairs. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 295-305. Asian Federation of Natural Language Processing, Taipei, Taiwan (Nov 2017)
Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., Specia, L.: ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4668-4679. Association for Computational Linguistics, Online (Jul 2020)
Alva-Manchego, F., Martin, L., Scarton, C., Specia, L.: EASSE: Easier automatic sentence simplification evaluation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. pp. 49-54. Association for Computational Linguistics, Hong Kong, China (Nov 2019)
Alva-Manchego, F., Scarton, C., Specia, L.: Data-driven sentence simplification: Survey and benchmark. Computational Linguistics 46(1), 135-187 (2020)
Amancio, M., Specia, L.: An analysis of crowdsourced text simplifications. In: Proceedings of the 3rdWorkshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). pp. 123-130. Association for Computational Linguistics, Gothenburg, Sweden (Apr 2014)
Campillos-Llanos, L., Terroba, A., Zakhir, S., Valverde-Mateos, A., Capllonch-Carrión, A.: Building a comparable corpus and a benchmark for spanish medical text simplification. Procesamiento del Lenguaje Natural 69, 189-196 (2022)
Campillos-Llanos, L., Valverde-Mateos, A., Capllonch-Carrión, A., Moreno-Sandoval, A.: A clinical trials corpus annotated with umls entities to enhance the access to evidence-based medicine (2021)
Carroll, J., Minnen, G., Canning, Y., Devlin, S., Tait, J.: Practical simplification of english newspaper text to assist aphasic readers. Proceedings of AAAI’98 Workshop on Integrating Artificial Intelligence and Assistive Technology pp. 7-10 (1998)
Chandrasekar, R., Doran, C., Srinivas, B.: Motivations and methods for text simplification. In: COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics (1996)
Clive, J., Cao, K., Rei, M.: Control prefixes for parameter-efficient text generation. In: Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). pp. 363-382. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid) (Dec 2022)
Cumbicus-Pineda, O.M., Gonzalez-Dios, I., Soroa, A.: A syntax-aware edit-based system for text simplification. In: Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). pp. 324-334. INCOMA Ltd., Held Online (Sep 2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding (2019)
Dong, Y., Li, Z., Rezagholizadeh, M., Cheung, J.C.K.: EditNTS: An neural programmerinterpreter model for sentence simplification through explicit editing. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 3393-3402. Association for Computational Linguistics, Florence, Italy (Jul 2019)
Drndarević, B., Saggion, H.: Towards automatic lexical simplification in Spanish: An empirical study. In: Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations. pp. 8-16. Association for Computational Linguistics, Montréal, Canada (Jun 2012)
Ermakova, L., Bellot, P., Braslavski, P., Kamps, J., Mothe, J., Nurbakova, D., Ovchinnikova, I., San-Juan, E.: Overview of simpletext clef 2021.workshop and pilot tasks. In Conference and Labs of the Evaluation Forum (2021)
Ermakova, L., Mothe, J., SanJuan, E.: Overview of the simpletext workshop at inforsid-2021: Scientific text simplification and popularization. In Conference and Labs of the Evaluation Forum (2021)
Ermakova, L., Ovchinnikov, I., Kamps, J., Nurbakova, D., Araújo, S., Hannachi, R.: Overview of the clef 2022 simpletext task 3: Query biased simplification of scientific texts. Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation, Bologna, Italy pp. 2792-2804 (2022)
Evans, R.J., Orasan, C., Dornescu, I.: An evaluation of syntactic simplification rules for people with autism. In: PITR@EACL (2014)
Feng, Y., Qiang, J., Li, Y., Yuan, Y., Zhu, Y.: Sentence simplification via large language models. ArXiv abs/2302.11957 (2023)
Févry, T., Phang, J.: Unsupervised sentence compression using denoising auto-encoders. In: Proceedings of the 22nd Conference on Computational Natural Language Learning. pp. 413- 422. Association for Computational Linguistics, Brussels, Belgium (Oct 2018)
Ficler, J., Goldberg, Y.: Controlling linguistic style aspects in neural language generation. In: Proceedings of theWorkshop on Stylistic Variation. pp. 94-104. Association for Computational Linguistics, Copenhagen, Denmark (Sep 2017)
Glavaš, G., Štajner, S.: Simplifying lexical simplification: Do we need simplified corpora? In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pp. 63-68. Association for Computational Linguistics, Beijing, China (Jul 2015)
Horn, C., Manduca, C., Kauchak, D.: Learning a lexical simplifier using Wikipedia. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 458-463. Association for Computational Linguistics, Baltimore, Maryland (Jun 2014)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 328-339. Association for Computational Linguistics, Melbourne, Australia (Jul 2018)
Khallaf, N., Sharoff, S., Soliman, R.: Towards Arabic sentence simplification via classification and generative approaches. In: Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP). pp. 43-52. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid) (Dec 2022)
Khan, A.W., Al-Obeidat, F., Khalid, A., Amin, A., Moreira, F.: Sentence embedding approach using lstm auto-encoder for discussion threads summarization. In: Computer Science and Information Systems, Vol. 20, No. 4. (2023)
Kikuchi, Y., Neubig, G., Sasano, R., Takamura, H., Okumura, M.: Controlling output length in neural encoder-decoders. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 1328-1338. Association for Computational Linguistics, Austin, Texas (Nov 2016)
Kincaid, P., Fishburne, R.P., Rogers, R.L., Chissom, B.S.: Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (1975)
Lastra-Díaz, J.J., Lara-Clares, A., Garcia-Serrano, A.: Hesml: a real-time semantic measures library for the biomedical domain with a reproducible survey. BMC bioinformatics 23(1), 23- 23 (2022)
Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady 10, 707-710 (1965)
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7871-7880. Association for Computational Linguistics, Online (Jul 2020)
Li, B., Yang, P., Zhao, H., Zhang, P., Liu, Z.: Hierarchical sliding inference generator for question-driven abstractive answer summarization. ACM Trans. Inf. Syst. 41(1) (2023)
Mallinson, J., Sennrich, R., Lapata, M.: Sentence compression for arbitrary languages via multilingual pivoting. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2453-2464. Association for Computational Linguistics, Brussels, Belgium (Oct-Nov 2018)
Martin, L., Fan, A., de la Clergerie, É., Bordes, A., Sagot, B.: MUSS: Multilingual unsupervised sentence simplification by mining paraphrases. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference. pp. 1651-1664. European Language Resources Association, Marseille, France (Jun 2022)
Martin, L., Humeau, S., Mazaré, P.E., de La Clergerie, É., Bordes, A., Sagot, B.: Referenceless quality estimation of text simplification systems. In: Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA). pp. 29-38. Association for Computational Linguistics, Tilburg, the Netherlands (Nov 2018)
Martin, L., Sagot, B., de la Clergerie, É., Bordes, A.: Controllable sentence simplification. arXiv preprint arXiv:1910.02677 (2019)
Menta, A., Sánchez-Salido, E., García-Serrano, A.: Transcripción de periódicos históricos: aproximación clara-hd. SEPLN-PD 2022: Annual Conference of the Spanish Association for Natural Language Processing 2022: Projects and Demonstrations 69, 70-74 (2022)
Menta, A., García-Serrano, A.: Controllable sentence simplification using transfer learning. In: Simple Text Task at PAN - CLEF 2022 - Conference and Labs of the Evaluation Forum, Bolonia, Septb. 5-8, 2022. CEUR-WS, vol 3180, pp: 2818-2825. ISSN: 1613-0073 https://ceurws.org/Vol-3180/ (2022)
Monteiro, J.C., Aguiar, M., Araújo, S.: Using a pre-trained simplet5 model for text simplification in a limited corpus. In: Conference and Labs of the Evaluation Forum (2022)
Moreno Sandoval, A., Gisbert, A., Montoro Zamorano, H.: Fint-esp: A corpus of financial reports in spanish (01 2020)
Nisioi, S., Štajner, S., Ponzetto, S.P., Dinu, L.P.: Exploring neural text simplification models. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 85-91. Association for Computational Linguistics, Vancouver, Canada (Jul 2017)
North, K., Ranasinghe, T., Shardlow, M., Zampieri, M.: Deep learning approaches to lexical simplification: A survey. arXiv (05 2023)
Paetzold, G., Specia, L.: SemEval 2016 task 11: Complex word identification. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). pp. 560-569. Association for Computational Linguistics, San Diego, California (Jun 2016)
Paetzold, G., Specia, L.: Lexical simplification with neural ranking. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. pp. 34-40. Association for Computational Linguistics, Valencia, Spain (Apr 2017)
Paetzold, G.H., Specia, L.: A survey on lexical simplification. J. Artif. Int. Res. 60(1), 549-593 (sep 2017)
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1) (jan 2020)
Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Conference on Empirical Methods in Natural Language Processing (2019)
Rello, L., Baeza-Yates, R.A., Dempere-Marco, L., Saggion, H.: Frequent words improve readability and short words improve understandability for people with dyslexia. In: IFIP TC13 International Conference on Human-Computer Interaction (2013)
Rubio, A., Martínez, P.: Hulat-uc3m at simpletext@clef-2022: Scientific text simplification using bart. In: Conference and Labs of the Evaluation Forum (2022)
Saggion, H., Hirst, G.: Automatic Text Simplification. Morgan & Claypool Publishers (2017)
Saggion, H., Štajner, S., Bott, S., Mille, S., Rello, L., Drndarevic, B.: Making it simplext: Implementation and evaluation of a text simplification system for spanish. ACM Transactions on Accessible Computing 6(4) (may 2015)
Scarton, C., Specia, L.: Learning simplifications for specific target audiences. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 712-718. Association for Computational Linguistics, Melbourne, Australia (Jul 2018)
Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: A unified embedding for face recognition and clustering. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (jun 2015)
Sheang, K.C., Saggion, H.: Controllable sentence simplification with a unified text-to-text transfer transformer. In: Proceedings of the 14th International Conference on Natural Language Generation. pp. 341-352. Association for Computational Linguistics, Aberdeen, Scotland, UK (Aug 2021)
Štajner, S., Sheang, K., Saggion, H.: Sentence simplification capabilities of transfer-based models. AAAI 36(11), 12172-12180 (2022)
Štajner, S.: Automatic text simplification for social good: Progress and challenges. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 2637-2652. Association for Computational Linguistics, Online (Aug 2021)
Štajner, S., Béchara, H., Saggion, H.: A deeper exploration of the standard PB-SMT approach to text simplification and its evaluation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pp. 823-828. Association for Computational Linguistics, Beijing, China (Jul 2015)
Stajner, S., Ferres, D., Shardlow, M., North, K., Zampieri, M., Saggion, H.: Lexical simplification benchmarks for english, portuguese, and spanish (2022)
Štajner, S., Popović, M.: Automated text simplification as a preprocessing step for machine translation into an under-resourced language. In: Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019). pp. 1141-1150. INCOMA Ltd., Varna, Bulgaria (Sep 2019)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. p. 3104-3112. NIPS’14, MIT Press, Cambridge, MA, USA (2014)
Tarouti, F.A., Kalita, J.K., McGrory, C.: Sentence simplification for question generation. In: Proceedings of the Conference of the North American Chapter of the ACL: Human Language Technologies. pp. 609-617 (2010)
Vu, T., Hu, B., Munkhdalai, T., Yu, H.: Sentence simplification with memory-augmented neural networks. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pp. 79-85. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018)
Xu,W., Callison-Burch, C., Napoles, C.: Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics 3, 283-297 (2015)
Xu, W., Napoles, C., Pavlick, E., Chen, Q., Callison-Burch, C.: Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics 4, 401-415 (2016)
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., Raffel, C.: mT5: A massively multilingual pre-trained text-to-text transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 483-498. Association for Computational Linguistics, Online (Jun 2021)
Zhang, X., Lapata, M.: Sentence simplification with deep reinforcement learning. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 584- 594. Association for Computational Linguistics, Copenhagen, Denmark (Sep 2017)
Zhang, Y., Ye, Z., Feng, Y., Zhao, D., Yan, R.: A constrained sequence-to-sequence neural model for sentence simplification. ArXiv abs/1704.02312 (2017)
Zhao, H., Cao, J., Xu, M., Lu, J.: Variational neural decoder for abstractive text summarization. In: Computer Science and Information Systems, Vol. 17, No. 2, 537-552. (2020)
Zhao, S., Meng, R., He, D., Saptono, A., Parmanto, B.: Integrating transformer and paraphrase rules for sentence simplification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3164-3173. Association for Computational Linguistics, Brussels, Belgium (Oct-Nov 2018)
Zilio, L., Braga Paraguassu, L., Leiva Hercules, L.A., Ponomarenko, G., Berwanger, L., Bocorny Finatto, M.J.: A lexical simplification tool for promoting health literacy. In: Proceedings of the 1st Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI). pp. 70-76. European Language Resources Association, Marseille, France (May 2020)
Zou, J., Aliannejadi, M., Kanoulas, E., Pera, M.S., Liu, Y.: Users meet clarifying questions: Toward a better understanding of user interactions for search clarification. ACM Transactions on Information Systems 41, 1 - 25 (2022)