Improving Machine Translation and Summarization with the Sinkhorn Divergence
Pages 149 - 161
Abstract
Important natural language processing tasks such as machine translation and document summarization have made enormous strides in recent years. However, their performance is still partially limited by the standard training objectives, which operate on single tokens rather than on more global features. Moreover, such standard objectives do not explicitly consider the source documents, potentially affecting their alignment with the predictions. For these reasons, in this paper, we propose using an Optimal Transport (OT) training objective to promote a global alignment between the model’s predictions and the source documents. In addition, we present an original implementation of the OT objective based on the Sinkhorn divergence between the final hidden states of the model’s encoder and decoder. Experimental results over machine translation and abstractive summarization tasks show that the proposed approach has been able to achieve statistically significant improvements across all experimental settings compared to our baseline and other alternative objectives. A qualitative analysis of the results also shows that the predictions have been able to better align with the source sentences thanks to the supervision of the proposed objective.
References
[1]
Alqahtani, S., Lalwani, G., Zhang, Y., Romeo, S., Mansour, S.: Using optimal transport as alignment objective for fine-tuning multilingual contextualized embeddings. In: EMNLP (2021)
[2]
Chen, L., et al.: Improving sequence-to-sequence learning via optimal transport. In: ICLR, pp. 1–16 (2019)
[3]
Chen, Y.C., Gan, Z., Cheng, Y., Liu, J., Liu, J.: Distilling knowledge learned in bert for text generation. In: ACL (2020)
[4]
Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. In: NIPS, pp. 2292–2300 (2013)
[5]
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019)
[6]
Feydy, J., Séjourné, T., Vialard, F.X., Amari, S., Trouvé, A., Peyré, G.: Interpolating between optimal transport and mmd using sinkhorn divergences. In: AISTATS (2019)
[7]
Garg, S., Peitz, S., Nallasamy, U., Paulik, M.: Jointly learning to align and translate with transformer models. In: EMNLP (2019)
[8]
Jauregi Unanue, I., Parnell, J., Piccardi, M.: Berttune: fine-tuning neural machine translation with bertscore. In: ACL/IJCNLP (2021)
[9]
Koehn, P.: Statistical significance tests for machine translation evaluation. In: EMNLP (2004)
[10]
Kusner, M.J., Sun, Y., Kolkin, N.I., Weinberger, K.Q.: From word embeddings to document distances. In: ICML (2015)
[11]
Lewis, M., et al.: Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv abs/1910.13461 (2020)
[12]
Li, C., et al.: Improving text generation with student-forcing optimal transport. In: EMNLP, pp. 9144–9156 (2020)
[13]
Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: ACL 2004 (2004)
[14]
Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: EMNLP (2015)
[15]
Nguyen, T., Luu, A.T.: Improving neural cross-lingual summarization via employing optimal transport distance for knowledge distillation. arXiv abs/2112.03473 (2021)
[16]
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL (2002)
[17]
Parnell, J., Unanue, I.J., Piccardi, M.: A multi-document coverage reward for relaxed multi-document summarization. In: ACL (2022)
[18]
Post, M.: A call for clarity in reporting bleu scores. In: WMT (2018)
[19]
Ranzato, M., Chopra, S., Auli, M., Zaremba, W.: Sequence level training with recurrent neural networks. CoRR abs/1511.06732 (2016)
[20]
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. arXiv abs/1508.07909 (2016)
[21]
Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. In: NAACL (2018)
[22]
Shi X, Huang H, Jian P, and Tang Y-K Lauw HW, Wong RC-W, Ntoulas A, Lim E-P, Ng S-K, and Pan SJ Case-Sensitive Neural Machine Translation Advances in Knowledge Discovery and Data Mining 2020 Cham Springer 662-674
[23]
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS (2014)
[24]
Vaswani, A., et al.: Attention is all you need. arXiv abs/1706.03762 (2017)
[25]
Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTscore: evaluating text generation with BERT. arXiv abs/1904.09675 (2020)
[26]
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C.M., Eger, S.: Moverscore: text generation evaluating with contextualized embeddings and earth mover distance. arXiv abs/1909.02622 (2019)
[27]
Zhou, Q., Yang, N., Wei, F., Zhou, M.: Selective encoding for abstractive sentence summarization. In: ACL (2017)
[28]
Zhou, Q., Yang, N., Wei, F., Zhou, M.: Sequential copying networks. In: AAAI (2018)
Recommendations
Evaluation of arabic machine translation system based on the universal networking language
NLDB'09: Proceedings of the 14th international conference on Applications of Natural Language to Information SystemsThis paper evaluates a machine translation (MT) system based on the interlingua approach, the Universal Network Language (UNL) system, designed for Multilanguage translation. The study addresses evaluation of English-Arabic translation and aims at ...
Example-Based English to Arabic Machine Translation: Matching Stage Using Internal Medicine Publications
ICSIE '18: Proceedings of the 7th International Conference on Software and Information EngineeringAutomatic machine translation becomes an important source of translation nowadays. It is a software system that translates a text from one natural language to one (many) natural language. On the web, there are many machine translation systems that give ...
Generating Arabic text in multilingual speech-to-speech machine translation framework
The interlingual approach to machine translation (MT) is used successfully in multilingual translation. It aims to achieve the translation task in two independent steps. First, meanings of the source-language sentences are represented in an intermediate ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2023
359 pages
ISBN:978-3-031-33382-8
DOI:10.1007/978-3-031-33383-5
- Editors:
- Hisashi Kashima,
- Tsuyoshi Ide,
- Wen-Chih Peng
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 26 May 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
View Options
View options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in