research-article

Multi-task Stack Propagation for Neural Quality Estimation

Authors:

Jong-Hyeok Lee,

Seung-Hoon NaAuthors Info & Claims

ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), Volume 18, Issue 4

Article No.: 48, Pages 1 - 18

https://doi.org/10.1145/3321127

Published: 21 May 2019 Publication History

Abstract

Quality estimation is an important task in machine translation that has attracted increased interest in recent years. A key problem in translation-quality estimation is the lack of a sufficient amount of the quality annotated training data. To address this shortcoming, the Predictor-Estimator was proposed recently by introducing “word prediction” as an additional pre-subtask that predicts a current target word with consideration of surrounding source and target contexts, resulting in a two-stage neural model composed of a predictor and an estimator. However, the original Predictor-Estimator is not trained on a continuous stacking model but instead in a cascaded manner that separately trains the predictor from the estimator. In addition, the Predictor-Estimator is trained based on single-task learning only, which uses target-specific quality-estimation data without using other training data that are available from other-level quality-estimation tasks. In this article, we thus propose a multi-task stack propagation, which extensively applies stack propagation to fully train the Predictor-Estimator on a continuous stacking architecture and multi-task learning to enhance the training data from related other-level quality-estimation tasks. Experimental results on WMT17 quality-estimation datasets show that the Predictor-Estimator trained with multi-task stack propagation provides statistically significant improvements over the baseline models. In particular, under an ensemble setting, the proposed multi-task stack propagation leads to state-of-the-art performance at all the sentence/word/phrase levels for WMT17 quality estimation tasks.

References

[1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the ICLR 2015.

[2]

Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the 2nd Conference on Machine Translation, Volume 2: Shared Task Papers. Association for Computational Linguistics, 169--214. Retrieved from http://www.aclweb.org/anthology/W17-4717.

[3]

Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 131--198. Retrieved from http://www.aclweb.org/anthology/W/W16/W16-2301.

[4]

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’14). Association for Computational Linguistics, 1724--1734. Retrieved from http://www.aclweb.org/anthology/D14-1179.

[5]

Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12 (Nov. 2011), 2493--2537. http://dl.acm.org/citation.cfm?id=1953048.2078186.

Digital Library

[6]

Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 1723--1732.

[7]

Mariano Felice and Lucia Specia. 2012. Linguistic features for quality estimation. In Proceedings of the 7th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 96--103. Retrieved from http://www.aclweb.org/anthology/W12-3110.

Digital Library

[8]

Jesús González-Rubio, J. Ramón Navarro-Cerdán, and Francisco Casacuberta. 2013. Dimensionality reduction methods for machine translation quality estimation. Mach. Trans. 27, 3 (2013), 281--301.

Digital Library

[9]

Jesús González-Rubio, Alberto Sanchís, and Francisco Casacuberta. 2012. PRHLT submission to the WMT12 quality estimation task. In Proceedings of the 7th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 104--108. Retrieved from http://www.aclweb.org/anthology/W12-3111.

Digital Library

[10]

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1923--1933. Retrieved from http://aclweb.org/anthology/D17-1206.

[11]

Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing. In Proceedings of the 1st Conference on Machine Translation (WMT’16). 751--758.

[12]

Lukasz Kaiser, Aidan N. Gomez, and François Chollet. 2017. Depthwise separable convolutions for neural machine translation. CoRR abs. Retrieved from http://arxiv.org/abs/1706.03059.

[13]

Hyun Kim, Hun-Young Jung, Hongseok Kwon, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator: Neural quality estimation based on target word prediction for machine translation. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 17, 1, Article 3 (Sept. 2017), 22 pages.

Digital Library

[14]

Hyun Kim and Jong-Hyeok Lee. 2016. Recurrent neural network based translation quality estimation. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 787--792. Retrieved from http://www.aclweb.org/anthology/W/W16/W16-2384.

[15]

Hyun Kim and Jong-Hyeok Lee. 2016. A recurrent neural networks approach for estimating the quality of machine translation output. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 494--498. Retrieved from http://www.aclweb.org/anthology/N16-1059.

[16]

Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the 2nd Conference on Machine Translation, Volume 2: Shared Task Papers. Association for Computational Linguistics, 562--568. Retrieved from http://www.aclweb.org/anthology/W17-4763.

[17]

Anna Kozlova, Mariya Shmatova, and Anton Frolov. 2016. YSDA participation in the WMT’16 quality estimation shared task. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 793--799. Retrieved from http://www.aclweb.org/anthology/W/W16/W16-2385.

[18]

Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality Estimation from scraTCH (QUETCH): Deep learning for word-level translation quality estimation. In Proceedings of the 10th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 316--322. Retrieved from http://aclweb.org/anthology/W15-3037.

[19]

Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In Proceedings of the International Conference on Learning Representations.

[20]

André F. T. Martins, Ramón Astudillo, Chris Hokamp, and Fabio Kepler. 2016. Unbabel’s participation in the WMT16 word-level translation quality estimation shared task. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 806--811. Retrieved from http://www.aclweb.org/anthology/W/W16/W16-2387.

[21]

André F. T. Martins, Junczys-Dowmunt Marcin, Fabio Kepler, and Ramón Astudillo. 2017. Pushing the limits of translation quality estimation. Trans. Assoc. Comput. Ling. 5 (2017) 205--218.

[22]

Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Meeting on Association for Computational Linguistics (ACL’03). Volume 1. 160--167.

Digital Library

[23]

Raj Nath Patel and Sasikumar M. 2016. Translation quality estimation using recurrent neural network. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, 819--824. Retrieved from http://www.aclweb.org/anthology/W/W16/W16-2389.

[24]

Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In Proceedings of the 55th Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2037--2048.

[25]

Raphael Rubino, José de Souza, Jennifer Foster, and Lucia Specia. 2013. Topic models for translation quality estimation for gisting purposes. In Proceedings of the 14th Machine Translation Summit. 295--302.

[26]

Kashif Shah, Trevor Cohn, and Lucia Specia. 2015. A Bayesian non-linear method for feature selection in machine translation quality estimation. Mach. Trans. 29, 2 (2015), 101--125.

Digital Library

[27]

Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Translation in the Americas. 223--231.

[28]

Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 231--235.

[29]

Radu Soricut and Abdessamad Echihabi. 2010. TrustRank: Inducing trust in automatic translations via ranking. In Proceedings of the 48th Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 612--621. Retrieved from http://www.aclweb.org/anthology/P10-1063.

Digital Library

[30]

Lucia Specia and Varvara Logacheva. 2017. WMT17 Quality estimation shared task training and development data. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. Retrieved from http://hdl.handle.net/11372/LRT-1974.

[31]

Lucia Specia, Kashif Shah, José G. C. de Souza, and Trevor Cohn. 2013. QuEst—A translation quality estimation framework. In Proceedings of the 51st Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics, 79--84. Retrieved from http://www.aclweb.org/anthology/P13-4014.

[32]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 5998--6008. Retrieved from http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf.

Digital Library

[33]

Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. In Proceedings of the 54th Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1557--1566. Retrieved from http://www.aclweb.org/anthology/P16-1147.

Cited By

Gu XXia KJiang YJolfaei A(2021)Multi-task Fuzzy Clustering–Based Multi-task TSK Fuzzy System for Text Sentiment ClassificationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/347610321:2(1-24)Online publication date: 18-Nov-2021
https://dl.acm.org/doi/10.1145/3476103
Zhou ZLi FYang S(2021)A Novel Resource Optimization Algorithm Based on Clustering and Improved Differential Evolution Strategy Under a Cloud EnvironmentACM Transactions on Asian and Low-Resource Language Information Processing10.1145/346276120:5(1-15)Online publication date: 30-Jun-2021
https://dl.acm.org/doi/10.1145/3462761
Kim HNa S(2020)Uniformly Interpolated Balancing for Robust Prediction in Translation Quality EstimationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/336591619:3(1-27)Online publication date: 19-Jan-2020
https://dl.acm.org/doi/10.1145/3365916

Index Terms

Multi-task Stack Propagation for Neural Quality Estimation
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Machine translation
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

MultiMatch: Multi-task Learning for Semi-supervised Domain Generalization
Domain generalization (DG) aims at learning a model on source domains to well generalize on the unseen target domain. Although it has achieved great success, most of the existing methods require the label information for all training samples in source ...
Multi-task learning neural framework for categorizing sexism
Abstract
Sexism, a form of oppression based on one’s sex, manifests itself in numerous ways and causes enormous suffering. In view of the growing number of experiences of sexism reported online, automatically classifying these recollections can ...
Highlights
- Propose a knowledge-based cascaded multi-task neural framework for the multi-label sexism classification.
Multi-atlas Segmentation Combining Multi-task Local Label Learning and Semi-supervised Label Propagation
Image and Graphics
Abstract
Multi-atlas based segmentation methods have achieved great success in hippocampal segmentation due to their promising performance. However, the correlation between voxels in the target image is often ignored. In this study, an image segmentation ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian and Low-Resource Language Information Processing

ACM Transactions on Asian and Low-Resource Language Information Processing Volume 18, Issue 4

December 2019

305 pages

ISSN:2375-4699

EISSN:2375-4702

DOI:10.1145/3327969

Editor:
Nianwen Xue
Brandeis University, Waltham, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 May 2019

Accepted: 01 March 2019

Revised: 01 November 2018

Received: 01 June 2018

Published in TALLIP Volume 18, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Development of Knowledge Evolutionary WiseQA Platform Technology for Human Knowledge Augmented Services
Korea government (MSIT)
Institute for Information 8 Communications Technology Planning 8 Evaluation (IITP)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
242
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gu XXia KJiang YJolfaei A(2021)Multi-task Fuzzy Clustering–Based Multi-task TSK Fuzzy System for Text Sentiment ClassificationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/347610321:2(1-24)Online publication date: 18-Nov-2021
https://dl.acm.org/doi/10.1145/3476103
Zhou ZLi FYang S(2021)A Novel Resource Optimization Algorithm Based on Clustering and Improved Differential Evolution Strategy Under a Cloud EnvironmentACM Transactions on Asian and Low-Resource Language Information Processing10.1145/346276120:5(1-15)Online publication date: 30-Jun-2021
https://dl.acm.org/doi/10.1145/3462761
Kim HNa S(2020)Uniformly Interpolated Balancing for Robust Prediction in Translation Quality EstimationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/336591619:3(1-27)Online publication date: 19-Jan-2020
https://dl.acm.org/doi/10.1145/3365916

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents