Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Leveraging Adversarial Training to Facilitate Grammatical Error Correction

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2021 (ICANN 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12891))

Included in the following conference series:

  • 3330 Accesses

Abstract

Grammatical error correction (GEC) task aims to detect and correct grammatical errors in sentences. Recently, the pre-trained language model has provided a strong baseline for GEC and achieved excellent results by fine-tuning on a small amount of annotated data. However, due to the lack of large-scale erroneous-corrected parallel datasets, these models tend to suffer from the problem of overfitting. Previous researchers have proposed a variety of data augmentation methods to generate more training data and enlarge the dataset, but these methods either rely on rules to generate grammatical errors and are not automated, or produce errors that do not match human writing errors. The pre-trained model only improves significantly after task-specific data fine-tuning; otherwise, the highly noisy data can impair the performance of the pre-trained model. To address this issue, we propose a method to enhance the robustness of the model based on adversarial training. This approach constructs the adversarial samples and treats them as the augmented data. Unlike previous methods that introduce token-level noise, our method introduces embedding-level noise and can obtain extra samples that are close to human writing errors. Besides, we employ the adversarial consistency constraint to reduce the gap between the adversarial sample and the original sample. The experimental results demonstrate that our method can further boost the performance of the pre-trained model on GEC task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/hunspell/hunspell.

  2. 2.

    https://github.com/nusnlp/m2scorer.

  3. 3.

    https://github.com/chrisjbryant/errant.

  4. 4.

    https://github.com/pytorch/fairseq.

References

  1. Awasthi, A., Sarawagi, S., Goyal, R., Ghosh, S., Piratla, V.: Parallel iterative edit models for local sequence transduction. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1435

  2. Bryant, C., Felice, M., Andersen, Ø.E., Briscoe, T.: The BEA-2019 shared task on grammatical error correction, pp. 52–75. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/w19-4406

  3. Bryant, C., Felice, M., Briscoe, T.: Automatic annotation and evaluation of error types for grammatical error correction. In: Barzilay, R., Kan, M. (eds.) ACL 2017, pp. 793–805. Association for Computational Linguistics (2017)

    Google Scholar 

  4. Chollampatt, S., Ng, H.T.: A Multilayer Convolutional Encoder-decoder Neural Network for Grammatical Error Correction, pp. 5755–5762. AAAI Press (2018). https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17308

  5. Chollampatt, S., Taghipour, K., Ng, H.T.: Neural Network Translation Models for Grammatical Error Correction, pp. 2768–2774. IJCAI/AAAI Press (2016). http://www.ijcai.org/Abstract/16/393

  6. Dahlmeier, D., Ng, H.T.: Better evaluation for grammatical error correction. The Association for Computational Linguistics (2012). https://www.aclweb.org/anthology/N12-1067/

  7. Dahlmeier, D., Ng, H.T., Wu, S.M.: Building a large annotated corpus of learner English: the NUS corpus of learner English, pp. 22–31. The Association for Computer Linguistics (2013). https://www.aclweb.org/anthology/W13-1703/

  8. De Felice, R., Pulman, S.: A classifier-based approach to preposition and determiner error correction in L2 English. In: Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008), pp. 169–176 (2008)

    Google Scholar 

  9. Felice, M., Bryant, C., Briscoe, T.: Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In: Calzolari, N., Matsumoto, Y., Prasad, R. (eds.) COLING 2016. pp. 825–835. ACL (2016)

    Google Scholar 

  10. Ge, T., Wei, F., Zhou, M.: Fluency boost learning and inference for neural grammatical error correction, pp. 1055–1065. Association for Computational Linguistics (2018). https://www.aclweb.org/anthology/P18-1097/

  11. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples (2015). http://arxiv.org/abs/1412.6572

  12. Grundkiewicz, R., Junczys-Dowmunt, M., Heafield, K.: Neural grammatical error correction systems with unsupervised pre-training on synthetic data, pp. 252–263. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/w19-4427

  13. Junczys-Dowmunt, M., Grundkiewicz, R.: Phrase-based machine translation is state-of-the-art for automatic grammatical error correction, pp. 1546–1556. The Association for Computational Linguistics (2016). https://doi.org/10.18653/v1/d16-1161

  14. Junczys-Dowmunt, M., Grundkiewicz, R., Guha, S., Heafield, K.: Approaching neural grammatical error correction as a low-resource machine translation task, pp. 595–606. Association for Computational Linguistics (2018). https://doi.org/10.18653/v1/n18-1055

  15. Kaneko, M., Mita, M., Kiyono, S., Suzuki, J., Inui, K.: Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction, pp. 4248–4254. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.acl-main.391

  16. Katsumata, S., Komachi, M.: Stronger baselines for grammatical error correction using a pretrained encoder-decoder model, pp. 827–832. Association for Computational Linguistics (2020). https://www.aclweb.org/anthology/2020.aacl-main.83/

  17. Kiyono, S., Suzuki, J., Mita, M., Mizumoto, T., Inui, K.: An empirical study of incorporating pseudo data into grammatical error correction, pp. 1236–1242. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1119

  18. Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, pp. 7871–7880 (2020). https://doi.org/10.18653/v1/2020.acl-main.703

  19. Lichtarge, J., Alberti, C., Kumar, S.: Data weighted training strategies for grammatical error correction. Trans. Assoc. Comput. Ling. 8, 634–646 (2020). https://transacl.org/ojs/index.php/tacl/article/view/2047

  20. Liu, X., et al.: Adversarial training for large neural language models. CoRR abs/2004.08994 (2020). https://arxiv.org/abs/2004.08994

  21. Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam. CoRR abs/1711.05101 (2017). http://arxiv.org/abs/1711.05101

  22. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. OpenReview.net (2018). https://openreview.net/forum?id=rJzIBfZAb

  23. Malmi, E., Krause, S., Rothe, S., Mirylenka, D., Severyn, A.: Encode, tag, realize: High-precision text editing, pp. 5053–5064. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1510

  24. Mita, M., Kiyono, S., Kaneko, M., Suzuki, J., Inui, K.: A self-refinement strategy for noise reduction in grammatical error correction, pp. 267–280. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.26

  25. Miyato, T., Dai, A.M., Goodfellow, I.J.: Adversarial training methods for semi-supervised text classification. OpenReview.net (2017). https://openreview.net/forum?id=r1X3g2_xl

  26. Naber, D., et al.: A rule-based style and grammar checker (2003)

    Google Scholar 

  27. Omelianchuk, K., Atrasevych, V., Chernodub, A.N., Skurzhanskyi, O.: Gector - grammatical error correction: tag, not rewrite, pp. 163–170. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.bea-1.16

  28. Sato, M., Suzuki, J., Kiyono, S.: Effective adversarial regularization for neural machine translation, pp. 204–210. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/p19-1020

  29. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL 2016. The Association for Computer Linguistics (2016)

    Google Scholar 

  30. Tajiri, T., Komachi, M., Matsumoto, Y.: Tense and aspect error correction for ESL learners using global context, pp. 198–202. The Association for Computer Linguistics (2012). https://www.aclweb.org/anthology/P12-2039/

  31. Wang, D., Gong, C., Liu, Q.: Improving neural language modeling via adversarial training, vol. 97, pp. 6555–6565. PMLR (2019). http://proceedings.mlr.press/v97/wang19f.html

  32. Wang, L., Zheng, X.: Improving grammatical error correction models with purpose-built adversarial examples. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) EMNLP 2020, pp. 2858–2869. Association for Computational Linguistics (2020)

    Google Scholar 

  33. Wang, Y., Wang, Y., Liu, J., Liu, Z.: A comprehensive survey of grammar error correction. arXiv preprint arXiv:2005.06600 (2020)

  34. Yannakoudakis, H., Andersen, Ø.E., Geranpayeh, A., Briscoe, T., Nicholls, D.: Developing an automated writing placement system for ESL learners. Appl. Measur. Educ. 31(3), 251–267 (2018)

    Article  Google Scholar 

  35. Yannakoudakis, H., Briscoe, T., Medlock, B.: A new dataset and method for automatically grading ESOL texts, pp. 180–189. The Association for Computer Linguistics (2011). https://www.aclweb.org/anthology/P11-1019/

  36. Yuan, Z., Briscoe, T.: Grammatical error correction using neural machine translation. The Association for Computational Linguistics (2016). https://doi.org/10.18653/v1/n16-1042

  37. Zhao, W., Wang, L., Shen, K., Jia, R., Liu, J.: Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data, pp. 156–165. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/n19-1014

  38. Zhao, Z., Wang, H.: MaskGEC: improving neural grammatical error correction via dynamic masking, pp. 1226–1233. AAAI Press (2020). https://aaai.org/ojs/index.php/AAAI/article/view/5476

  39. Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., Liu, J.: FreeLB: enhanced adversarial training for natural language understanding. OpenReview.net (2020). https://openreview.net/forum?id=BygzbyHFvB

Download references

Acknowledgments

This research is supported by the National Natural Science Foundation of China under the grant [No. 61976119] and the Natural Science Foundation of Tianjin under the grant [No. 18ZXZNGX00310].

Author information

Authors and Affiliations

  1. College of Artificial Intelligence, Nankai University, Tianjin, China

    Kai Dang, Jiaying Xie & Jie Liu

Authors
  1. Kai Dang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiaying Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Liu .

Editor information

Editors and Affiliations

  1. Comenius University in Bratislava, Bratislava, Slovakia

    Igor Farkaš

  2. iMotions A/S, Copenhagen, Denmark

    Paolo Masulli

  3. University of Tübingen, Tübingen, Baden-Württemberg, Germany

    Sebastian Otte

  4. Universität Hamburg, Hamburg, Germany

    Stefan Wermter

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dang, K., Xie, J., Liu, J. (2021). Leveraging Adversarial Training to Facilitate Grammatical Error Correction. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2021. ICANN 2021. Lecture Notes in Computer Science(), vol 12891. Springer, Cham. https://doi.org/10.1007/978-3-030-86362-3_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86362-3_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86361-6

  • Online ISBN: 978-3-030-86362-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation