Abstract
The Winograd Schema Challenge (WSC) is a popular benchmark for commonsense reasoning. Each WSC instance has a component that corresponds to the mention of the correct answer option of the two options in the context. We observe that the answers of many instances are insensitive to the options. In this paper, based on this observation, we propose an approach based on fine-tuning the pre-trained language model for WSC by distinguishing sensitive and insensitive options. First, we split WSC instances into option-sensitive and insensitive categories, and use option expanding and option masking strategies to weaken the options so that the model does not pay attention to options when they are insensitive during fine-tuning. Second, we treat the two categories as intermediate-task of each other, and use transfer learning to improve the performance. We fine-tune BERT-Large and T5-XXL with our approach on WINOGRANDE, a new dataset of WSC, and the experiment shows our method outperforms baselines by a large margin, achieving state-of-the-art, which indicates the effectiveness of our instance-distinguishing strategy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: Flair: an easy-to-use framework for state-of-the-art NLP. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 54–59 (2019)
Chang, T.-Y., Chi-Jen, L.: Rethinking why intermediate-task fine-tuning works. In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 706–713 (2021)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Emami, A., De La Cruz, N., Trischler, A., Suleman, K., Cheung, J.C.K.: A knowledge hunting framework for common sense reasoning. In: EMNLP (2018)
Khashabi, D., et al.: UnifiedQA: crossing format boundaries with a single QA system. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896–1907 (2020)
Levesque, H., Davis, E., Morgenstern, L.: The Winograd schema challenge. In: Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning (2012)
Liu, Q., Jiang, H., Ling, Z.-H., Zhu, X., Wei, S., Hu, Y.: Combing context and commonsense knowledge through neural networks for solving Winograd schema problems. In: 2017 AAAI Spring Symposium Series (2017)
Lourie, N., Le Bras, R., Bhagavatula, C., Choi, Y.: Unicorn on rainbow: a universal commonsense reasoning model on a new multitask benchmark. In: Proceedings of the AAAI Conference on Artificial Intelligence, 35, pp. 13480–13488 (2021)
Opitz, J., Frank, A.: Addressing the Winograd schema challenge as a sequence ranking task. In: Proceedings of the First International Workshop on Language Cognition and Computational Models, pp. 41–52 (2018)
Peng, H., Khashabi, D., Roth, D.: Solving hard coreference problems. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 809–819 (2015)
Pruksachatkun, Y., et al.: Intermediate-task transfer learning with pretrained language models: when and why does it work? In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5231–5247 (2020)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019)
Roberts, A., Raffel, C., Shazeer, N.: How much knowledge can you pack into the parameters of a language model? In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418–5426 (2020)
Sakaguchi, K., Le Bras, R., Bhagavatula, C., Choi, Y.: Winogrande: an adversarial Winograd schema challenge at scale. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8732–8740 (2020)
Trinh, T.H., Le, Q.V.: A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847 (2018)
Zhang, H., Ding, H., Song, Y.: Sp-10k: a large-scale evaluation set for selectional preference acquisition. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 722–731 (2019)
Acknowledgements
We would like to thank the anonymous reviewers for their helpful comments. This work was supported by the National Key Research and Development Project of China (No. 2021ZD0110700).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, D. et al. (2023). Distinguishing Sensitive and Insensitive Options for the Winograd Schema Challenge. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13945. Springer, Cham. https://doi.org/10.1007/978-3-031-30675-4_52
Download citation
DOI: https://doi.org/10.1007/978-3-031-30675-4_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30674-7
Online ISBN: 978-3-031-30675-4
eBook Packages: Computer ScienceComputer Science (R0)