research-article

Gated Character-aware Convolutional Neural Network for Effective Automated Essay Scoring

Authors:

Siu Cheung HuiAuthors Info & Claims

WI-IAT '21: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology

Pages 351 - 359

https://doi.org/10.1145/3486622.3493945

Published: 13 April 2022 Publication History

Abstract

Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Many current state-of-the-art approaches are based on deep learning models. However, most AES models overlook the importance of character-level information, which is important to both the performance and the fairness. The character-level information is able to provide orthographic knowledge (e.g., spelling) and help the learning of infrequent and ⟨UNK⟩ tokens. In this paper, we propose a Gated Character-aware Convolutional Neural Network (GCCNN) model for the AES task. The proposed GCCNN model incorporates character-level information by a character-level encoder and a gated fusion mechanism. First, the character-level encoder learns word embeddings from sequences of characters by a hierarchical convolutional neural network. Next, the gated fusion mechanism adaptively controls the amount of word-level and character-level information to be fused using vector gating. Then, the essay-level encoder learns an essay representation based on the fused word embeddings. Finally, the fully connected layer maps the essay representation into its corresponding score. The experimental results show that our GCCNN model outperforms the baseline deep learning models. In addition, our qualitative analysis also demonstrates the importance of character-level information for tackling the out-of-vocabulary problem in grading essays.

References

[1]

Yue Cao, Hanqi Jin, Xiaojun Wan, and Zhiwei Yu. 2020. Domain-Adaptive Neural Automated Essay Scoring. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 1011–1020. https://doi.org/10.1145/3397271.3401037

Digital Library

[2]

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning (Corvalis, Oregon, USA) (ICML ’07). Association for Computing Machinery, New York, NY, USA, 129–136. https://doi.org/10.1145/1273496.1273513

Digital Library

[3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).

[4]

Fei Dong and Yue Zhang. 2016. Automatic Features for Essay Scoring – An Empirical Study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 1072–1077. https://doi.org/10.18653/v1/D16-1115

[5]

Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). Association for Computational Linguistics, Vancouver, Canada, 153–162. https://doi.org/10.18653/v1/K17-1017

[6]

Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Barcelona, Spain (Online), 6903–6915. https://doi.org/10.18653/v1/2020.coling-main.609

[7]

Yukun Feng, Hidetaka Kamigaito, Hiroya Takamura, and Manabu Okumura. 2019. A Simple and Effective Method for Injecting Word-Level Information into Character-Aware Neural Language Models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). Association for Computational Linguistics, Hong Kong, China, 920–928. https://doi.org/10.18653/v1/K19-1086

[8]

Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction. Transactions of the Association for Computational Linguistics 6 (2018), 451–465. https://doi.org/10.1162/tacl_a_00032

[9]

Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 328–339. https://doi.org/10.18653/v1/P18-1031

[10]

Cancan Jin, Ben He, Kai Hui, and Le Sun. 2018. TDNN: A Two-stage Deep Neural Network for Prompt-independent Automated Essay Scoring. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 1088–1097. https://doi.org/10.18653/v1/P18-1100

[11]

Rie Johnson and Tong Zhang. 2017. Deep Pyramid Convolutional Neural Networks for Text Categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 562–570. https://doi.org/10.18653/v1/P17-1052

[12]

Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A Convolutional Neural Network for Modelling Sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Baltimore, Maryland, 655–665. https://doi.org/10.3115/v1/P14-1062

[13]

Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-Aware Neural Language Models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI’16). AAAI Press, 2741–2749.

Digital Library

[14]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014).

[15]

Jiawei Liu, Yang Xu, and Lingzhe Zhao. 2019. Automated Essay Scoring based on Two-Stage Learning. CoRR abs/1901.07744(2019). arxiv:1901.07744http://arxiv.org/abs/1901.07744

[16]

Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. CharBERT: Character-aware Pre-trained Language Model. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Barcelona, Spain (Online), 39–50. https://doi.org/10.18653/v1/2020.coling-main.4

[17]

Elijah Mayfield and Alan W Black. 2020. Should You Fine-Tune BERT for Automated Essay Scoring?. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, 151–162. https://doi.org/10.18653/v1/2020.bea-1.15

[18]

Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated Word-Character Recurrent Language Model. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 1992–1997. https://doi.org/10.18653/v1/D16-1209

[19]

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1532–1543. https://doi.org/10.3115/v1/D14-1162

[20]

Pedro Uria Rodriguez, Amir Jafari, and Christopher M. Ormerod. 2019. Language models and Automated Essay Scoring. CoRR abs/1909.09482(2019). arxiv:1909.09482http://arxiv.org/abs/1909.09482

[21]

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 1715–1725. https://doi.org/10.18653/v1/P16-1162

[22]

Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip S. Yu, and Caiming Xiong. 2020. Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT. CoRR abs/2003.04985(2020). arXiv:2003.04985https://arxiv.org/abs/2003.04985

[23]

Kaveh Taghipour and Hwee Tou Ng. 2016. A Neural Approach to Automated Essay Scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 1882–1891. https://doi.org/10.18653/v1/D16-1193

[24]

Yi Tay, Minh Phan, Luu Anh Tuan, and Siu Cheung Hui. 2018. Skipflow: Incorporating neural coherence features for end-to-end automatic text scoring. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.

[25]

Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 1560–1569. https://doi.org/10.18653/v1/2020.findings-emnlp.141

[26]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. CoRR abs/1906.08237(2019). arxiv:1906.08237http://arxiv.org/abs/1906.08237

Cited By

Cho MHuang JKwon O(2024)Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay gradingETRI Journal10.4218/etrij.2023-032446:1(82-95)Online publication date: 28-Feb-2024
https://doi.org/10.4218/etrij.2023-0324
Bai HHui S(2024)A crowdsourcing-based incremental learning framework for automated essays scoringExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121755238:PBOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121755
Atil BSheikhi Karizaki MJ. Passonneau R(2024)VerAs: Verify Then Assess STEM Lab ReportsArtificial Intelligence in Education10.1007/978-3-031-64302-6_10(133-148)Online publication date: 2-Jul-2024
https://doi.org/10.1007/978-3-031-64302-6_10

Recommendations

MFDNN: Mixed Features Deep Neural Network Model for Prompt-independent Automated Essay Scoring
ACAI '21: Proceedings of the 2021 4th International Conference on Algorithms, Computing and Artificial Intelligence

Most of the existing Automatic Essay Scoring (AES) models are prompt-dependent models that need the rated essays of specific prompt for training. However, there are few studies on prompt-independent AES. This paper studies how to fully use the effective ...
Machine Learning-based Automated Essay Scoring System for Chinese Proficiency Test (HSK)
NLPIR '20: Proceedings of the 4th International Conference on Natural Language Processing and Information Retrieval

Automated essay scoring (AES) gains momentum recently in English-based environment. However, the development of Chinese AES system is slow and fruitless. Many foreign students participate in the Chinese Proficiency Test (HSK) so a HSK automated essay ...
The Application of Deep Learning in Automated Essay Evaluation
Emerging Technologies for Education
Abstract
The shift from Automated Essay Scoring (AES) to Automated Essay Evaluation (AEE) indicates that natural language processing (NLP) researchers respond positively to the request from language teaching field. Writers and teachers need more feedback ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WI-IAT '21: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology

December 2021

698 pages

ISBN:9781450391153

DOI:10.1145/3486622

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WI-IAT '21

Sponsor:

SIGAI

WI-IAT '21: IEEE/WIC/ACM International Conference on Web Intelligence

December 14 - 17, 2021

VIC, Melbourne, Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
98
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cho MHuang JKwon O(2024)Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay gradingETRI Journal10.4218/etrij.2023-032446:1(82-95)Online publication date: 28-Feb-2024
https://doi.org/10.4218/etrij.2023-0324
Bai HHui S(2024)A crowdsourcing-based incremental learning framework for automated essays scoringExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121755238:PBOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121755
Atil BSheikhi Karizaki MJ. Passonneau R(2024)VerAs: Verify Then Assess STEM Lab ReportsArtificial Intelligence in Education10.1007/978-3-031-64302-6_10(133-148)Online publication date: 2-Jul-2024
https://doi.org/10.1007/978-3-031-64302-6_10

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents