Elamur Research⌕
☰
home / papers / 2017 / Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring

Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring

Summary

The paper addresses automatic essay scoring (AES), where a system must grade student essays in a way that agrees with human raters. Earlier neural AES models encoded an essay as one vector using either an LSTM or a CNN. They never compared these two architectures under identical conditions, and they weighted all parts of the text equally. The authors build a hierarchical sentence-document model. A CNN with attention pooling turns each sentence's words into a sentence vector. An LSTM with a second attention-pooling layer then combines the sentence vectors into an essay representation, and a sigmoid layer produces the score. Word inputs can be either word embeddings or character-level CNN embeddings. Training minimizes mean squared error. Experiments on the ASAP essay data, evaluated with quadratic weighted kappa (QWK), show two things. CNNs work better at the sentence level, while LSTMs work better at the document level, which is plausibly a matter of local n-gram cues versus global coherence. Attention also brings significant gains. The full model reports the best published result on the ASAP test set. This matters because it gives feature-free, partly interpretable essay grading.

Key points

  • Data and evaluation: experiments use the ASAP (Automated Student Assessment Prize, Kaggle) essay data. The metric is quadratic weighted kappa, following the official competition criteria, and significance is checked with a one-tailed t-test.
  • Architecture: the model builds on the two-level design of Dong and Zhang (2016). It has a sentence-level CNN with a window size of 5, so attention operates over 5-gram features, followed by a document-level LSTM. Attention pooling is applied at both the word/n-gram level and the sentence level.
  • Architecture comparison: under matched settings, CNNs were found more suitable for modelling sentences and LSTMs for modelling whole documents.
  • Interpretability: examples from Prompt 4 and Prompt 8 show how attention weights spread across sentences and n-grams. They illustrate which parts of an essay the model emphasizes.
  • Code release: the code is publicly available at github.com/feidong1991/aes. Suggested future work is cross-domain AES.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k17-1017

@inproceedings{dong2017attention,
  title = {{Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring}},
  author = {Dong, Fei and Zhang, Yue and Yang, Jie},
  booktitle = {Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)},
  year = {2017},
  doi = {10.18653/v1/k17-1017},
  url = {https://doi.org/10.18653/v1/k17-1017}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jiangming Liu and Yue Zhang. 2017. Attention modeling for targeted sentiment. find
  2. Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. arXiv preprint arXiv:1606.04289 . find
  3. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 . find
  4. Yigal Attali and Jill Burstein. 2004. Automated essay scoring with e-rater R v. 2.0. ETS Research Report Series 2004(2):i–21. find
  5. Hongbo Chen and Ben He. 2013. Automated essay scoring by maximizing human-machine agreement. In EMNLP. pages 1741–1752. find
  6. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Interspeech. volume 2, page 3. find
  7. Yen-Yu Chen, Chien-Liang Liu, Chia-Hoang Lee, Tao-Hsing Chang, et al. 2010. An unsupervised automated essay scoring system. IEEE Intelligent systems 25(5):61–67. find
  8. Ellis B Page. 1968. The use of the computer in analyzing student essays. International review of education 14(2):210–225. find
  9. Ellis Batten Page. 1994. Computer grading of student prose, using modern concepts and software. The Journal of experimental education 62(2):127–142. find
  10. Ronan Cummins, Meng Zhang, and Ted Briscoe. 2016. Constrained multi-task learning for automated essay scoring. Association for Computational Linguistics. find
  11. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. ICML (3) 28:1310–1318. find
  12. Yann Dauphin, Harm de Vries, and Yoshua Bengio. 2015. Equilibrated adaptive learning rates for nonconvex optimization. In Advances in Neural Information Processing Systems. pages 1504–1512. find
  13. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. volume 14, pages 1532– 1543. find
  14. Fei Dong and Yue Zhang. 2016. Automatic features for essay scoring an empirical study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 968974,. Association for Computational Linguistics, Austin, Texas, pages 1072–1077. https://www.aclweb.org/anthology… ↗
  15. Peter Phandi, Kian Ming A Chai, and Hwee Tou Ng. 2015. Flexible domain adaptation for automated essay scoring using correlated linear regression . find
  16. Yafeng Ren and Yue Zhang. 2016. Deceptive opinion spam detection using neural network. In Proceedings of COLING 2016. Association for Computational Linguistics, Osaka, Japan, pages 140– 150. http://www.aclweb.org/anthology/C/C16/C16-1014.pdf. ↗
  17. Peter W Foltz, Darrell Laham, and Thomas K Landauer. 1999. Automated essay scoring: Applications to educational technology. In proceedings of EdMedia. volume 99, pages 40–64. find
  18. Lawrence M Rudner and Tahung Liang. 2002. Automated essay scoring using bayes’ theorem. The Journal of Technology, Learning and Assessment 1(2). find
  19. Bonggun Shin, Timothy Lee, and Jinho D Choi. 2016. Lexicon integrated cnn models with attention for sentiment analysis. arXiv preprint arXiv:1610.06272 . find
  20. Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. pages 1882–1891. http://aclweb.org/anthology/D/D16/D16-1193.pdf. ↗
  21. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP. pages 1422– 1432. find
  22. Yequan Wang, Minlie Huang, Li Zhao, and Xiaoyan Zhu. 2016. Attention-based lstm for aspect-level sentiment classification. In EMNLP. pages 606– 615. find
  23. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation 1(2):270–280. find
  24. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML. volume 14, pages 77–81. find our card
  25. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL-HLT. pages 1480–1489. find our card
  26. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading esol texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, pages 1… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dong-2017-attention-based-recurrent-convolutional-neural/