Elamur Research⌕
☰
home / papers / 2016 / A Neural Approach to Automated Essay Scoring

A Neural Approach to Automated Essay Scoring

Paper · 2016 · Kaveh Taghipour and Hwee Tou Ng

Summary

Traditional automated essay scoring (AES) systems depend on hand-built features such as length, grammaticality, readability or coherence, and designing these features takes a great deal of effort. Taghipour and Ng replace feature engineering with an end-to-end neural regressor that reads the raw essay and predicts its score. The architecture has four stages: a learned word-embedding lookup, an optional convolution over word windows, a recurrent layer, and a sigmoid output. Instead of using only the recurrent layer's last state, the model averages all of its hidden states over time. The authors compare basic RNN, GRU and LSTM units, along with several architectural variants, on the eight-prompt ASAP dataset from the Kaggle competition, using quadratic weighted Kappa (QWK) as the metric. LSTM with mean-over-time works best. An ensemble of CNN and LSTM models beats a strong open-source baseline by 5.6% QWK, and its agreement with human raters comes close to the agreement between two humans. The work shows that competitive essay scoring is possible without manual features, and the code is released publicly.

Key points

  • Data and evaluation:
    • Uses the ASAP dataset (Kaggle, sponsored by the Hewlett Foundation), which covers 8 prompts.
    • QWK is computed per prompt and then averaged.
    • Significance is checked with a one-tailed paired t-test.
  • Recurrent unit: RNN, GRU and LSTM units were compared, and LSTM performed best.
  • Pooling and variants:
    • Mean-over-time pooling of hidden states beat both an attention mechanism and using only the final state.
    • Deeper and bidirectional LSTM variants gave no further gains, which the authors attribute to limited training data relative to model capacity.
  • Agreement with humans: The best system (CNN+LSTM ensemble) reaches an average QWK of 0.717 against rater H1 and 0.710 against rater H2, versus 0.754 between the two humans. On some prompts the system agrees with a human rater more than the humans agree with each other.
  • Comparison with Alikaniotis et al. (2016):
    • The authors reproduced that paper's 80/20 split, where QWK is computed over the whole test set on a 0–60 scale.
    • Under this setup the LSTM scores 0.987, compared with 0.96 for the earlier system.
    • The authors note that this setup inflates QWK.
  • Code: released at github.com/nusnlp/nea.

Interesting ideas

  • I1. Average all recurrent states instead of keeping the last one. For long inputs such as essays, mean-over-time pooling keeps information that a single final state would lose. This trick carries over to any long-document classification or regression task.
  • I2. Add a convolution layer before the RNN. It extracts n-gram-level local features first, giving the recurrent layer richer inputs. This is a cheap way to add local context to sequence models.
  • I3. Question how the metric is aggregated. Pooling QWK over prompts with very different score ranges can make results look far better than they are, so per-prompt computation is the more honest choice. This lesson applies to any benchmark that mixes heterogeneous label scales.
  • I4. Benchmark against human–human agreement. Comparing system–human agreement with agreement between two human raters gives a realistic upper bound. It is useful wherever gold labels are subjective, such as annotation-heavy NLP tasks.
  • I5. Track the prediction word by word. Watching how the score changes after each token showed that the model penalizes essays shorter than about 50 words regardless of content. Simple probing of this kind can expose shortcut features in other neural models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 1Research subjects and languages the work is about.Grammatical error correction & learner language · 80 works

Cite

Cite as Kaveh Taghipour and Hwee Tou Ng. 2016. A Neural Approach to Automated Essay Scoring. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1193

@inproceedings{taghipour2016neural,
  title = {{A Neural Approach to Automated Essay Scoring}},
  author = {Kaveh Taghipour and Hwee Tou Ng},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1193},
  url = {https://doi.org/10.18653/v1/d16-1193}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  2. Yigal Attali and Jill Burstein. 2004. Automated essay scoring with e-rater R v. 2.0. Technical report, Educational Testing Service. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations.
  4. Hongbo Chen and Ben He. 2013. Automated essay scoring by maximizing human-machine agreement. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find
  5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  6. Yann N. Dauphin, Harm de Vries, and Yoshua Bengio. 2015. Equilibrated adaptive learning rates for non-convex optimization. In Advances in Neural Information Processing Systems 28. find
  7. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  8. Peter W Foltz, Darrell Laham, and Thomas K Landauer. 1999. The Intelligent Essay Assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):… find
  9. Ozan Irsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. find
  10. Rudolf Kadlec, Martin Schmid, and Jan Kleindienst. 2015. Improved deep learning baselines for Ubuntu corpus dialogs. In Proceesings of the NIPS 2015 find
  11. Workshop on Machine Learning for Spoken Language Understanding and Interaction. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.
  12. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  13. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning. find
  14. Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.
  15. Isaac Persing, Alan Davis, and Vincent Ng. 2010. Modeling organization in student essays. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.
  16. Peter Phandi, Kian Ming A. Chai, and Hwee Tou Ng. 2015. Flexible domain adaptation for automated essay scoring using correlated linear regression. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Mark D. Shermis and Jill Burstein, editors. 2013. Handbook of… find
  17. Routledge. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. Martin Sundermeyer, Hermann Ney, and Ralf Schlüter. 2015. From fe… find
  18. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems 28. find
  19. Helen Yannakoudakis and Ronan Cummins. 2015. Evaluating the performance of automated text scoring systems. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. find
  20. Torsten Zesch, Michael Wojatzki, and Dirk Scholten-Akoun. 2015. Task-independent features for automated essay grading. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. find
  21. Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/taghipour-2016-neural-approach-automated-essay-scoring/