Elamur Research⌕
☰
home / papers / 2011 / A New Dataset and Method for Automatically Grading ESOL Texts

A New Dataset and Method for Automatically Grading ESOL Texts

Summary

Automated essay scoring systems were already used in high-stakes exams, but they could not be compared fairly because no shared public data existed. The authors release such a resource: First Certificate in English (FCE) scripts from the Cambridge Learner Corpus (CLC). Each script comes with an overall mark, error annotations, prompts and learner metadata. They frame grading as learning to order texts by quality rather than predicting a score. To do this, they train a rank preference SVM (SVM light) on features produced by the generic RASP parser. These features include word and PoS ngrams, phrase-structure rules, grammatical-relation distances, script length and an error-rate estimate. Ablation tests measure what each feature group contributes, and the ranker is compared against a regression SVM. On the held-out test set, which four senior examiners re-marked, the system comes close to the human upper bound, measured as the agreement between examiners. Tests on deliberately unusual "outlier" texts show where the model can be fooled. The resulting benchmark and prompt-independent method made reproducible AA research possible.

Key points

  • Dataset: 1141 FCE scripts from 2000 for training and 97 scripts from 2001 for testing. Each script has a different author, and the two years share no prompts. Marks range from 1 to 40 and are adjusted with a RASCH model.
  • Each script is hand-coded for learner errors using a scheme of about 80 error types (Nicholls, 2003), such as "TV" for wrong verb tense.
  • Model: a linear rank preference SVM trained on pairwise difference vectors. It is compared with an SVM in ε-insensitive regression mode.
  • Features come from RASP and the CLAWS tagset:
    • lower-cased word uni/bigrams;
    • PoS uni/bi/trigrams;
    • PS rule names;
    • GR distance measures;
    • script length;
    • error-rate.
  • Error-rate turned out to be among the most discriminative signals. The outlier experiments showed that the system lacks features for discourse coherence.

Interesting ideas

  • I1. Rank instead of regress: The model learns pairwise orderings of texts, so no extra step is needed to map classifier outputs onto a grade scale. This applies to any task with ordinal labels, such as readability, translation quality or annotation confidence.
  • I2. Human agreement as the ceiling: Test data was re-marked by several examiners, and model scores are reported against inter-annotator agreement. This shows how much room for improvement is realistically left, which suits any subjective labelling task.
  • I3. Prompt-agnostic design: The model deliberately ignores the prompt and is tested on unseen prompts from a different year. This lets a grader move to new tasks without retraining, which matters when labelled data per topic is scarce.
  • I4. Generic parser output as features: Rule names from an off-the-shelf parser encode detailed grammatical constructions, including rare or extragrammatical ones. This gives cheap proxies for syntactic sophistication that could be reproduced with UD parsers for other languages, such as Ukrainian learner texts.
  • I5. Adversarial "outlier" probing: The authors feed the model artificially odd texts to find where its scores diverge from an examiner's. This is a lightweight robustness check against gaming that any deployed scoring or classification model could adopt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

356 papers of our library cite it (of the 140884 whose bibliographies we read); 27 of them have a page here, 329 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yannakoudakis2011new,
  title = {{A New Dataset and Method for Automatically Grading ESOL Texts}},
  author = {Yannakoudakis, Helen and Briscoe, Ted and Medlock, Ben},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1019/}
}

References

32 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2. Journal of Technology, Learning, and Assessment, 4(3):1–30. find
  2. Burton H. Bloom. 1970. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422–426, July. find
  3. E.J. Briscoe, J. Carroll, and R Watson. 2006. The second release of the RASP system. In ACL-Coling’06 Interactive Presentation Session, pages 77–80, Sydney, Australia. find our card
  4. E.J. Briscoe, B. Medlock, and Ø. Andersen. 2010. Automated Assessment of ESOL Free Text Examinations. Cambridge University, Computer Laboratory, TR-790. find
  5. Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu, Martin Chodorow, Lisa Braden-Harder, and Mary Dee Harris. 1998. Automated scoring using a hybrid feature identification technique. Proceedings of the 36th annual meeting on Association for Computational Linguistics, pages 206–210. find
  6. YY Chen, CL Liu, TH Chang, and CH Lee. 2010. An Unsupervised Automated Essay Scoring System. IEEE Intelligent Systems, pages 61–67. find
  7. Semire Dikli. 2006. An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1). find
  8. S. Elliot. 2003. IntelliMetric: From here to validity. In M.D. Shermis and J.C. Burstein, editors, Automated essay scoring: A cross-disciplinary perspective, pages 71–86. find
  9. Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008. Introducing and evaluating ukWaC, a very large web-derived corpus of English. find
  10. In S. Evert, A. Kilgarriff, and S. Sharoff, editors, Proceedings of the 4th Web as Corpus Workshop (WAC-4). find
  11. G.H. Fischer and I.W. Molenaar. 1995. Rasch models: Foundations, recent developments, and applications. Springer. find
  12. Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. In Proceedings of the European Conference on Machine Learning, pages 137–142. Springer. find our card
  13. Thorsten Joachims. 1999. Making large scale SVM learning practical. In B. Schölkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods - Support Vector Learning. MIT Press. find
  14. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), pages 133–142. ACM. find
  15. T.K. Landauer and P.W. Foltz. 1998. An introduction to latent semantic analysis. Discourse processes, pages 259–284. find
  16. T.K. Landauer, D. Laham, and P.W. Foltz. 2003. Automated scoring and annotation of essays with the Intelligent Essay Assessor. In M.D. Shermis and J.C. Burstein, editors, Automated essay scoring: A cross-disciplinary perspective, pages 87–112. find
  17. Deryle Lonsdale and D. Strong-Krause. 2003. Automated rating of ESL essays. In Proceedings of the HLT-NAACL 2003 Workshop: Building Educational Applications Using Natural Language Processing. find
  18. Eleni Miltsakaki and Karen Kukich. 2000. Automated evaluation of coherence in student essays. In Proceedings of LREC 2000. find
  19. Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering, 10(01):25–55, March. find
  20. D. Nicholls. 2003. The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 conference, pages 572–581. find
  21. E.B. Page. 2003. Project essay grade: PEG. In M.D. Shermis and J.C. Burstein, editors, Automated essay scoring: A cross-disciplinary perspective, pages 43– 54. find
  22. D. Pérez-Marı́n, Ismael Pascual-Nieto, and P. Rodrı́guez. 2009. Computer-assisted assessment of free-text answers. The Knowledge Engineering Review, 24(04):353–374, December. find
  23. D.E. Powers, J.C. Burstein, M. Chodorow, M.E. Fowles, and K. Kukich. 2002. Stumping e-rater: challenging the validity of automated essay scoring. Computers in Human Behavior, 18(2):103–134. find
  24. L.M. Rudner and Tahung Liang. 2002. Automated essay scoring using Bayes’ theorem. The Journal of Technology, Learning and Assessment, 1(2):3–21. find
  25. L.M. Rudner, Veronica Garcia, and Catherine Welch. 2006. An Evaluation of the IntelliMetric Essay Scoring System. Journal of Technology, Learning, and Assessment, 4(4):1–21. find
  26. D.D.K. Sleator and D. Templerley. 1995. Parsing English with a link grammar. Proceedings of the 3rd International Workshop on Parsing Technologies, ACL. find our card
  27. AJ Smola. 1996. Regression estimation with support vector learning machines. Master’s thesis, Technische Universität Munchen. find
  28. J.H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245– 251. find
  29. Salvatore Valenti, Francesca Neri, and Alessandro Cucchiarelli. 2003. An overview of current research on automated essay grading. Journal of Information Technology Education, 2:3–118. find
  30. Vladimir N. Vapnik. 1995. The nature of statistical learning theory. Springer. find
  31. E. J. Williams. 1959. The Comparison of Regression Variables. Journal of the Royal Statistical Society. Series B (Methodological), 21(2):396–399. find
  32. DM Williamson. 2009. A Framework for Implementing Automated Scoring. In Annual Meeting of the American Educational Research Association and the National Council on Measurement in Education, San Diego, CA. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yannakoudakis-2011-new-dataset-method-automatically-grading/