Elamur Research⌕
☰
home / papers / 2026 / Multi-component student writing profiles for expert-aligned automated evaluation of English learner essays.

Multi-component student writing profiles for expert-aligned automated evaluation of English learner essays.

Paper · 2026 · Russell Moore, Andrew Caines and Paula Buttery

Summary

Automated Writing Evaluation tools usually give learners one score or coarse statistics, and these do not match well with what expert teachers would say. The authors represent each essay as a "writing profile" with three parts: an estimated CEFR level, rates for 26 grammatical error categories, and a histogram of vocabulary by CEFR level. They use the Write & Improve Corpus 2024 (WI_CORPUS), where final drafts carry expert CEFR grades and human corrections. Expert profiles built from these serve as targets. Auto profiles come from the platform automarker, Language Tool, ERRANT, spaCy and the English Vocabulary Profile.

A three-branch feedforward network combines user and task features and is trained to map auto profiles onto expert ones. The model gets substantially closer to expert judgements than the raw automarker and than Zaidi et al. (2019). It also forecasts final-draft profiles from early drafts and predicts profiles for a student's next task. This makes interpretable, longitudinal feedback and curriculum planning more feasible.

Key points

  • Data: the filtered WI_CORPUS contains 679 learners, 50 prompts and about 23k submissions. Drafting series have a median length of 3.
  • Task 1 (calibration), best configuration:
    • CEFR RMSE is 0.809, GER cosine proximity is about 0.67, and VOCAB EMD is 0.011.
    • Compared with the automarker, this is a 36% lower RMSE and an 84% gain in GER similarity.
    • Compared with Zaidi et al. (2019), RMSE improves from 1.093 and cosine proximity from 0.426, i.e. 26% and 57% better.
  • Forecasting:
    • From earlier drafts (Task 2), CEFR RMSE is 0.874, GER cosine proximity is 0.659 and EMD is 0.014.
    • For the next prompt (Task 3), the figures are 0.963, 0.649 and 0.028 respectively.
  • Loss choice and learned representations:
    • Using a Euclidean loss for error rates keeps GER RMSE at roughly 0.006–0.007. With a cosine loss instead, GER RMSE rises to 0.332.
    • User-branch embeddings, projected with MDS, cluster by expert CEFR level and by prompt difficulty, even though neither label is used in training.
  • Resource and limitations:
    • An open visualiser (github.com/rjm49/bea26_visualiser) shows each student's predicted, expert and automarker trajectories.
    • The model is Markovian, and the study covers only one dataset and platform.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Russell Moore, Andrew Caines and Paula Buttery. 2026. Multi-component student writing profiles for expert-aligned automated evaluation of English learner essays. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.39

@inproceedings{moore2026multi,
  title = {{Multi-component student writing profiles for expert-aligned automated evaluation of English learner essays.}},
  author = {Russell Moore, Andrew Caines and Paula Buttery},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.39},
  url = {https://doi.org/10.18653/v1/2026.bea-1.39}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, and 21 others. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Øistein E Andersen, Helen Yannakoudakis, Fiona Barker, and Tim Parish. 2013. Developing and testing a self-assessment and tutoring system. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 32–41. find
  3. John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. 1995. Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2):167–207. ↗
  4. John S. Bridle. 1990. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neural Networks for Signal Processing, pages 227–236. MIT Press. ↗
  5. Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  6. Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical error correction: A survey of the state of the art. Computational Linguistics, 49(3):643–701. ↗
  7. Annette Capel. 2015. The English Vocabulary Profile. In J. Harrison and F. Barker, editors, English Profile in Practice. Cambridge: Cambridge University Press. find
  8. Hao Cen, Kenneth Koedinger, and Brian Junker. 2006. Learning factors analysis–a general method for cognitive model evaluation and improvement. In International Conference on Intelligent Tutoring Systems, pages 164–175. Springer. find
  9. François Chollet and 1 others. 2015. Keras. https://keras.io. ↗
  10. Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4:253–278. ↗
  11. Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5):757–798. ↗
  12. Joseph B Kruskal. 1964. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1):1–27. ↗
  13. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10(8):707–710. find
  14. Watheq Ahmad Mansour, Salam Albatarni, Sohaila Eltanbouly, and Tamer Elsayed. 2024. Can large language models automatically score proficiency of written essays? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2777–2786, Torino, Italia. ELRA and ICCL. find
  15. Elijah Mayfield and Alan W Black. 2020. Should you fine-tune BERT for automated essay scoring? In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 151–162, Seattle, WA, USA → Online. Association for Computational Linguistics. find
  16. Marcin Miłkowski. 2010. Developing an open-source, rule-based proofreading tool. Software: Practice and Experience, 40(7):543–566. ↗
  17. Ines Montani, Matthew Honnibal, Matthew Honnibal, Adriane Boyd, Sofie Van Landeghem, and Henning Peters. 2023. explosion/spacy: v3.7.2: Fixes for apis and requirements. find
  18. Diane Nicholls, Andrew Caines, and Paula Buttery. 2024. find
  19. The Write & Improve Corpus 2024: Error-annotated and CEFR-labelled essays by learners of English. Cambridge University Press & Assessment. find
  20. Brian North. 2005. The CEFR levels and descriptor scales. In Lynda Taylor and Cyril Weir, editors, Multilingualism and assessment: Achieving transparency, assuring quality, sustaining diversity. Proceedings of the ALTE Berlin Conference. Cambridge: Cambridge University Press. find
  21. Anne O’Keeffe and Geraldine Mark. 2017. The English Grammar Profile of learner competence: Methodology and key findings. International Journal of Corpus Linguistics, 22(4):457–489. ↗
  22. Philip I Pavlik Jr, Hao Cen, and Kenneth R Koedinger. 2009. Performance factors analysis–a new alternative to knowledge tracing. In Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling. ↗
  23. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  24. Radek Pelánek. 2018. The details matter: methodological nuances in the evaluation of student models. User Modeling and User-Adapted Interaction, 28(3):207– 235. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moore-2026-multi-component-student-writing-profiles/