Elamur Research⌕
☰
home / papers / 2025 / Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems

Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems

Paper · 2025 · Steven Coyne, Diana Galvan-Sosa, Ryan Spring et al.

Summary

Automated writing evaluation tools fix grammar well, but they mostly offer one-click direct edits. Language learners often gain more from short explanations or indirect hints, especially when an error breaks a general rule. The authors propose an annotation framework that records, for each learner error, an error type, its generalizability ("treatability"), a highlighted comment span, and a human-written feedback comment marked as either a direct correction or a hint. The new hierarchical typology classifies errors by the knowledge gap a teacher would infer behind them, not by the surface edit. Using the annotated data, they compare three LLM feedback generators: keyword-guided (using their tags, ERRANT, or EXPECT), keyword-free, and template-guided. Practicing teachers rated the outputs. All systems scored well, roughly on par with human writers, and the choice of tag set made little difference. Only the template system came close to the human mix of hints and corrections, but it degraded when no template fit. The work matters because it supports feedback that targets learning rather than just correcting text.

Key points

  • Typology: six collections (e.g., "Vocabulary", "Grammar") with nested tags and 81 terminal tags in total. Tag names follow textbook-style teaching concepts, so they can be linked to learning resources. Guidelines and resources are released on GitHub (coynestevencharles/annotating-errors-wcf).
  • Motivation from prior work (Ferris, Hyland & Hyland): teachers corrected untreatable errors directly 65.3% of the time. For treatable errors, 58.7% of feedback was indirect.
  • Teacher ratings:
    • Overall quality: about 4.45–4.50 out of 5 for the human, ERRANT, EXPECT, own-tag and no-tag sources, versus 4.184 for templates.
    • Factuality: never reached 100%, and was lowest for templates (0.921).
  • Template system:
    • Picked the correct template in 76.65% of cases (F1 0.76).
    • Recall for the "None" (withhold feedback) choice was only 0.37.
    • 4.57% of outputs contained leftover template syntax.
    • Mean quality was 4.40 with a correct template, 3.51 with a wrong one, and 2.88 when it filled a template where "None" was correct.
  • Directness: humans gave hints in 40.86% of cases. Keyword-based and keyword-free LLMs gave hints in only 0–3%, despite prompting. The template system gave hints in 39.77% (hint F1 0.78).

Interesting ideas

  • I1. Label errors by the underlying construction the writer failed at, not by the surface edit: "missing preposition" splits into phrasal verb, prepositional phrase, and so on, so the label says what knowledge is missing.
  • I2. Mark each error as rule-governed or idiosyncratic/lexical, and use that flag to choose a different strategy for each case.
  • I3. Extend an annotated span beyond the minimal edit so it covers the whole lexical unit (highlighting "put down", not just "down"), because the explanation concerns the unit as a whole.
  • I4. Template-filled generation gives fine control over output style, but it needs an explicit abstain option, robust detection of coverage gaps, and an automatic check for unfilled template syntax.
  • I5. A no-tag baseline revealed that the supplied labels barely changed output quality, which is a cheap test of whether conditioning information is actually used.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Self-induced repair rules and parser-disagreement logs could be tagged by the UD construction involved, making failure clusters explainable and easier to target.
  • I2 (learn): When a repair or correction is learned, a treatability-style flag could decide whether it becomes a general rule or stays a lexical exception, reducing overgeneralization.
  • I3 (tree): Phrasal verbs and other multiword units (compound:prt, fixed) could be treated as one span in diagnostics and frame lookup, instead of as separate tokens.
  • I4 (back): Graph-to-text templates should include a first-class "no fitting template" outcome and a post-filter, rather than forcing the nearest template.
  • I5 (learn): Run each LLM-teacher tier with and without the structured hints (tree, frames) as a negative control, to verify that the model really uses them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🎓Aix-Marseille UniversitéMarseille · France24 works · 39,160 citations🎓Tohoku UniversitySendai · Japan20 works · 514 citations🔬RIKENWako · Japan26 works · 526 citations🔬Laboratoire d’Informatique et SystèmesMarseille · France8 works · 106 citations🎓Institut National des Sciences Appliquées de LyonLyon · France3 works · 4 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Steven Coyne, Diana Galvan-Sosa, Ryan Spring, Camelia Guerraoui, Michael Zock, Keisuke Sakaguchi and Kentaro Inui. 2025. Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems. arXiv:2508.06810 · AIED LNCS 15,880 (2025) 292-306.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1007/978-3-031-98459-4_21 · arXiv 2508.06810

@misc{coyne2025annotating,
  title = {{Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems}},
  author = {Steven Coyne, Diana Galvan-Sosa, Ryan Spring, Camelia Guerraoui, Michael Zock, Keisuke Sakaguchi and Kentaro Inui},
  year = {2025},
  doi = {10.1007/978-3-031-98459-4\_21},
  eprint = {2508.06810},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1007/978-3-031-98459-4_21}
}

References

29 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bitchener, J.: Written Corrective Feedback, p. 207–225. Cambridge Handbooks in Language and Linguistics, Cambridge University Press (2021) find
  2. Brown, D., Liu, Q., Norouzian, R.: Effectiveness of written corrective feedback in developing l2 accuracy: A bayesian meta-analysis. Language Teaching Research 0(0), 13621688221147374 (2023). https://doi.org/10.1177/13621688221147374 ↗
  3. Bryant, C., Felice, M., Briscoe, T.: Automatic annotation and evaluation of error types for grammatical error correction. In: Barzilay, R., Kan, M.Y. (eds.) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 793–805. Association for Computational Linguistics, Vancouver, Canada (Jul 2017). https://doi.org/10.18653/v1/P17-1074 ↗
  4. Coyne, S.: Template-guided grammatical error feedback comment generation. In: Bassignana, E., Lindemann, M., Petit, A. (eds.) Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop. pp. 94–104. Association for Computational Linguistics, Dubrovnik, Croatia (May 2023). https://doi.org/10.18653/v1/2023.eacl-srw.10 ↗
  5. Dahlmeier, D., Ng, H.T., Wu, S.M.: Building a large annotated corpus of learner English: The NUS corpus of learner English. In: Tetreault, J., Burstein, J., Leacock, C. (eds.) Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications. pp. 22–31. Association for Computational Linguistics, Atlanta, Georgia (Jun 2013), https://aclanthology.org/W13-1703 ↗
  6. Ellis, R.: A typology of written corrective feedback types. ELT Journal 63(2), 97–107 (may 2008). https://doi.org/10.1093/elt/ccn023 ↗
  7. Fei, Y., Cui, L., Yang, S., Lam, W., Lan, Z., Shi, S.: Enhancing grammatical error correction systems with explanations. In: Rogers, A., Boyd-Graber, S. Coyne et al. J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7489–7501. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.acl-long.413 ↗
  8. Ferris, D.: The case for grammar correction in l2 writing classes: A response to truscott (1996). Journal of Second Language Writing 8(1), 1–11 (1999). https://doi.org/10.1016/S1060-3743(99)80110-6 ↗
  9. Ferris, D.: Treatment of Error in Second Language Student Writing, Second Edition. University of Michigan Press/ELT, Ann Arbor, MI (2011). https://doi.org/10.3998/mpub.2173290 ↗
  10. Ferris, D., Hyland, K., Hyland, F.: Does error feedback help student writers? New evidence on the short- and long-term effects of written error correction, p. 81–104. Cambridge Applied Linguistics, Cambridge University Press (2006). https://doi.org/10.1017/CBO9781139524742.007 ↗
  11. Goodwin, B., Miller, K.: Good feedback is targeted, specific, timely. Educational leadership: journal of the Department of Supervision and Curriculum Development, N.E.A 70, 82–83 (sep 2012) find
  12. Hanawa, K., Nagata, R., Inui, K.: Exploring methods for generating feedback comments for writing learning. In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 9719–9730. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.766 ↗
  13. Kaneko, M., Okazaki, N.: Controlled generation with prompt insertion for natural language explanations in grammatical error correction. In: Calzolari, N., Kan, M.Y., Hoste, V., Lenci, A., Sakti, S., Xue, N. (eds.) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). pp. 3955–3961. ELRA and ICCL, Torino, Italia (May 2024), https://aclanthology.org/2024.lrec-main.350 ↗
  14. Kang, E.Y., Han, Z.: The efficacy of written corrective feedback in improving l2 written accuracy: A meta-analysis. The Modern Language Journal 99, 1–18 (2015). https://doi.org/10.1111/modl.12189 ↗
  15. Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. SAGE Publications, Inc., Thousand Oaks, California, fourth edition edn. (2019). https://doi.org/10.4135/9781071878781 ↗
  16. Kulhavy, R.W., Stock, W.A.: Feedback in written instruction: The place of response certitude. Educational Psychology Review 1(4), 279–308 (1989). https://doi.org/10.1007/BF01320096 ↗
  17. Lai, Y.H., Chang, J.: TellMeWhy: Learning to explain corrective feedback for second language learners. In: Padó, S., Huang, R. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. pp. 235–240. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-3040 ↗
  18. Lee, I.: Research into practice: Written corrective feedback. Language Teaching 46(1), 108–119 (2013). https://doi.org/10.1017/S0261444812000390 ↗
  19. Li, X., Lan, Y.: Large language models are good annotators for type-aware data augmentation in grammatical error correction. In: Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B.D., Schockaert, S. (eds.) Proceedings of the 31st International Conference on Computational Linguistics. pp. 199– Annotating Learner Errors for Automated Feedback 213. Association for Computational Linguistics, Abu Dhabi, UAE (Jan 2025), https://aclanthology.org/2025.coling-main.14/ ↗
  20. Nagata, R.: Toward a task of feedback comment generation for writing learning. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3206– 3215. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1316 ↗
  21. Nagata, R., Hagiwara, M., Hanawa, K., Mita, M., Chernodub, A., Nahorna, O.: Shared task on feedback comment generation for language learners. In: Belz, A., Fan, A., Reiter, E., Sripada, Y. (eds.) Proceedings of the 14th International Conference on Natural Language Generation. pp. 320–324. Association for Computational Linguistics, Aberdeen, Scotland, UK (Aug 2021). https://doi.org/10.18653/v1/2021.inlg-1.35 ↗
  22. Nagata, R., Inui, K., Ishikawa, S.: Creating corpora for research in feedback comment generation. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Twelfth Language Resources and Evaluation Conference. pp. 340–345. European Language Resources Association, Marseille, France (May 2020), https://aclanthology.org/2020.lrec-1.42 ↗
  23. Narciss, S.: Designing and evaluating tutoring feedback strategies for digital learning environments on the basis of the interactive tutoring feedback model. Digital Education Review 23,, 7–26 (06 2013), https://eric.ed.gov/?id=EJ1013726 ↗
  24. Nicholls, D.: The cambridge learner corpus: Error coding and analysis for lexicography and elt. Proceedings of the Corpus Linguistics 2003 conference p. 572–581 (2003), https://ucrel.lancs.ac.uk/publications/CL2003/papers/nicholls.pdf ↗
  25. OpenAI: Gpt-4o system card (2024), https://arxiv.org/abs/2410.21276 ↗
  26. Sheen, Y.: Corrective Feedback, Individual Differences and Second Language Learning, Educational Linguistics, vol. 13. Springer Netherlands, Dordrecht (2011). https://doi.org/10.1007/978-94-007-0548-7 ↗
  27. Shute, V.J.: Focus on formative feedback. Review of Educational Research 78(1), 153–189 (2008). https://doi.org/10.3102/0034654307313795 ↗
  28. Song, Y., Krishna, K., Bhatt, R., Gimpel, K., Iyyer, M.: GEE! grammar error explanation with large language models. In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024. pp. 754– 781. Association for Computational Linguistics, Mexico City, Mexico (Jun 2024). https://doi.org/10.18653/v1/2024.findings-naacl.49 ↗
  29. Yannakoudakis, H., Briscoe, T., Medlock, B.: A new dataset and method for automatically grading ESOL texts. In: Lin, D., Matsumoto, Y., Mihalcea, R. (eds.) Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pp. 180–189. Association for Computational Linguistics, Portland, Oregon, USA (Jun 2011), https://aclanthology.org/P11-1019 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/coyne-2025-annotating-errors-english-learners-written/