Elamur Research⌕
☰
home / papers / 2025 / A Taxonomy of Errors in English as she is spoke: Toward an AI-Based Method of Error Analysis for EFL Writing Instruction

A Taxonomy of Errors in English as she is spoke: Toward an AI-Based Method of Error Analysis for EFL Writing Instruction

Paper · 2025 · Damian Heywood; Joseph Andrew Carrier; and Kyu-Hong Hwang

Summary

The paper asks whether an LLM can go beyond holistic, rubric-style scoring of EFL writing and instead find, label and correct individual errors against a theory-grounded taxonomy. The authors build a Python pipeline that makes API calls to Claude 3.5 Sonnet and DeepSeek R1. The pipeline is guided by an error taxonomy that starts from Corder's (1981) linguistic-level scheme, is adapted for written EFL production via Seddik (2023), and also draws on Richards and James. Errors are coded at word and sentence level, covering spelling, grammar and punctuation.

The taxonomy was first refined on isolated single errors, which exposed categories that overlapped. The final test used a dialogue from Fonseca's 1855 phrasebook English as she is spoke, a famously error-saturated text, and an expert panel reviewed the output. The models detected a wide range of error types. They were weaker on context, and when an error had no code they sometimes invented new categories. The work matters because fine-grained, consistent error feedback could ease teacher workload and give learners far more specific guidance. Wider validation on real student writing is still needed.

Key points

  • Two hypotheses were tested:
    • H1: an AI system can consistently apply an established error taxonomy.
    • H2: expert linguists confirm the resulting classifications.
  • Models used: Claude 3.5 Sonnet and DeepSeek R1, called from Python code.
  • The study builds on the authors' earlier rubric-based scoring work, which reached over 99% consistency across five repeated runs (Heywood & Carrier, 2024).
  • The taxonomy rests mainly on Corder (1981), with input from Richards (1971) and James (1998), and is adapted from Seddik (2023). It is hierarchical, with word-level and sentence-level branches.
  • Acknowledged limitations:
    • Only a single short text was analysed.
    • The training examples did not cover every category.
    • API costs and local hardware limited the scale.
    • Human verification was narrow.

Interesting ideas

  • I1. Using a historical text that is naturally packed with authentic, layered errors (Fonseca's phrasebook) gives a ready-made adversarial benchmark with no need for synthetic corruption.
  • I2. The taxonomy was tuned first on isolated, single-error items and only then run on a dense mixed-error passage, a unit-test-then-integration-test progression that exposed overlapping categories early.
  • I3. When the model met an error it had no code for, it coined a new category; this can be read as a signal about gaps in the label scheme, not just as a failure.
  • I4. Burt and Kiparsky's split between global errors, which break sentence organisation and meaning, and local errors, which only damage a single element, offers a natural way to prioritise what to fix.
  • I5. Repeating the same LLM judgment several times and measuring agreement served as the reliability measure in the authors' prior work.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Parse such a text with the perceptron parser and gparse to see how gracefully UD trees degrade on non-native, garbled English, and where the two parsers disagree.
  • I2 (learn): Build evaluation sets of minimal sentences with one injected error each, used as negative controls, before scoring on messy real text.
  • I3 (learn): Log LLM-teacher labels that fall outside the allowed schema and review them as candidates for missing repair rules or label-set extensions, instead of silently discarding them.
  • I4 (frames): Flag input errors by whether they corrupt who-did-what-to-whom frames (global) or leave the roles recoverable (local), and let only global ones block downstream reasoning.
  • I5 (learn): Query the LLM teacher tiers multiple times per item and use run-to-run agreement as a confidence weight when admitting silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Damian Heywood; Joseph Andrew Carrier; and Kyu-Hong Hwang. 2025. A Taxonomy of Errors in English as she is spoke: Toward an AI-Based Method of Error Analysis for EFL Writing Instruction. arXiv:2512.00392.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2512.00392 · arXiv 2512.00392

@misc{carrier2025taxonomy,
  title = {{A Taxonomy of Errors in English as she is spoke: Toward an AI-Based Method of Error Analysis for EFL Writing Instruction}},
  author = {Damian Heywood; Joseph Andrew Carrier; and Kyu-Hong Hwang},
  year = {2025},
  doi = {10.48550/arxiv.2512.00392},
  eprint = {2512.00392},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2512.00392}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Corder, S. P. (1981). Error analysis and interlanguage. Oxford University Press. find
  2. Dulay, H. C., Burt, M. K., & Krashen, S. D. (1982). Language two. Oxford University Press. find
  3. Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20(1), 57. https://doi.org/10.1186/s41239-023-00425-2 ↗
  4. Fonseca, J. da, Carolino, P., & Collins, P. S. (2004). English as she is spoke: The new guide of the conversation, in Portuguese and English, in two parts (Rev.). McSweeney’s Books. find
  5. Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and instructed language learning. Language Learning and Technology, 26(2), 5–24. https://hdl.handle.net/10125/73474 ↗
  6. Heywood, D., & Carrier, J. (2024). Ai-automated writing assessment. 한국언어과학회 학술대회, 219–233. https://www.earticle.net/Article/A457474 ↗
  7. Jabeen, A., Kazemian, B., & Shahbaz Mustafai, M. (2015). The role of error analysis in teaching and learning of second and foreign language. Education and Linguistics Research, 1(2), 52. https://doi.org/10.5296/elr.v1i2.8189 ↗
  8. James, C. (1998). Errors in language learning and use (1st ed.). Routledge. https://doi.org/10.4324/9781315842912 Karim, A., Mohamed, A. R., Ismail, S. A. M. M., Shahed, F. H., Rahman, M. M., & Haque, Md. ↗
  9. H. (2018). Error analysis in efl writing classroom. International Journal of English Linguistics, 8(4), 122. https://doi.org/10.5539/ijel.v8n4p122 ↗
  10. Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2023). Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense. arXiv. https://doi.org/10.48550/ARXIV.2303.13408 ↗
  11. Kurt, G., & Kurt, Y. (2024). Enhancing l2 writing skills: Chatgpt as an automated feedback tool. Journal of Information Technology Education: Research, 23, 024. https://doi.org/10.28945/5370 ↗
  12. Nikoopour, J., & Zoghi, A. (2014). Analyzing efl learners’ errors: The plausibility of teachers’ feedbacks and students’ uptakes. Journal of Language Teaching and Research, 5(1), 226–233. https://doi.org/10.4304/jltr.5.1.226-233 ↗
  13. Ren, M. (2024). Advancements and applications of large language models in natural language processing: A comprehensive review. Applied and Computational Engineering, 97(1), 55–63. https://doi.org/10.54254/2755-2721/97/20241406 ↗
  14. Richards, J. C. (1971). A non-contrastive approach to error analysis. ELT Journal, XXV(3), 204–219. https://doi.org/10.1093/elt/XXV.3.204 ↗
  15. Richards, J. C. (Ed.). (1974). Error analysis: Perspectives on second language acquisition. Longman. find
  16. Seddik, M. (2023). Error analysis in EFL students’ writing skill. Journal of English Language Teaching and Applied Linguistics, 5(4), 163–172. https://doi.org/10.32996/jeltal.2023.5.4.16 ↗
  17. Shi, H., & Aryadoust, V. (2024). A systematic review of AI-based automated written feedback research. ReCALL, 36(2), 187–209. https://doi.org/10.1017/S0958344023000265 ↗
  18. Tate, T. P., Doroudi, S., Ritchie, D., Xu, Y., & Uci, M. W. (2023). Educational research and AI-generated writing: Confronting the coming tsunami [Preprint]. EdArXiv, January, 10. https://doi.org/10.35542/osf.io/4mec3 ↗
  19. Tseng, W., & Warschauer, M. (2023a). AI-writing tools in education: If you can’t beat them, join them. Journal of China Computer-Assisted Language Learning, 3(2), 258–262. https://doi.org/10.1515/jccall-2023-0008 ↗
  20. Wang, H., & Troia, G. A. (2023). How students’ writing motivation, teachers’ personal and professional attributes, and writing instruction impact student writing achievement: A two-level hierarchical linear modeling study. Frontiers in Psychology, 14, 1213929. https://doi.org/10.3389/fpsyg.2023.1213929 ↗
  21. Wu, E., Wu, K., & Zou, J. (2024). FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs? arXiv. https://doi.org/10.48550/ARXIV.2411.05059 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/heywood-2025-taxonomy-errors-english-she-spoke/