Elamur Research⌕
☰
home / papers / 2026 / Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs

Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs

Paper · 2026 · Stefano Banno, Penny Karanasou, Kate Knill et al.

Summary

The paper tackles automatic analysis of L2 English grammar that goes beyond error spotting. It also records which constructions learners attempt and whether they get them right. The authors build on the English Grammar Profile (EGP), a catalogue of over 1,000 constructs. Each construct is written as a can-do statement, tied to a CEFR level, and illustrated with learner examples. The framework compares each learner sentence with a corrected version. A construct found in the corrected text counts as attempted, and its form in the original shows success or failure. For 12 selected EGP statements, the authors annotate a subset of learner essays. They then compare two LLMs (Qwen 2.5 32B, GPT-4.1) with two rule-based systems (their own RB and the public POLKE). LLMs are stronger on constructs that depend on meaning or pragmatics, while rules stay competitive on purely morphosyntactic ones. Detected attempts, especially successful ones, predict holistic CEFR proficiency, and a rule pre-filter followed by a zero-shot LLM gives the best scoring. Replacing manual corrections with GECToR output costs little, mainly for successful attempts. The approach enables positive, actionable feedback.

Key points

  • The resource is the English Grammar Profile. It was chosen over the GSE and the CEFR-J Grammar Profile because it is downloadable, covers many L1 backgrounds, and includes learner examples.
  • The new annotated dataset targets 12 EGP can-do statements on essays drawn from a publicly available L2 learner corpus.
  • Four systems are compared: RB, POLKE, Qwen 2.5 32B and GPT-4.1. Each is scored on general, successful and unsuccessful attempts using precision, recall and F1.
  • There are two pipelines. A semi-automated one uses manual corrections, and a fully automated one uses GECToR corrections.
  • Unsuccessful attempts are much harder for every system than general or successful ones. With GECToR corrections, the averaged F1 for unsuccessful attempts stays below 50.

Interesting ideas

  • I1. Analyse a sentence together with its corrected version: a construct present in both is a success, while one present only after correction is a failed attempt, so the system captures intent as well as errors.
  • I2. Use cheap high-recall rules as a pre-filter and let an LLM confirm only the flagged candidates, which keeps costs low and reduces false positives.
  • I3. Assign each construct to rules or to a model according to whether it needs semantic or pragmatic interpretation, rather than using one method for everything.
  • I4. Use interpretable, level-tagged can-do statements as features for proficiency scoring, so every score can be traced to concrete constructions the writer produced.
  • I5. Measure error propagation by running the same evaluation with gold upstream input (manual corrections) and with predicted upstream input (GEC output).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Parse both the learner text and its correction into UD trees and diff the matched subtrees, giving an explainable "attempted but malformed" signal without any bag-of-words scoring.
  • I2 (learn): UD-pattern matchers could propose candidate constructions, and an LLM teacher tier would judge only those, producing silver labels more cheaply.
  • I3 (frames): Purely morphosyntactic constructs can stay in deterministic tree patterns, while politeness forms and other pragmatic constructs go to the prag crate's speech-act and implicature logic.
  • I4 (answer): When grading student answers in school-course tests, report which levelled constructions were used correctly as positive, explainable feedback.
  • I5 (learn): Evaluate frames and world steps on both gold and parser-produced trees to see how much of each downstream error comes from the parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Stefano Banno, Penny Karanasou, Kate Knill and Mark Gales. 2026. Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs. arXiv:2603.17171.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.17171 · arXiv 2603.17171

@misc{banno2026exploiting,
  title = {{Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs}},
  author = {Stefano Banno, Penny Karanasou, Kate Knill and Mark Gales},
  year = {2026},
  doi = {10.48550/arxiv.2603.17171},
  eprint = {2603.17171},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.17171}
}

References

72 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. B. Klebanov and N. Madnani, Automated essay scoring. n.p.: Springer Nature, 2022. find
  2. S. Li and V. Ng, “Automated essay scoring: Recent successes and future directions,” in Proc. of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, 8 2024, pp. 8114–8122, Survey Track. [Online]. Available: https://doi.org/10.24963/ijcai.2024/897 ↗
  3. L. Hamp-Lyons, “Rating nonnative writing: The trouble with holistic scoring,” TESOL Quarterly, vol. 29, no. 4, pp. 759–762, 1995. [Online]. Available: http://www.jstor.org/stable/3588173 ↗
  4. S. C. Weigle, Assessing Writing, ser. Cambridge Language Assessment. Cambridge: Cambridge University Press, 2002. find
  5. B. Azar, “Grammar-based teaching: A practitioner’s perspective,” TESL-EJ, vol. 11, no. 2, 2007. find
  6. C. Bryant, Z. Yuan, M. R. Qorib et al., “Grammatical Error Correction: A Survey of the State of the Art,” Computational Linguistics, pp. 1–59, 2023. ↗
  7. H. Yannakoudakis, T. Briscoe, and B. Medlock, “A new dataset and method for automatically grading ESOL texts,” in Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Jun. 2011, pp. 180–189. [Online]. Available: https://aclanthology.org/P11-1019 ↗
  8. M. Gamon, M. Chodorow, C. Leacock, and J. Tetreault, “Grammatical error detection in automatic essay scoring and feedback,” in Handbook of Automated Essay Evaluation, M. Shermis and J. Burstein, Eds. New York: Routledge, 2013, ch. 15, pp. 251–266. ↗
  9. J. A. Hawkins and P. Buttery, “Criterial features in learner corpora: Theory and illustrations,” English Profile Journal, vol. 1, pp. 1–23, 2010. ↗
  10. B. Bulté and A. Housen, “Defining and operationalising L2 complexity,” Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA, vol. 32, p. 21, 2012. ↗
  11. G. Lan, Q. Liu, and S. Staples, “Grammatical complexity: ‘What Does It Mean’ and ‘So What’ for L2 writing classrooms?” Journal of Second Language Writing, vol. 46, p. 100673, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1060374319303649 ↗
  12. K. E. Wolfe-Quintero, S. Inagaki, and H.-Y. Kim, Second Language Development in Writing: Measures of Fluency, Accuracy, & Complexity. Honolulu, HI: University of Hawai’i, Second Language Teaching & Curriculum Center, 1998. find
  13. L. Ortega, “Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing,” Applied Linguistics, vol. 24, no. 4, pp. 492–518, 2003. ↗
  14. X. Lu, “Automatic analysis of syntactic complexity in second language writing,” International journal of corpus linguistics, vol. 15, no. 4, pp. 474–496, 2010. ↗
  15. D. S. McNamara, S. A. Crossley, and P. M. McCarthy, “Linguistic features of writing quality,” Written Communication, vol. 27, no. 1, pp. 57–86, 2010. ↗
  16. N. Iwashita, “Syntactic complexity measures and their relation to oral proficiency in Japanese as a foreign language,” Language Assessment Quarterly: An International Journal, vol. 3, no. 2, pp. 151–169, 2006. ↗
  17. M. Chen and K. Zechner, “Computing and evaluating syntactic complexity features for automated scoring of spontaneous non-native speech,” in Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Jun. 2011, pp. 722–731. [Online]. Available: https://aclanthology.org/P11-1073 ↗
  18. S.-Y. Yoon and S. Bhat, “Assessment of ESL learners’ syntactic competence based on similarity measures,” in Proc. of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jul. 2012, pp. 600–608. [Online]. Available: https://aclanthology.org/D12-1055 ↗
  19. A. O’Keeffe and G. Mark, “The English Grammar Profile of learner competence: Methodology and key findings,” International Journal of Corpus Linguistics, vol. 22, no. 4, pp. 457–489, 2017. ↗
  20. Council of Europe, Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press, 2001. [Online]. Available: https://rm.coe.int/1680459f97 ↗
  21. K. P. Yancey, G. Laflair, A. Verardi, and J. Burstein, “Rating short L2 essays on the CEFR scale with GPT-4,” in Proc. of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Jul. 2023, pp. 576–584. [Online]. Available: https://aclanthology.org/2023.bea-1.49 ↗
  22. S. Bannò, H. K. Vydana, K. Knill, and M. Gales, “Can GPT-4 do L2 analytic assessment?” in Proc. of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Jun. 2024, pp. 149–164. [Online]. Available: https://aclanthology.org/2024.bea-1.14 ↗
  23. B. Naismith, P. Mulcaire, and J. Burstein, “Automated evaluation of written discourse coherence using GPT-4,” in Proc. of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Jul. 2023, pp. 394–403. [Online]. Available: https://aclanthology.org/2023.bea-1.32 ↗
  24. S. Bannò, K. Knill, and M. Gales, “Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs,” in Proc. of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), Jul. 2025, pp. 632–646. [Online]. Available: https://aclanthology.org/anthology-files/pdf/bea/2025.bea-1.45.pdf ↗
  25. A. Katinskaia and R. Yangarber, “GPT-3.5 for grammatical error correction,” in Proc. of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024, pp. 7831–7843. [Online]. Available: https://aclanthology.org/2024.lrec-main.692 ↗
  26. K. Omelianchuk, A. Liubonko, O. Skurzhanskyi et al., “Pillars of grammatical error correction: Comprehensive inspection of contemporary approaches in the era of large language models,” in Proc. of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Jun. 2024, pp. 17–33. [Online]. Available: https://aclanthology.org/2024.bea-1.3 ↗
  27. Y. Song, K. Krishna, R. Bhatt et al., “GEE! grammar error explanation with large language models,” in Findings of the Association for Computational Linguistics: NAACL 2024, Jun. 2024, pp. 754–781. [Online]. Available: https://aclanthology.org/2024.findings-naacl.49 ↗
  28. P. Kerr, “Giving feedback to language learners.” Part of the Cambridge papers in ELT series, pp. 1–28, 2020. find
  29. K. Ehret, A. Berdicevskis, C. Bentz, and A. Blumenthal-Dramé, “Measuring language complexity: Challenges and opportunities,” Linguistics Vanguard, vol. 9, no. s1, pp. 1–8, 2023. ↗
  30. P. Foster and P. Skehan, “The influence of planning and task type on second language performance,” Studies in Second Language Acquisition, vol. 18, no. 3, p. 299–323, 1996. ↗
  31. D. Biber, B. Gray, and K. Poonpon, “Should We Use Characteristics of Conversation to Measure Grammatical Complexity in L2 Writing Development?” TESOL Quarterly, vol. 45, no. 1, pp. 5–35, 2011. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.5054/tq.2011.244483 ↗
  32. B. Bulté, A. Housen, and G. Pallotti, “Complexity and difficulty in second language acquisition: A theoretical and methodological overview,” Language Learning, vol. 75, no. 2, pp. 533–574, 2025. ↗
  33. D. Biber, B. Gray, T. Larsson, and S. Staples, “Grammatical analysis is required to describe grammatical (and “syntactic”) complexity,” Language Learning, 2025. find
  34. S. Staples, J. Egbert, D. Biber, and B. Gray, “Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre,” Written Communication, vol. 33, no. 2, pp. 149–183, 2016. [Online]. Available: https://doi.org/10.1177/0741088316631527 ↗
  35. D. Biber, B. Gray, and S. Staples, “Predicting patterns of grammatical complexity across language exam task types and proficiency levels,” Applied Linguistics, vol. 37, no. 5, pp. 639–668, 2016. find
  36. X. Lu, “Automated measurement of syntactic complexity in corpus-based L2 writing research and implications for writing assessment,” Language Testing, vol. 34, no. 4, pp. 493–511, 2017. ↗
  37. K. Kyle, “Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication,” Ph.D. dissertation, ScholarWorks@ Georgia State University, 2016. find
  38. K. Kyle and S. A. Crossley, “Measuring syntactic complexity in L2 writing using fine-grained clausal and phrasal indices,” The Modern Language Journal, vol. 102, no. 2, pp. 333–349, 2018. ↗
  39. X. Chen and D. Meurers, “CTAP: A web-based tool supporting automatic complexity analysis,” in Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), D. Brunato, F. Dell’Orletta, G. Venturi, T. François, and P. Blache, Eds. Osaka, Japan: The COLING 2016 Organizing Committee, Dec. 2016, pp. 113–119. [Online]. Available: https://aclanthology.org/W16-4113/ ↗
  40. N. Sagirov and X. Chen, “POLKE: A system for comprehensively annotating pedagogically oriented grammatical structure use in language production,” Manuscript submitted for publication to Behavior Research Methods, 2025. ↗
  41. D. Verratti-Souto, N. Sagirov, and X. Chen, “NLP-powered quantitative verification of the English Grammar Profile’s structure-level assignment,” Annual Review of Applied Linguistics, pp. 1–22, 2025. find
  42. J. Geertzen, T. Alexopoulou, and A. Korhonen, “Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database (EFCAMDAT),” in Proc. of the 31st Second Language Research Forum. Somerville: Cascadilla Proc. Project, 2013, pp. 240–254. [Online]. Available: http://www.lingref.com/cpp/slrf/2012/paper3100.pdf ↗
  43. M. Davies, “The 385+ million word Corpus of Contemporary American English (1990–2008+): Design, architecture, and linguistic insights,” International Journal of Corpus Linguistics, vol. 14, no. 2, pp. 159– 190, 2009. ↗
  44. Y. Berzak, J. Kenney, C. Spadine et al., “Universal Dependencies for learner English,” in Proc. of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug. 2016, pp. 737–746. [Online]. Available: https://aclanthology.org/P16-1070/ ↗
  45. Q. H. Châu and B. Bulté, “A comparison of automated and manual analyses of syntactic complexity in L2 English writing,” International Journal of Corpus Linguistics, vol. 28, no. 2, pp. 232–262, 2023. ↗
  46. S. Kim, “Generalizability of CEFR Criterial Grammatical Features in a Korean EFL Corpus across A1, A2, B1, and B2 Levels,” Language Assessment Quarterly, vol. 18, no. 3, pp. 273–295, 2021. [Online]. Available: https://doi.org/10.1080/15434303.2020.1855647 ↗
  47. L. Weissweiler, V. Hofmann, A. Köksal, and H. Schütze, “The better your syntax, the better your semantics? probing pretrained language models for the English comparative correlative,” in Proc. of the 2022 Conference on Empirical Methods in Natural Language Processing, Dec. 2022, pp. 10 859–10 882. [Online]. Available: https://aclanthology.org/2022.emnlp-main.746/ ↗
  48. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423 ↗
  49. Y. Liu, M. Ott, N. Goyal et al., “RoBERTa: A robustly optimized BERT pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692 ↗
  50. P. He, X. Liu, J. Gao, and W. Chen, “DeBERTA: Decoding-enhanced BERT with disentangled attention,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=XPZIaotutsD ↗
  51. Y. Okano, K. Funakoshi, R. Nagata, and M. Okumura, “Generating dialog responses with specified grammatical items for second language learning,” in Proc. of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), 2023, pp. 184–194. find
  52. K. Chujo, K. Oghigian, and S. Akasegawa, “A corpus and grammatical browsing system for remedial EFL learners,” in Multiple Affordances of Language Corpora for Data-driven Learning. Amsterdam: John Benjamins Publishing Company, 2015, pp. 109–128. ↗
  53. D. Glandorf and D. Meurers, “Towards fine-grained pedagogical control over English grammar complexity in educational text generation,” in Proc. of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Jun. 2024, pp. 299–308. [Online]. Available: https://aclanthology.org/2024.bea-1.24 ↗
  54. Y. Ide, Y. Nishida, J. Vasselli et al., “How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments,” in Proc. of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025, pp. 7416–7432. find
  55. M. Kennedy, “Evidence of generative syntax in LLMs,” in Proc. of the 29th Conference on Computational Natural Language Learning, G. Boleda and M. Roth, Eds., Jul. 2025, pp. 377–396. [Online]. Available: https://aclanthology.org/2025.conll-1.25/ ↗
  56. J. Cheng and H. Amiri, “Linguistic blind spots of large language models,” in Proc. of the Workshop on Cognitive Modeling and Computational Linguistics, May 2025, pp. 1–17. [Online]. Available: https://aclanthology.org/2025.cmcl-1.3/ ↗
  57. T. Brown, B. Mann, N. Ryder et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020. ↗
  58. S. Sravanthi, M. Doshi, P. Tankala et al., “PUB: A pragmatics understanding benchmark for assessing LLMs’ pragmatics capabilities,” in Findings of the Association for Computational Linguistics: ACL 2024, Aug. 2024, pp. 12 075–12 097. [Online]. Available: https://aclanthology.org/2024.findings-acl.719/ — PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities ↗
  59. H. Yannakoudakis, Ø. E. Andersen, A. Geranpayeh et al., “Developing an automated writing placement system for ESL learners,” Applied Measurement in Education, vol. 31, no. 3, pp. 251–267, 2018. ↗
  60. D. Nicholls, A. Caines, and P. Buttery, “The Write & Improve Corpus 2024: Error-annotated and CEFR-labelled essays by learners of English,” 2024. [Online]. Available: https://doi.org/10.17863/CAM.112997 ↗
  61. Y. Berzak, J. Kenney, C. Spadine, J. X. Wang, L. Lam, K. S. Mori, S. Garza, and B. Katz, “Universal Dependencies for learner English,” in Proc. of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), K. Erk and N. A. Smith, Eds. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 737–746. [Online]. Available: https://aclanthology.org/P16-1070/ ↗
  62. C. Bryant, M. Felice, Ø. E. Andersen, and T. Briscoe, “The BEA-2019 shared task on grammatical error correction,” in Proc. of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, Aug. 2019, pp. 52–75.
  63. G. Abercrombie, T. Dinkar, A. Cercas Curry, V. Rieser, and D. Hovy, “Consistency is key: Disentangling label variation in natural language processing with intra-annotator agreement,” in Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP, G. Abercrombie, V. Basile, S. Frenda, S. Tonelli, and S. Dudy, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 63–74. [Online]. Available: https://aclanthology.org/2025.nlperspectives-1.6/ ↗
  64. D. Ferrucci and A. Lally, “UIMA: An architectural approach to unstructured information processing in the corporate research environment,” Natural Language Engineering, vol. 10, no. 3-4, pp. 327–348, 2004. ↗
  65. P. Kluegl, M. Toepfer, P.-D. Beck et al., “UIMA Ruta: Rapid development of rule-based information extraction applications,” Natural Language Engineering, vol. 22, no. 1, pp. 1–40, 2016. find
  66. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024. ↗
  67. OpenAI, “GPT-4 Technical Report,” 2023.
  68. K. Omelianchuk, V. Atrasevych, A. Chernodub, and O. Skurzhanskyi, “GECToR – grammatical error correction: Tag, not rewrite,” in Proc. of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Jul. 2020, pp. 163–170. [Online]. Available: https://aclanthology.org/2020.bea-1.16 ↗
  69. H. T. Ng, S. M. Wu, T. Briscoe et al., “The CoNLL-2014 shared task on grammatical error correction,” in Proc. of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, Jun. 2014, pp. 1–14. find
  70. C. Bryant, M. Felice, and T. Briscoe, “Automatic annotation and evaluation of error types for grammatical error correction,” in Proc. of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul. 2017, pp. 793–805.
  71. Llama Team, “The Llama 3 herd of models,” 2024. [Online]. Available: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/ ↗
  72. J. C. Richards, Moving beyond the plateau. Cambridge University Press New York, NY, 2008. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banno-2026-exploiting-english-grammar-profile-l2/