Elamur Research⌕
☰
home / papers / 2025 / Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing

Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing

Paper · 2025 · Timur Jaganov, John Blake, Julian Villegas et al.

Summary

The paper asks whether large language models can make Dynamic Assessment (DA) of learner grammar practical for whole classes. In DA, a teacher responds to a learner's error with hints that start vague and grow more explicit until the learner fixes it. This works well one-on-one but is too labour-intensive for large groups. Existing correction tools such as Grammarly or Criterion repair text but do not offer graduated help, and earlier computerized DA systems mostly covered reading and listening rather than free writing. The authors built DynaWrite, a modular, microservices-based tutoring application that can plug in many LLMs, either through the OpenAI API or locally via Ollama, to produce stepwise grammar hints. A screening of 21 LLMs narrowed the field to GPT-4o and neural chat. Tested on a set of 200 learner sentences, the two finalists found errors about equally well. GPT-4o, however, gave clearer, more consistent and better-graduated hints, and it was also fast and stable enough for real-time use. The work suggests that LLM-driven DA could reach far more learners than teacher-led sessions.

Key points

  • DynaWrite is a microservices-based grammar tutor that supports several LLMs, including GPT-4o, Llama3.2 and Vicuna. It is available at dynawrite.org.
  • An initial screening covered 21 LLMs. GPT-4o and neural chat came out as the most promising for scaling DA.
  • The two finalists performed similarly at spotting grammatical errors. GPT-4o was consistently better at producing progressively explicit, coherent hints.
  • The feedback scale follows a four-step ladder:
    1. ask whether the sentence contains a problem;
    2. repeat the phrase that holds the error;
    3. point to the wrong word(s);
    4. give the correct form and explain the error.
  • The evaluation set had 200 learner sentences, such as "Teens play online games best than others." Performance tests reported response-time percentiles: gpt-4o-mini was more predictable than o1-preview.

Interesting ideas

  • I1. Feedback is a ladder of increasing explicitness, and the rung a learner needs before succeeding works as a diagnostic signal. This tells you what is being learned, not only what is already known.
  • I2. Detecting an error and the quality of the help given about it are scored separately. Two models can tie on detection yet differ sharply on hint quality.
  • I3. A model-agnostic backend lets the same task and prompts be swapped across many LLMs, both cloud and local, for a fair broad screening before a deep comparison of finalists.
  • I4. Latency is reported as percentiles (25th/50th/75th) rather than a single average, because predictability matters for interactive use.
  • I5. A small fixed set of authentic learner sentences with typical L2 errors serves as a reusable test bed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course tests could grade students by how many explainability levels they need, from a vague nudge up to naming the exact token and rule. Mova's step-by-step traces naturally supply each rung.
  • I2 (learn): When ranking LLM teacher tiers, judge "found the problem" and "explained it usefully" as separate metrics. A tier that only labels correctly should not be trusted to write repair rules.
  • I3 (learn): A uniform witness or teacher interface would let new LLMs or parsers be screened cheaply on a fixed set before entering the silver-data agreement pool.
  • I4 (tree): Benchmark the perceptron parser and gparse with latency percentiles to expose tail behaviour that matters for interactive or batch pipelines.
  • I5 (tokens): Collect learner-error sentences as a robustness test for the tagger and lemmatizer, checking that errors like "best than" get sensible tags rather than breaking downstream analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AizuFukushima · Japan2 works · 8 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Timur Jaganov, John Blake, Julian Villegas and Nicholas Carr. 2025. Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing. arXiv:2505.00931 · IEEE ACCESS, 2025, Volume 13, pp. 151538-151550.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/access.2025.3603191 · arXiv 2505.00931

@misc{jaganov2025large,
  title = {{Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing}},
  author = {Timur Jaganov, John Blake, Julian Villegas and Nicholas Carr},
  year = {2025},
  doi = {10.1109/access.2025.3603191},
  eprint = {2505.00931},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/access.2025.3603191}
}

References

43 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew E Poehner. Probing and provoking L2 development: The object of mediation in dynamic assessment and mediated development. In The Routledge Handbook of Sociocultural Theory and Second Language Development, pages 249–265. Routledge, 2018. find
  2. Ali Aljaafreh and James P Lantolf. Negative feedback as regulation and second language learning in the zone of proximal development. The Modern Language Journal, 78(4):465–483, 1994. ↗
  3. John Bitchener and Neomy Storch. Written corrective feedback for L2 development, volume 96. Multilingual Matters, 2016. find
  4. Qiandi Liu and Dan Brown. Methodological synthesis of research on the effectiveness of corrective feedback in L2 writing. Journal of Second Language Writing, 30:66–81, 2015. ↗
  5. Abdulaziz Alshahrani and Neomy Storch. Investigating teachers’ written corrective feedback practices in a Saudi EFL context: How do they align with their beliefs, institutional guidelines, and students’ preferences? Australian Review of Applied Linguistics, 37(2):101–122, 2014. ↗
  6. Icy Lee. Error correction in L2 secondary writing classrooms: The case of Hong Kong. Journal of Second Language Writing, 13(4):285–312, 2004. ↗
  7. Matthew E Poehner. Group dynamic assessment: Mediation for the L2 classroom. TESOL Quarterly, 43(3):471–491, 2009. ↗
  8. Hossein Nassaji and Merrill Swain. A Vygotskian perspective on corrective feedback in L2: The effect of random versus negotiated help on the learning of English articles. Language awareness, 9(1):34–51, 2000. ↗
  9. Ehsan Rassaei. Effects of dynamic and non-dynamic corrective feedback on EFL writing accuracy during dyadic and small group interactions. International Review of Applied Linguistics in Language Teaching, 59(2):233–265, 2021. find
  10. Rosemary Erlam, Rod Ellis, and Rob Batstone. Oral corrective feedback on L2 writing: Two approaches compared. System, 41(2):257–268, 2013. ↗
  11. Lev S Vygotsky. The collected works of LS Vygotsky: Volume 1: Problems of general psychology, including the volume Thinking and Speech, volume 1. Springer Science & Business Media, 1987. find
  12. James P Lantolf and Matthew E Poehner. Sociocultural theory and the pedagogical imperative in L2 education: Vy-gotskia praxis and the research/practice divide. Routledge, 2014. ↗
  13. Lev S Vygotsky. The Collected Works of L. S. Vygotsky: Volume 5, Child Psychology. Plenum, 1998. find
  14. Lev S Vygotsky. The collected works of LS Vygotsky: Volume 5: Child psychology, volume 5. Plenum, 1998. find
  15. Lev S Vygotsky. Thought and language. MIT Press, 2012. find
  16. Tina S. Randall and Kimberly Urbanski. Development of a computerized dynamic assessment program for second language grammar instruction and assessment. Language and Sociocultural Theory, 10(1):50–81, 2023. ↗
  17. Jill Burstein, Martin Chodorow, and Claudia Leacock. Automated essay evaluation: The criterion online writing service. AI magazine, 25(3):27–27, 2004. ↗
  18. Claudia Leacock, Michael Gamon, and Chris Brockett. User input and interactions on microsoft research ESL assistant. In Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications, pages 73–81, 2009. find
  19. Linqian Ding and Di Zou. Automated writing evaluation systems: A systematic review of grammarly, pigai, and criterion with a perspective on future directions in the age of generative artificial intelligence. Education and Information Technologies, 29(11):14151–14203, 2024. ↗
  20. Carol A. Chapelle, Elena Cotos, and Jooyoung Lee. Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32(3):385–405, 2015. ↗
  21. Evgeny Chukharev-Hudilainen. Empowering automated writing evaluation with keystroke logging. In Observing writing, pages 125–142. Brill, 2019. ↗
  22. Hui-Hsien Feng, Aysel Saricaoglu, and Evgeny Chukharev-Hudilainen. Automated error detection for developing grammar proficiency of ESL learners. Calico Journal, 33(1):49– 70, 2016. ↗
  23. Haiyang Ai. Providing graduated corrective feedback in an intelligent computer-assisted language learning environment. ReCALL, 29(3):313–334, 2017. ↗
  24. Jie Zhang and Xiaofei Lu. Measuring and supporting second language development using computerized dynamic assessment. Language and Sociocultural Theory, 6(1):92– 115, 2019. ↗
  25. Dmitri Leontjev. Dynamic assessment of word derivational knowledge. Eesti Rakenduslingvistika Ühingu aastaraamat, (12):141–160, 2016. find
  26. Ali Mehri Kamrood, Mohammad Davoudi, Saeed Ghaniabadi, and Seyyed Mohammad Reza Amirian. Diagnosing L2 learners’ development through online computerized dynamic assessment. Computer Assisted Language Learning, 34(7):868–897, 2021. find
  27. Matthew E Poehner and James P Lantolf. Bringing the zpd into the equation: Capturing L2 development during computerized dynamic assessment (C-DA). Language Teaching Research, 17(3):323–342, 2013. ↗
  28. Matthew E Poehner, Jie Zhang, and Xiaofei Lu. Computerized dynamic assessment (C-DA): Diagnosing L2 development according to learner responsiveness to mediation. Language testing, 32(3):337–357, 2015. ↗
  29. Yanfeng Yang and David D Qian. Assessing English reading comprehension by Chinese EFL learners in computerized dynamic assessment. Language Testing in Asia, 7:1–15, 2017. ↗
  30. John Blake. Genre-specific error detection with multimodal feedback. RELC Journal, 51(1):179–187, 2020. ↗
  31. Allan Nicholas, John Blake, Jeremy Perkins, and Maxim Mozgovoy. Evaluating the effectiveness of a computerised dynamic assessment of L2 English email requests. Computer Assisted Language Learning, pages 1–33, 2024. ↗
  32. Allan Nicholas and John Blake. Profiling learner development through the application of a computerized dynamic assessment methodology to a Japanese learner’s L2 English email writing. Research Methods in Applied Linguistics, 3(3):100164, 2024. find
  33. John Blake, Maxim Mozgovoy, and Allan Nicholas. Computerized dynamic assessment for pragmatic competence in second language learners’ english email requests. In International Conference in Methodologies and intelligent Systems for Techhnology Enhanced Learning, pages 17–28. Springer, 2025. find
  34. John Blake. Unleashing the potential: Positive impacts of generative AI on learning and teaching. In Generative AI in Teaching and Learning, pages 31–45. IGI Global, 2024. find
  35. OpenAI Team. ChatGPT: Optimizing language models for dialogue, 2022. Accessed on 31 March 2025. find
  36. Wan Chong Choi and Iek Chong Choi. Exploring the impact of Code.org’s block-based coding curriculum on student motivation in K-12 education. In 2024 12th International Conference on Information and Education Technology (ICIET), pages 93–97. IEEE, 2024. find
  37. Wesley Morris, Langdon Holmes, Joon Suh Choi, and Scott Crossley. Automated scoring of constructed response items in math assessment using large language models. International Journal of Artificial Intelligence in Education, pages 1–28, 2024. ↗
  38. Evan You et al. Vue.js: The progressive javascript framework. https://vuejs.org/, 2025. Accessed: 2025-04-21. ↗
  39. Chen Ma and Yuhong Chi. Evaluation test and improvement of load balancing algorithms of nginx. Ieee Access, 10:14311–14324, 2022. ↗
  40. Apache Software Foundation. Apache kafka: A distributed event streaming platform. https://kafka.apache.org/, 2025. Accessed: 2025-04-21. ↗
  41. Red Hat. Keycloak: Open source identity and access management. https://www.keycloak.org/, 2025. Accessed: 2025-04-21. ↗
  42. OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/, 2024. Accessed: 2025-04-21. ↗
  43. Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/, 2024. Accessed: 2025-04-21. TIMUR JAGANOV is pursuing a Master’s in Information Technology and Project Management at the University of Aizu, Japan. He holds a Bachelor’s in Computer Science from the Techno-Economic Academy of Film and Television, Kazakhstan. With extensive experience in software development, he has worked on designing scalable systems, implementing cloud-based solutions, and developing intelligent applications. His research uses modern computational methods to enhance digital learning environments and optimize user interaction. He is particularly interested in information systems, learning assistants, and distributed architectures. Currently, he is conducting research in the School of Computer Science and Engineering at the University of Aizu, exploring innovative approaches to interactive education and knowledge management. JOHN BLAKE received his Ph.D. in Applied Linguistics from the University of Aston, Birmingham, UK. He holds multiple postgraduate degrees, including an M.Sc. in Computer Science, an M.B.A., an M.Ed. in Applied Linguistics, an M.A. in Creative Writing, and an M.A. in Mathematics Education. He is a Chartered IT Professional and a professional member of the British Computer Society. His research primarily utilizes corpus linguistics to analyze texts and computational linguistics to develop pattern-searching tools and pipelines. Currently, he is a Professor in the School of Computer Science and Engineering at the University of Aizu, Japan. He also serves as an academic editor for PLOS One and reviews for multiple journals in the fields of natural language processing, linguistics and education. He has authored or co-authored over 80 papers. JULIÁN VILLEGAS Senior Associate Professor at the Graduate School of Computer Science and Engineering, University of Aizu, Japan, with interests in speech intelligibility, music, and spatial sound. NICHOLAS CARR received his Ph.D in Education (TESOL) from Deakin University, Melbourne, Australia. He holds a Master of TESOL from Deakin University, a Diploma of Music Performance from Box Hill Institute, and Bachelor of Business from Swinburne University of Technology. His research interests include cognitive linguistics, concept-based-language instruction, sociocultural theory and computer assisted language learning. He currently serves as an Associate Professor in the School of Computer Science and Engineering at the University of Aizu, Japan. He also acts as a reviewer for multiple journals in the field of additional language learning and education. He has been published in several journals and edited volumes. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jaganov-2025-large-language-model-driven-dynamic/