Elamur Research⌕
☰
home / papers / 2025 / LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing

LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing

Paper · 2025 · Zhengxiang Wang, Veronika Makarova, Zhi Li et al.

Summary

The paper asks whether LLMs can perform multi-dimensional analytic writing assessment: giving a score and a comment for each of several criteria, rather than one holistic grade. The authors release a collection of English literature reviews produced by graduate students writing in a second language. Human experts assessed each review against 9 criteria, with a 10-point score and an optional comment for each. Several popular LLMs were prompted to do the same task under different conditions, and their outputs were compared with the human assessments. Comment quality is hard to judge by hand, so the authors also introduce ProEval. It is an LLM-driven pipeline that breaks a comment down step by step and ends with classification questions. This makes comment evaluation interpretable, cheap, scalable and reproducible. The main finding is that LLM assessments are reasonably good and generally reliable, though the authors do not argue for replacing human assessors. The work matters because analytic feedback is valuable for L2 learners but scarce and costly to produce.

Key points

  • Corpus size and topics: 141 literature reviews by 51 L2 graduate students, averaging 1321 words (930 without references). They cover five humanities and social science topics, and two of those topics were written collaboratively.
  • Assessment criteria: C1–C9 cover material selection, integration and citation, key components, structure logic, content and clarity, coherence, cohesion, grammar, and academic vocabulary. C2, C8 and C9 are treated as the more objective, rule-governed criteria.
  • Human assessors: 94.3% of essays had three independent assessors, drawn from six people (graduate RAs and faculty). In a student survey, 80% of 30 respondents rated the feedback "very useful".
  • No contamination risk: the corpus predates ChatGPT and was never published before, so it contains no LLM text and no models could have seen it.
  • ProEval validation: an LLM performs every step of the pipeline. It was checked using inter-annotator agreement and exact match rate between two human annotators.

Interesting ideas

  • I1. ProEval replaces a single Likert-style quality rating with a chain of small, checkable classification decisions about a comment, which makes judging open-ended text fine-grained and auditable.
  • I2. Splitting criteria into objective ones (citation, grammar, vocabulary) and interpretive ones lets you expect different levels of agreement and read results per criterion instead of as one average.
  • I3. Choosing an evaluation corpus that was created before LLMs and never released guarantees a clean test free of memorization.
  • I4. Testing reliability by changing only one factor at a time (prompt, condition) isolates what drives output instability, though the authors admit this design is limited.
  • I5. Pairing each score with a criterion-specific comment ties the number to a stated reason.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Explanations or answers produced by Mova or by an LLM teacher could be graded with the same chain of narrow questions, so that every evaluation verdict can be traced.
  • I2 (learn): Silver-data agreement between witness parsers could be reported separately for rule-governed relations and judgment-heavy ones, so disagreement is not misread as noise.
  • I3 (answer): School-test and fable-QA benchmarks could include privately written, never-published items as a negative control against LLM-teacher leakage.
  • I4 (learn): Reruns of teacher tiers could vary one setting per experiment to locate the source of instability in silver labels.
  • I5 (back): Generated answers could attach a short textual justification per dimension, produced from the meaning graph, so scores never stand alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stony Brook UniversityStony Brook · United States43 works · 2,571 citations🎓University of SaskatchewanSaskatoon · Canada2 works · 5 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as Zhengxiang Wang, Veronika Makarova, Zhi Li, Jordan Kodner and Owen Rambow. 2025. LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.423

@inproceedings{wang2025llms,
  title = {{LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing}},
  author = {Zhengxiang Wang, Veronika Makarova, Zhi Li, Jordan Kodner and Owen Rambow},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.423},
  url = {https://doi.org/10.18653/v1/2025.acl-long.423}
}

References

54 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153–162, Vancouver, Canada. Association for Computational Linguistics.
  2. Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 715–725, Berlin, Germany. Association for Computational Linguistics.
  3. Nikolay Babakov, Maria Lysyuk, Alexander Shvets, Lilya Kazakova, and Alexander Panchenko. 2023. Error syntax aware augmentation of feedback comment generation dataset. In Proceedings of the 16th International Natural Language Generation Conference: Generation Challenges, pages 37–44, Prague, Czechia. Association for Computational Linguistics. find
  4. Stefano Banno, Hari Krishna Vydana, Kate Knill, and Mark Gales. 2024. Can GPT-4 do L2 analytic assessment? In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 149–164, Mexico City, Mexico. Association for Computational Linguistics. ↗
  5. Shabnam Behzad, Omid Kashefi, and Swapna Somasundaran. 2024a. Assessing online writing feedback resources: Generative AI vs. good samaritans. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1638– 1644, Torino, Italia. ELRA and ICCL. find
  6. Shabnam Behzad, Omid Kashefi, and Swapna Somasundaran. 2024b. LEAF: Language learners’ English essays and feedback corpus. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 433–442, Mexico City, Mexico. Association for Computational Linguistics.
  7. Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Sshubam Verma, and Mitesh M Khapra. 2024. Finding blind spots in evaluator LLMs with interpretable checklists. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16279–16309, Miami, Florida, USA. Association for Computational Linguistics. find
  8. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. ↗
  9. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, and 1118 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530. ↗
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  11. Kai Guo and Deliang Wang. 2024. To resist it or to embrace it? examining ChatGPT’s potential to support teacher feedback in EFL writing. Educ. Inf. Technol., 29(7):8435–8463. ↗
  12. Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. Toefl11: A corpus of non-native english. ETS Research Report Series, 2013(2):i–15. ↗
  13. Jieun Han, Haneul Yoo, Junho Myung, Minsun Kim, Hyunseung Lim, Yoonsu Kim, Tak Yeon Lee, Hwajung Hong, Juho Kim, So-Yeon Ahn, and Alice Oh. 2024. LLM-as-a-tutor in EFL writing education: Focusing on evaluation of student-LLM interaction. In Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U), pages 284–293, Miami, Florida, USA. Association for Computational Linguistics. ↗
  14. Hongwen Cai. 2015. Weight-based classification of raters and rater cognition in an efl speaking test. Language Assessment Quarterly, 12(3):262–282. ↗
  15. Wen-Bin Han, Jhih-Jie Chen, Chingyu Yang, and Jason Chang. 2019. Level-up: Learning to improve proficiency level of essays. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 207–212, Florence, Italy. Association for Computational Linguistics. find
  16. Nicky Hockly. 2019. Automated writing evaluation. ELT Journal, 73:82–88. ↗
  17. Mohamed Abdellatif Hussein, Hesham Hassan, and Mohammad Nassef. 2019. Automated language essay scoring systems: a literature review. PeerJ Computer Science, 5:e208. ↗
  18. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5075–5084, Singapore. Association for Computational Linguistics. find
  19. You-Jin Jong, Yong-Jin Kim, and Ok-Chol Ri. 2023. Review of feedback in automated essay scoring. Preprint, arXiv:2307.05553. ↗
  20. Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-2019, page 6300–6308. International Joint Conferences on Artificial Intelligence Organization. find
  21. Paraskevas Lagakis and Stavros Demetriadis. 2021. Automated essay scoring: A review of the field. In 2021 International Conference on Computer, Information and Telecommunication Systems (CITS), pages 1–6. find
  22. Zhi Li, Veronika Makarova, and Zhengxiang Wang. 2023a. Assessment of academic esl writing in an online tutorial for graduate students. In 22nd European Conference on e-Learning: ECEL 2023. Academic Conferences and publishing limited. find
  23. Zhi Li, Veronika Makarova, and Zhengxiang Wang. 2023b. Developing literature review writing and citation practices through an online writing tutorial series: Corpus-based evidence. Frontiers in Communication, 8:1035394. ↗
  24. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  25. Veronika Makarova, Zhi Li, and Zhengxiang Wang. 2024. Can chatgpt grade non-native academic english writing? In AI Approaches to Literacy in Higher Education, pages 97–116. IGI Global. ↗
  26. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. ↗
  27. Atsushi Mizumoto, Natsuko Shintani, Miyuki Sasaki, and Mark Feng Teng. 2024. Testing the viability of chatgpt as a companion in l2 writing accuracy assessment. Research Methods in Applied Linguistics, 3(2):100116. ↗
  28. Ryo Nagata. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3206–3215, Hong Kong, China. Association for Computational Linguistics. find
  29. Ben Naismith, Phoebe Mulcaire, and Jill Burstein. 2023. Automated evaluation of written discourse coherence using GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394–403, Toronto, Canada. Association for Computational Linguistics.
  30. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024a. Gpt-4o system card. Preprint, arXiv:2410.21276. ↗
  31. OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, and 244 others. 2024b. Openai o1 system card. Preprint, arXiv:2412.16720. ↗
  32. OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. ↗
  33. Preprint, Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  34. Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243. find
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗
  36. Pedro Uria Rodriguez, Amir Jafari, and Christopher M. Ormerod. 2019. Language models and automated essay scoring. Preprint, arXiv:1909.09482. ↗
  37. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore. Association for Computational Linguistics.
  38. Mark D Shermis and Jill C Burstein, editors. 2013. Handbook of automated essay evaluation. Routledge, London, England. find
  39. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, Kevin Gimpel, and Mohit Iyyer. 2024. GEE! grammar error explanation with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 754–781, Mexico City, Mexico. Association for Computational Linguistics. find
  40. Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024. Exploring LLM prompting strategies for joint essay scoring and feedback generation. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 283–298, Mexico City, Mexico. Association for Computational Linguistics. ↗
  41. Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882–1891, Austin, Texas. Association for Computational Linguistics.
  42. Masaki Uto. 2021. A review of deep-neural automated essay scoring models. Behaviormetrika, 48(2):459–484. ↗
  43. Bertus van Rooy. 2015. Annotating learner corpora, page 79–106. Cambridge Handbooks in Language and Linguistics. Cambridge University Press. find
  44. Qiao Wang and John Maurice Gayed. 2024. Effectiveness of large language models in automated evaluation of argumentative essays: finetuning vs. zero-shot prompting. Computer Assisted Language Learning, page 1–29. ↗
  45. Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022. Automated essay scoring via pairwise contrastive regression. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2724–2733, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  46. Kevin P. Yancey, Geoffrey Laflair, Anthony Verardi, and Jill Burstein. 2023. Rating short L2 essays on the CEFR scale with GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 576– 584, Toronto, Canada. Association for Computational Linguistics. find
  47. Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569, Online. Association for Computational Linguistics. find
  48. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.
  49. Helen Yannakoudakis and Ronan Cummins. 2015. Evaluating the performance of automated text scoring systems. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 213–223, Denver, Colorado. Association for Computational Linguistics. find
  50. Helen Yannakoudakis, Øistein E Andersen, Ardeshir Geranpayeh, Ted Briscoe, and Diane Nicholls. 2018. Developing an automated writing placement system for esl learners. Applied Measurement in Education, 31(3):251–267. ↗
  51. Fatih Yavuz, Özgür undefinedelik, and Gamze Yavaş undefinedelik. 2024. Utilizing large language models for efl essay grading: An examination of reliability and validity in rubric-based assessments. British Journal of Educational Technology. find
  52. Ruofei Zhang and Di Zou. 2020. Types, purposes, and effectiveness of state-of-the-art technologies for second and foreign language learning. Computer Assisted Language Learning, 35(4):696–742. ↗
  53. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. ↗
  54. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2025-llms-can-perform-multi-dimensional/