Elamur Research⌕
☰
home / papers / 2024 / Large Language Models For Second Language English Writing Assessments: An Exploratory Comparison

Large Language Models For Second Language English Writing Assessments: An Exploratory Comparison

Paper · 2024 · Zhuang Qiu, Peizhi Yan and Zhenguang Cai

Summary

The paper tests how well current large language models can grade second-language English essays. It uses IELTS writing task 2 as the testbed and official examiner bands as ground truth. Two exploratory studies are run. In the first, RoBERTa is fine-tuned as a classifier over 19 half-band classes on two public collections: a Kaggle set (about 700 task 2 essays) and a much larger HuggingFace set (10,324 essays). In the second, GPT-4o and Llama-3-70B-Instruct score 400 essays without any training. They are guided only by prompts built from the official band descriptors, with or without sample essays at several band levels. Agreement with human scores is measured by Pearson correlation and RMSE. Fully fine-tuned RoBERTa and prompted GPT-4o both reach correlations above 0.7. Llama-3 trails and actually improves when the sample essays are dropped. The study shows that both fine-tuning and prompting are workable routes to automated essay scoring. It also deliberately avoids training on machine-generated sub-scores, so its evaluation stays tied to real examiner judgments.

Key points

  • Data: Kaggle IELTS set, with task 2 split 7:3 into 497 train and 214 test essays. HuggingFace set (chillies/IELTS-writing-task-2-evaluation), split 9:1 into 9,291 train and 1,033 test essays. Task 1 was excluded because it depends on charts and tables.
  • RoBERTa setup: a two-layer head on the [CLS] vector (768-unit tanh layer, then softmax over 19 classes from 0 to 9 in half steps). Training used cross-entropy, Adam at 2×10⁻⁵ with the rate cut 20% per epoch, 20 epochs, batch size 16, on one A6000 (~12.7 GB). Runs took about 4 and 74 minutes.
  • Training only the head versus all parameters (correlation / RMSE):
    • Kaggle: 0.651/0.830 for the head only, 0.731/0.784 for all parameters.
    • HuggingFace: 0.707/0.757 for the head only, 0.735/0.770 for all parameters. RMSE got slightly worse here despite the higher correlation.
  • Prompted scoring (correlation / RMSE):
    • GPT-4o: 0.71/1.05 with example essays, 0.72/1.13 without.
    • Llama-3-70B-Instruct: 0.56/1.25 with examples, 0.63/0.99 without.
  • The authors chose RMSE as the main metric because a value under 1 means the model is off by less than one band on average. They admit this makes comparison with QWK-based studies hard.

Interesting ideas

  • I1. Treat an ordinal grade as a classification over a fixed set of discrete steps (here 19 half-bands), so the model never outputs off-scale values; the price is that it ignores the ordering of the classes.
  • I2. Run a cheap ablation of freezing the encoder and training only the head versus full fine-tuning; it shows how much the task needs new representations rather than a new readout.
  • I3. Adding example essays to a prompt is not automatically helpful: it left the stronger model unchanged and hurt the weaker one, so few-shot exemplars must be tested as a variable per model.
  • I4. Keep the benchmark anchored only to genuine human labels and refuse to train on scores another model produced, to avoid a circular loop of models grading models.
  • I5. Validate generated outputs against the legal range, and re-query (up to three attempts) when an answer is missing or invalid, instead of silently accepting it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Graded answers to school-course tests could be predicted as discrete rubric levels, with a separate check that confusions land on neighbouring levels.
  • I2 (tree): Before retraining gparse fully for a new treebank or UD dialect, a frozen-encoder baseline would show whether a new arc/label head alone recovers most of the LAS.
  • I3 (learn): LLM teacher tiers should be evaluated with and without in-prompt exemplars, separately for each teacher, before their silver labels are trusted.
  • I4 (learn): Silver data agreed by witness parsers should always be checked against a held-out slice of human-annotated gold, so agreement among models is never mistaken for correctness.
  • I5 (learn): LLM teacher outputs (frames, roles, labels) could pass a schema/range check with bounded retries, logging every rejection as an explainable record.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Zhuang Qiu, Peizhi Yan and Zhenguang Cai. 2024. Large Language Models For Second Language English Writing Assessments: An Exploratory Comparison. Proceedings of the 38th Pacific Asia Conference on Language, Information and Computation.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{qiu2024large,
  title = {{Large Language Models For Second Language English Writing Assessments: An Exploratory Comparison}},
  author = {Zhuang Qiu, Peizhi Yan and Zhenguang Cai},
  booktitle = {Proceedings of the 38th Pacific Asia Conference on Language, Information and Computation},
  year = {2024},
  url = {https://aclanthology.org/2024.paclic-1.36/}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, and D. Amodei. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. ↗
  2. Chunkit Chan, Cheng Jiayang, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2024. Exploring the potential of ChatGPT on sentence level relations: A focus on temporal, causal, and discourse relations. In Findings of the Association for Computational Linguistics: EACL 2024, pages 684–721, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  3. Bill Cope, Mary Kalantzis, and Duane Searsmith. 2021. Artificial intelligence for education: Knowledge and its assessment in ai-enabled learning ecologies. Educational philosophy and theory, 53(12):1229–1245. ↗
  4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  5. W. Jiao, W. Wang, J. T. Huang, X. Wang, and Z. Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745. ↗
  6. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274. ↗
  7. Paraskevas Lagakis and Stavros Demetriadis. 2021. Automated essay scoring: A review of the field. In 2021 International Conference on Computer, Information and Telecommunication Systems (CITS), pages 1–6. IEEE. find
  8. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023a. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439. ↗
  9. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023b. Gpt understands, too. AI Open. ↗
  10. Y Liu. 2019. Roberta: A robustly optimized bert pre-training approach. arXiv preprint arXiv:1907.11692. ↗
  11. Watheq Mansour, Salam Albatarni, Sohaila Eltanbouly, and Tamer Elsayed. 2024. Can large language models automatically score proficiency of written essays? arXiv preprint arXiv:2403.06149. ↗
  12. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. ↗
  13. OpenAI. 2023. Chatgpt. https://www.openai.com/ chatgpt. Accessed: 2024-08-27. ↗
  14. OpenAI. 2024. Hello gpt-4o. https://openai.com/ index/hello-gpt-4o/. Accessed: 2024-08-27. ↗
  15. M. Ortega-Martín, Ó. García-Sierra, A. Ardoiz, J. Álvarez, J. C. Armenteros, and A. Alonso. 2023. Linguistic ambiguity analysis in chatgpt. arXiv preprint arXiv:2302.06426. ↗
  16. Ellis B Page. 1966. The imminence of grading essays by computer. The Phi Delta Kappan, 47(5):238–243. find
  17. Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55(3):2495–2527. ↗
  18. Kevin Schoepp. 2018. Predictive validity of the ielts in an english as a medium of instruction environment. Higher Education Quarterly, 72(4):271–285. ↗
  19. MD Shermis. 2003. Automated essay scoring: A cross-disciplinary perspective. find
  20. Kun Sun and Rong Wang. 2024. Automatic essay multidimensional scoring with fine-tuning and multiple regression. arXiv preprint arXiv:2406.01198. ↗
  21. Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022. Automated essay scoring via pairwise contrastive regression. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2724–2733. find
  22. Olaf Zawacki-Richter, Victoria I Marín, Melissa Bond, and Franziska Gouverneur. 2019. Systematic review of research on artificial intelligence applications in higher education–where are the educators? International Journal of Educational Technology in Higher Education, 16(1):1–27. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qiu-2024-large-language-models-second-language/