Elamur Research⌕
☰
home / papers / 2026 / S-GRADES – Studying Generalization of Student Response Assessments in Diverse Evaluative Settings

S-GRADES – Studying Generalization of Student Response Assessments in Diverse Evaluative Settings

Summary

Automated Essay Scoring (AES) and Automatic Short Answer Grading (ASAG) both aim to automate the grading of free-text student work. Yet they have grown apart, with separate datasets, metrics and communities, and no shared blind-test evaluation. The authors present S-GRADES, an open-source, extensible benchmark with a web platform. It brings 14 English grading datasets into one standardized format: a common schema of ID, response and score, stratified splits, and each dataset's original score scale.

Users download training and validation data, train locally, and upload test predictions. The test labels are hidden. The server validates each submission, computes metrics per dataset and updates a leaderboard.

To show what the benchmark can do, the authors test three LLMs with six prompting setups built from inductive, deductive and abductive reasoning. They also study how stable exemplar selection is and how exemplars transfer between datasets. The results show three main patterns:

  • Short-answer grading is harder than essay scoring.
  • Transfer is often unstable.
  • Moving essay exemplars to short-answer tasks hurts performance.

This makes S-GRADES a useful shared testbed for checking whether automated graders are reliable and generalize.

Key points

  • Resource: 14 AES and ASAG datasets with over 37K graded responses, including ASAP-AES, ASAP 2.0, ASAP-SAS, CSEE, Rice_Chem, OS_Dataset, SciEntSBank, BEEtlE and Mohlar. Platform: sgrades.eng.unt.edu; code at github.com/nlpatunt/sgrades.
  • Models and prompting: Three models (Llama 4 Scout, Gemini 2.5 Flash and GPT-4o mini) were evaluated at temperature 0.1 under zero-shot (deductive), few-shot (inductive), Chain-of-Thought (abductive) and combined configurations.
  • Overall difficulty: mean QWK was 0.34–0.41 for ASAG versus 0.42–0.43 for AES.
  • Exemplar transfer, ASAP2.0 → ASAP-AES: QWK fell by 96.2% for LLaMA-4-Scout but rose by 9.4% for Gemini-2.5-Flash. Separately, CSEE → ASAP-AES exemplars improved Gemini by 12.5%.
  • Transfer from AES to ASAG and limitations: using AES exemplars on ASAG lowered QWK for all models, by 13.2% to 75.5%. All datasets are English and text-only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2hybhgodz2en

@inproceedings{seuti2026s,
  title = {{S-GRADES – Studying Generalization of Student Response Assessments in Diverse Evaluative Settings}},
  author = {Seuti, Tasfia and Choudhury, Sagnik Ray},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2hybhgodz2en},
  url = {https://doi.org/10.63317/2hybhgodz2en}
}

References

35 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Attali and J. Burstein. 2006. Automated essay scoring with e-rater v.2. The Journal of Technology, Learning, and Assessment, 4(3). find
  2. Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The eras and trends of automatic short answer grading. International journal of artificial intelligence in education, 25(1):60–117. find
  3. Li-Hsin Chang and Filip Ginter. 2024. Automatic short answer grading for finnish with chatgpt. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23173–23181. find
  4. Scott A Crossley, Perpetual Baffour, L Burleigh, and Jules King. 2025. A large-scale corpus for assessing source-based writing quality: Asap 2.0. Assessing Writing, 65:100954. find
  5. Scott A Crossley, Y Tian, P Baffour, Abigail Franklin, Margaret Benner, and Ulrich Boser. 2024. A large-scale corpus for assessing written argumentation: Persuade 2.0. Assessing Writing, 61:100865. find
  6. Nigel Fernandez, Aritra Ghosh, Naiming Liu, Zichao Wang, Benoît Choffin, Richard Baraniuk, and Andrew Lan. 2022. Automated scoring for reading comprehension via in-context bert tuning. In International conference on artificial intelligence in education, pages 691–697. Springer. find
  7. P. W. Foltz, D. Laham, and T. K. Landauer. 1999. The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2):939–944. find
  8. Rujun Gao, Xiaosu Guo, Xiaodi Li, Arun Balajiee Lekshmi Narayanan, Naveen Thomas, and Arun R Srinivasa. 2024. Towards scalable automated grading: Leveraging large language models for conceptual question evaluation in engineering. arXiv preprint arXiv:2411.03659. — [arXiv:2411.03659](https://arxiv.org/abs/2411.03659) find
  9. Rujun Gao, Naveen Thomas, and Arun Srinivasa. 2023. Work in progress: Large language model based automatic grading study. In 2023 IEEE Frontiers in Education Conference (FIE), pages 1–4. IEEE. find
  10. Christian Grévisse. 2024. Llm-based automatic short answer grading in undergraduate medical education. BMC Medical Education, 24(1):1060. find
  11. Stefan Haller, Adina Aldea, Christin Seifert, and Nicola Strisciuglio. 2022. Survey on automated short answer grading with deep learning: from word embeddings to transformers. arXiv preprint arXiv:2204.03503. — [arXiv:2204.03503](https://arxiv.org/abs/2204.03503) find
  12. Owen Henkel, Libby Hills, Bill Roberts, and Joshua McGrane. 2023. Can llms grade short-answer reading comprehension questions: An empirical study with a novel dataset. arXiv preprint arXiv:2310.18373. — [arXiv:2310.18373](https://arxiv.org/abs/2310.18373) find
  13. Rositsa Ivanova and Siegfried Handschuh. 2024. Evaluating llms’ performance at automatic short-answer grading. find
  14. Lan Jiang and Nigel Bosch. 2024. Short answer scoring with gpt-4. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 438–442. find
  15. Gerd Kortemeyer. 2024. Performance of the pre-trained large language model gpt-4 on automated short answer grading. Discover Artificial Intelligence, 4(1):47. find
  16. T. K. Landauer, P. W. Foltz, and D. Laham. 1998. An introduction to latent semantic analysis. Discourse Processes, 25(2–3):259–284. find
  17. Emmy Liu, Graham Neubig, and Jacob Andreas. 2024. An incomplete loop: Deductive, inductive, and abductive learning in large language models. In Conference on Language Modeling (COLM), Philadelphia, USA. find
  18. S. Ludwig, C. Mayer, C. Hansen, K. Eilers, and S. Brandt. 2021. Automated essay scoring using transformer models. Psych, 3(4):897–915. find
  19. Sandeep Mathias and Pushpak Bhattacharyya. 2018. Asap++: Enriching the asap automated essay grading dataset with essay attribute scores. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  20. Mary L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia Medica, 22(3):276– 282. find
  21. A. Mizumoto and M. Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. find
  22. Christopher M Ormerod, Akanksha Malhotra, and Amir Jafari. 2021. Automated essay scoring using efficient transformer-based language models. arXiv preprint arXiv:2102.13136. — [arXiv:2102.13136](https://arxiv.org/abs/2102.13136) find
  23. A. Pack, Y. Li, L. Chen, M. Luo, Y. Xu, and Y. Liu. 2024. Large language models and automated essay scoring of english language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6:100234. find
  24. E. B. Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243. find
  25. Zhuang Qiu, Peizhi Yan, and Zhenguang Cai. 2024. Large language models for second language english writing assessments: An exploratory comparison. In Proceedings of the 38th Pacific Asia Conference on Language, Information and Computation, pages 363–370. find
  26. Pedro Uria Rodriguez, Amir Jafari, and Christopher M Ormerod. 2019. Language models and automated essay scoring. arXiv preprint arXiv:1909.09482. — [arXiv:1909.09482](https://arxiv.org/abs/1909.09482) find
  27. A. Schneider, M. Schuetz, T. Henneke, and J. Kuhn. 2023. Towards llm-based autograding for short textual answers. arXiv preprint arXiv:2311.04621. — [arXiv:2311.04621](https://arxiv.org/abs/2311.04621) find
  28. Kathrin Seßler, Maurice Fürstenberg, Babette Bühler, and Enkelejda Kasneci. 2025. Can ai grade your essays? a comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pag… find
  29. Shashank Sonkar, Kangqi Ni, Lesa Tran Lu, Kristi Kincaid, John S Hutchinson, and Richard G Baraniuk. 2024. Automated long answer grading with ricechem dataset. In International Conference on Artificial Intelligence in Education, pages 163– 176. Springer. find
  30. Chul Sung, Tejas Dhamecha, Swarnadeep Saha, Tengfei Ma, Vinay Reddy, and Rishi Arora. 2019. Pre-training bert on domain resources for short answer grading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find
  31. K. Taghipour and H. T. Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1882–1891. find our card
  32. C. Xiao, J. Chen, S. Li, T. Wang, R. Zhao, Z. Tang, and Q. Liu. 2024. From automation to augmentation: Large language models elevating the essay scoring landscape. arXiv preprint arXiv:2401.06431. — [arXiv:2401.06431](https://arxiv.org/abs/2401.06431) find
  33. Changrong Xiao, Wenxing Ma, Qingping Song, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Qi Fu. 2025. Human-ai collaborative essay scoring: A dual-process framework with llms. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 293–305. find
  34. Wenjing Xie, Juxin Niu, Chun Jason Xue, and Nan Guan. 2024. Grade like a human: Rethinking automated assessment with large language models. arXiv preprint arXiv:2405.19694. — [arXiv:2405.19694](https://arxiv.org/abs/2405.19694) find
  35. Comprehensive Reasoning Strategy Analysis A. This appendix provides detailed sensitivity analysis across all reasoning configurations, models, and datasets evaluated in S-GRADES. We quantify the extent to which performance varies with reasoning-strategy selection, complementing the primary results r… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seuti-2026-s-grades-studying-generalization-student/