Elamur Research⌕
☰
home / papers / 2026 / Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation

Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation

Paper · 2026 · Abigail Gurin Schleifer, Moriah Ariely, Beata Beigman Klebanov et al.

Summary

The paper asks whether automated short answer scoring (ASAS) is equally accurate for weak, partial and strong student answers, and how this depends on the amount of task-specific data a model receives. The authors used responses from roughly 800 Israeli high-school students to two open-ended biology questions on cellular respiration. Two biology education experts graded the responses with an analytic rubric. One expert's scores served as ground truth. Against this reference they compared a second human expert, a fine-tuned BERT-based encoder, and three LLMs (GPT-5.2, GPT-4o, Claude Opus 4.5) prompted few-shot with different numbers of examples. Agreement was measured separately for each level of response quality.

Agreement between the two humans was the highest and stayed stable across all quality levels. Every AI scorer handled fully correct and fully wrong answers well but lost accuracy on partially correct answers, giving a U-shaped error pattern. This weakness was worst for LLMs given few examples, shrank as more labelled data was added, and was smallest for the fine-tuned encoder. The authors frame this as a "quality-conditioned fairness" concern, because students with developing understanding may be scored least reliably.

Key points

  • Data: about 800 students answered two conceptually parallel biology items in the style of national matriculation exam questions. The items ask how smoking and anemia affect physical exercise via cellular respiration.
  • Scorers compared:
    • GPT-5.2, GPT-4o and Claude Opus 4.5, each in several few-shot settings;
    • a fine-tuned BERT-based encoder;
    • a second human expert.
  • Main finding: AI errors cluster in the mid-range of response quality, which the authors call mid-range degradation. The degradation lessens as task-specific adaptation grows, from few examples to more examples to fine-tuning. Human–human agreement shows no such dip.
  • Practical suggestion: few-shot LLMs could act as a first pass for clearly correct or clearly incorrect answers. Responses predicted as mid-range would go to human graders until enough labelled data exists for fine-tuning.
  • Limitations:
    • The prompts were first developed and tested with GPT-4o in mind.
    • Only two items from a single domain were studied.
    • Standard metrics such as QWK and Pearson's correlation were not reported, which makes comparison with ASAS benchmarks harder.
  • Venue: BEA 2026 workshop proceedings, pp. 432–442.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Weizmann Institute of ScienceRehovot · Israel8 works · 2,326 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Abigail Gurin Schleifer, Moriah Ariely, Beata Beigman Klebanov, Asaf Salman and Giora Alexandron. 2026. Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.29

@inproceedings{schleifer2026quality,
  title = {{Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation}},
  author = {Abigail Gurin Schleifer, Moriah Ariely, Beata Beigman Klebanov, Asaf Salman and Giora Alexandron},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.29},
  url = {https://doi.org/10.18653/v1/2026.bea-1.29}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude opus 4.5 system card. Technical report, Anthropic. System card. find
  2. Moriah Ariely, Tanya Nazaretsky, and Giora Alexandron. 2023. Machine learning and Hebrew NLP for automated assessment of open-ended questions in biology. International Journal of Artificial Intelligence in Education, 33(1):1–34. find
  3. Moriah Ariely, Tanya Nazaretsky, and Giora Alexandron. 2024. Causal-mechanical explanations in biology: Applying automated assessment for personalized learning in the science classroom. Journal of Research in Science Teaching, 61(8):1858–1889. find
  4. Moriah Ariely, Asaf Salman, Anat Yarden, and Giora Alexandron. 2025. Reflective prompt engineering: a new strategy for automated short answer scoring in biology. International Journal of Science Education, pages 1–23. find
  5. Ryan Baker and Aaron Hawn. 2022. Algorithmic bias in education. International journal of artificial intelligence in education, 32(4):1052–1092. find
  6. Beata Beigman Klebanov and Nitin Madnani. 2022. Automated essay scoring. Springer Nature. find
  7. Leonardo Berti, Flavio Giorgi, and Gjergji Kasneci. 2025. Emergent abilities in large language models: A survey. arXiv preprint arXiv:2503.05788. — [arXiv:2503.05788](https://arxiv.org/abs/2503.05788)
  8. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2023. Similarity-based content scoring-a more classroom-suitable alternative to instance-based scoring? In Findings of the association for computational linguistics: Acl 2023, pages 1892–1903. find
  9. Sridevi Bonthu, S Rama Sree, and MHM Krishna Prasad. 2021. Automated short answer grading using deep learning: A survey. In International cross-domain conference for machine learning and knowledge extraction, pages 61–78. Springer. find
  10. Gregory Camilli. 2006. Test fairness. Educational measurement, 4:221–256. find
  11. Imran Chamieh, Torsten Zesch, and Klaus Giebermann. 2024. LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 309–315, Mexico City, Mexico. Ass… find
  12. Li-Hsin Chang and Filip Ginter. 2024. Automatic short answer grading for finnish with chatgpt. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23173–23181. find
  13. Lei Chen and Raymond Ng. 2004. On the marriage of lp-norms and edit distance. In Proceedings of the Thirtieth international conference on Very large data bases-Volume 30, pages 792–803. find
  14. Sribala Vidyadhari Chinta, Zichong Wang, Zhipeng Yin, Nhat Hoang, Matthew Gonzalez, T Le Quy, and Wenbin Zhang. 2024. Fairaied: Navigating fairness, bias, and ethics in educational AI applications. arXiv preprint arXiv:2407.18745. — [arXiv:2407.18745](https://arxiv.org/abs/2407.18745) find
  15. Aubrey Condor. 2020. Exploring automatic short answer grading as a tool to assist in human rating. In International Conference on Artificial Intelligence in Education, pages 74–79. Springer. find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  17. Yuning Ding, Brian Riordan, Andrea Horbach, Aoife Cahill, and Torsten Zesch. 2020. Don’t take “nswvtnvakgxpm” for an answer –the surprising vulnerability of automatic content scoring systems to adversarial input. In Proceedings of the 28th International Conference on Computational Linguistics, pages… find
  18. Emrah Emirtekin and Yasin Özarslan. 2026. Automatic short-answer grading in sustainability education: AI– human agreement. Journal of Computer Assisted Learning, 42(1):e70160. find
  19. Rafael Ferreira Mello, Cleon Pereira Junior, Luiz Rodrigues, Filipe Dwan Pereira, Luciano Cabral, Newar-ney Costa, Geber Ramalho, and Dragan Gasevic. 2025. Automatic short answer grading in the LLM era: Does GPT-4 with prompt engineering beat traditional models? In Proceedings of the 15th Internatio… find
  20. Guher Gorgun and Seyma N. Yildirim-Erbasli. 2026. Algorithmic bias in BERT for response accuracy prediction: A case study for investigating population validity. Journal of Educational Measurement, 63(1):e12420. find
  21. Christian Grévisse. 2024. LLM-based automatic short answer grading in undergraduate medical education. BMC Medical Education, 24(1):1060. find
  22. Abigail Gurin Schleifer, Beata Beigman Klebanov, and Giora Alexandron. 2025. Uncovering measurement biases in LLM embedding spaces: The anna karenina principle and its implications for automated feedback. International Journal of Artificial Intelligence in Education, 35(5):2821–2855. find
  23. Abigail Gurin Schleifer, Beata Beigman Klebanov, Moriah Ariely, and Giora Alexandron. 2023. Transformer-based hebrew nlp models for short answer scoring in biology. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 550–555.
  24. Stefan Haller, Adina Aldea, Christin Seifert, and Nicola Strisciuglio. 2022. Survey on automated short answer grading with deep learning: from word embeddings to transformers. arXiv preprint arXiv:2204.03503. — [arXiv:2204.03503](https://arxiv.org/abs/2204.03503) find
  25. Zhaohui Li, Yajur Tomar, and Rebecca J Passonneau. 2021. A semantic feature-wise transformation relation network for automatic short answer grading. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6030–6040. find
  26. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01552.
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  28. Anastassia Loukina, Nitin Madnani, and Klaus Zechner. 2019. The many dimensions of algorithmic fairness in educational applications. In Proceedings of the fourteenth workshop on innovative use of NLP for building educational applications, pages 1–10. find
  29. Michael Madaio, Su Lin Blodgett, Elijah Mayfield, and Ezekiel Dixon-Román. 2022. Beyond “fairness”: Structural (in) justice lenses on AI for education. In The ethics of artificial intelligence in education, pages 203–239. Routledge. find
  30. Nitin Madnani, Anastassia Loukina, Alina von Davier, Jill Burstein, and Aoife Cahill. 2017. Building better open-source tools to support fairness in automated scoring. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 41– 52, Valencia, Spain. Association for Co… find
  31. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  32. Owen Henkel, Libby Hills, Adam Boxer, Bill Roberts, and Zach Levonian. 2024. Can large language models make the grade? an empirical study evaluating LLMs ability to mark short answer questions in k-12 education. In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, page 300–304… find
  33. Matthew S Johnson and Daniel F McCaffrey. 2023. Evaluating fairness of automated scoring in educational measurement. In Advancing natural language processing in educational assessment, pages 142– 164. Routledge. find
  34. Gerd Kortemeyer. 2023. Toward AI grading of student problem solutions in introductory physics: A feasibility study. Phys. Rev. Phys. Educ. Res., 19:020163. find
  35. Gerd Kortemeyer. 2024. Performance of the pre-trained large language model gpt-4 on automated short answer grading. Discover Artificial Intelligence, 4(1):47. find
  36. Ido Roll, Ryan SJ d Baker, Vincent Aleven, and Kenneth R Koedinger. 2014. On the benefits of seeking (and avoiding) help in online problem-solving environments. Journal of the Learning Sciences, 23(4):537–560. find
  37. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew. arXiv preprint arXiv:2308.16687.
  38. Valerie J Shute. 2008. Focus on formative feedback. Review of educational research, 78(1):153–189. find
  39. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 others. 2025. OpenAI gpt-5 system card. arXiv preprint arXiv:2601.03267.
  40. Maya Usher and Montathar Faraon. 2025. Who grades best? comparing chatgpt, peer, and instructor evaluations across varying levels of student project quality. Assessment & Evaluation in Higher Education, pages 1–20. find
  41. David M. Williamson, Xiaoming Xi, and F. Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1):2–13. find
  42. X. Wu, P. P. Saraf, G. Lee, and 1 others. 2025. Unveiling scoring processes: Dissecting the differences between LLMs and human graders in automatic scoring. Technology, Knowledge and Learning. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schleifer-2026-quality-conditioned-agreement-automated-short/