Elamur Research⌕
☰
home / papers / 2026 / SDPA at BEA 2026 Shared Task 2: Efficient LLM Fine-Tuning for Rubric-based Short Answer Scoring

SDPA at BEA 2026 Shared Task 2: Efficient LLM Fine-Tuning for Rubric-based Short Answer Scoring

Paper · 2026 · Zhexiong Liu and Jing Zhang

Summary

The paper addresses automated short-answer scoring (ASA), where open-ended student answers must be graded against interrelated rubrics. Prior LLM work on this task has mostly relied on prompting. The authors submitted to the BEA 2026 Shared Task 2, which uses the German ALICE-LP-1.0 corpus of secondary-school STEM answers. The task has four tracks, crossing unseen answers or unseen questions with two-way or three-way labels.

They compared two families of systems:

  • Encoder ensemble: fine-tuned GELECTRA-large, XLM-RoBERTa-large and BERT-base-german-cased, combined through weighted soft voting. Weights were tuned with Nelder-Mead to maximize QWK. Few-shot Claude-Opus-4.6 labels were optionally added.
  • Fine-tuned LLM: Llama3.1-8B with a classification head, adapted with LoRA or DoRA so that it predicts label probabilities directly instead of generating tokens.

The fine-tuned Llama models beat every ensemble configuration on all tracks. The margin was largest on unseen questions. The team placed second on the binary unseen-question track and third on the other three. The study suggests that parameter-efficient fine-tuning is a practical alternative to prompting for low-resource, non-English educational scoring.

Key points

  • Dataset (ALICE-LP-1.0): 7,899 training responses and 827 trial responses. The test set contains 2,008 unseen-answer and 3,168 unseen-question responses. It covers physics, mathematics, biology and chemistry, collected through Moodle activities.
  • Three-way track: Llama3.1-8B + LoRA reached QWK 0.7760 (UA) and 0.6444 (UQ). The best ensemble (with Claude-Opus-4.6) reached 0.7566 and 0.5446.
  • Two-way track: LoRA scored QWK 0.6822 (UA) and 0.5348 (UQ), compared with 0.6617 and 0.4609 for the Claude-augmented ensemble.
  • Encoder comparison: GELECTRA-large was the strongest single encoder. XLM-RoBERTa-large trained unstably, with some seeds reaching QWK = 0, so it was left out of the two-way submissions.
  • LoRA vs. DoRA: LoRA generally achieved higher QWK at similar weighted F1 and is cheaper, so the authors prefer it. Stated limitations are that experiments cover German only and that the method requires GPUs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PittsburghPittsburgh · United States59 works · 22,324 citations🎓Emory UniversityAtlanta · United States36 works · 39,779 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Zhexiong Liu and Jing Zhang. 2026. SDPA at BEA 2026 Shared Task 2: Efficient LLM Fine-Tuning for Rubric-based Short Answer Scoring. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.92

@inproceedings{liu2026sdpa,
  title = {{SDPA at BEA 2026 Shared Task 2: Efficient LLM Fine-Tuning for Rubric-based Short Answer Scoring}},
  author = {Zhexiong Liu and Jing Zhang},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.92},
  url = {https://doi.org/10.18653/v1/2026.bea-1.92}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dishank Aggarwal, Pritam Sil, Bhaskaran Raman, and Pushpak Bhattacharyya. 2025. “i understand why i got this grade”: Automatic short answer grading (asag) with feedback. pages 304–318. find
  2. Anthropic. 2026. Introducing claude opus 4.6. find
  3. Darina Benikova, Chris Biemann, Max Kisselew, and Sebastian Pado. 2014. find
  4. Germeval 2014 named entity recognition shared task: companion paper. In Workshop Proceedings of the 12th edition of the KON-VENS conference, pages 104–112. find
  5. David C Berliner and Barak Rosenshine. 2017. The acquisition of knowledge in the classroom. In Schooling and the acquisition of knowledge, pages 375–396. Routledge. find
  6. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2022. Similarity-based content scoring-how to make s-bert keep up with bert. In Proceedings of the 17th workshop on innovative use of NLP for building educational applications (BEA 2022), pages 118–123. find
  7. Sridevi Bonthu, S. Rama Sree, and M.H.M. Krishna Prasad. 2023. Improving the performance of automatic short answer grading using transfer learning and augmentation. Engineering Applications of Artificial Intelligence, 123:106292. find
  8. Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25:60–117. find
  9. Dan Carpenter, Wookhee Min, Seung Lee, Gamze Ozogul, Xiaoying Zheng, and James Lester. 2024. Assessing student explanations with large language models using fine-tuning and few-shot learning. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA), pa… find
  10. Romain D Cazé and Matthijs AA van der Meer. 2013. Adaptive properties of differential learning rates for positive and negative outcomes. Biological cybernetics, 107(6):711–719. find
  11. Imran Chamieh, Torsten Zesch, and Klaus Giebermann. 2024. LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA), pages 309–315. find
  12. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. find
  14. Alexis Conneau, Kartik Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL, pages 8440–8451. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  16. Myroslava O Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Second Joint Conference on Lexical…
  17. Rafael Ferreira Mello, Cleon Pereira Junior, Luiz Rodrigues, Filipe Dwan Pereira, Luciano Cabral, Newar-ney Costa, Geber Ramalho, and Dragan Gasevic. 2025. Automatic short answer grading in the llm era: Does gpt-4 with prompt engineering beat traditional models? In Proceedings of the 15th Internatio… find
  18. Anna Filighera, Siddharth Parihar, Tim Steuer, Tobias Meuser, and Sebastian Ochs. 2022. Your answer is incorrect... would you like to know why? introducing a bilingual short answer feedback dataset. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
  19. Hiroaki Funayama, Yuichiroh Matsubayashi, Yuya Asazuma, Tomoya Mizumoto, and Kentaro Inui. 2025. Cross-prompt pre-finetuning of language models for short answer scoring. International Journal of Artificial Intelligence in Education, pages 1–22. find
  20. Sebastian Gombert, Zhifan Sun, Fabian Zehner, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Sascha Bernholt, Knut Neumann, Ute Harms, Aiso Heinze, and Hendrik Drachsler. 2026. Report on the bea 2026 shared task on rubric-based short answer scoring for german. In Proceedin…
  21. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and 1 others. 2024. The LLaMA 3 herd of models. arXiv preprint arXiv:2407.21783.
  22. Stefan Haller, Adina Aldea, Christin Seifert, and Nicola Strisciuglio. 2022. Survey on automated short answer grading with deep learning: from word embeddings to transformers. arXiv preprint arXiv:2204.03503. — [arXiv:2204.03503](https://arxiv.org/abs/2204.03503) find
  23. Owen Henkel, Libby Hills, Adam Boxer, Bill Roberts, and Zach Levonian. 2024. Can large language models make the grade? an empirical study evaluating llms ability to mark short answer questions in k-12 education. In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, page 300–304… find
  24. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shanen Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. Proceedings of ICLR.
  25. Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing,…
  26. Tianwen Li, Zhexiong Liu, Lindsay Matsumura, Elaine Wang, Diane Litman, and Richard Correnti. 2024. Using large language models to assess young students’ writing revisions. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 365–380, M… find
  27. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. DoRA: Weight-decomposed low-rank adaptation. Proceedings of ICML. find
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  29. Zhexiong Liu. 2025. Understanding and Generating Text Revisions With Intention-Adaptive Large Language Models. Ph.D. thesis, University of Pittsburgh. find
  30. Zhexiong Liu and Diane Litman. 2025. Efficient layer-wise LLM fine-tuning for revision intention prediction. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15319– 15334, Suzhou, China. Association for Computational Linguistics. find
  31. Zhexiong Liu and Diane Litman. 2026. Intention-adaptive LLM fine-tuning for text revision generation. In Findings of the Association for Computational Linguistics: EACL 2026, pages 1263–1281, Rabat, Morocco. Association for Computational Linguistics.
  32. Zhexiong Liu, Jing Zhang, Jiaying Lu, Wenjing Ma, and Joyce C. Ho. 2024b. Logicprpbank: A corpus for logical implication and equivalence. In Proceedings of the 2024 AAAI Conference on Artificial Intelligence, volume 257 of Proceedings of Machine Learning Research, pages 57–65. PMLR. find
  33. Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 752–762…
  34. John A Nelder and Roger Mead. 1965. A simplex method for function minimization. The Computer Journal, 7(4):308–313. find
  35. Ulrike Padó, Yunus Eryilmaz, and Larissa Kirschner. 2024. Short-answer grading for german: Addressing the challenges. International Journal of Artificial Intelligence in Education, 34(4):1321–1352. find
  36. Lakshmi Ramachandran, Jian Cheng, and Peter Foltz. 2015. Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. In Proceedings of the tenth workshop on innovative use of NLP for building educational applications, pages 97–106. find
  37. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992.
  38. Chul Sung, Tejas Dhamecha, Swarnadeep Saha, Tengfei Ma, Vinay Reddy, and Rishi Arora. 2019. Pre-training BERT on domain resources for short answer grading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find
  39. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm.stanford. edu/2023/03/13/alpaca. html, 3(6):… ↗
  40. Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the germeval 2018 shared task on the identification of offensive language. find
  41. Jiaxin Zhang, Zhong-Zhi Li, Ming-Liang Zhang, Fei Yin, Cheng-Lin Liu, and Yashar Moshfeghi. 2024. Geoeval: benchmark for evaluating llms and multimodal models on geometry problem-solving. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1258–1276. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-sdpa-bea-2026-shared-task/