Elamur Research⌕
☰
home / papers / 2026 / Report on the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German

Report on the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German

Paper · 2026 · Sebastian Gombert, Zhifan Sun, Fabian Zehner et al.

Summary

This report gives an overview of the BEA 2026 shared task, which focused on scoring short answers against rubrics. In this task, a system receives a question, a student's answer and a textual rubric that states the criteria for each score level, and it must pick the matching score. The organisers argue that this setting avoids two weaknesses of earlier approaches. Instance-based scorers transfer poorly to new questions, and similarity-based scorers need reference answers that cover every valid response.

The organisers released ALICE-LP 1.1, a new German benchmark. It contains middle- and high-school answers from four STEM subjects. The evaluation was split along two axes: answers to seen versus unseen questions, and 2-way versus 3-way scoring. That gives four tracks, ranked by QWK.

The submissions ranged from fine-tuned LLMs and prompting to ensembles and human–AI workflows. The strongest systems were LLMs that were adapted to the task and that encoded rubric structure explicitly. Their lead shrank on unseen questions, so generalisation remains the main open problem. The dataset also adds to the scarce supply of non-English ASAS resources.

Key points

  • Dataset. ALICE-LP 1.1 covers physics, mathematics, biology and chemistry. Answers were collected in Moodle lessons at schools in Schleswig-Holstein, grades 7–11, within the ALICE project.
  • Labels and rubrics. Every answer is scored on three levels: incorrect, partially correct and correct. A 2-way rubric was derived by merging the incorrect and partially correct levels. Raters were trained until they reached κ ≥ .65.
  • Splits.
    • Training: 7,899 answers to 78 questions.
    • Unseen answers: 2,008 answers to the same questions, from a question-wise stratified 80/20 split.
    • Unseen questions: 3,086 answers to 39 new questions. One question with 82 responses was dropped because it could not be graded without a reference answer.
  • Metrics. Each track reports weighted precision, recall and F1, plus QWK. QWK is the primary leaderboard metric.
  • Winning system. IWM-DKM applied LoRA fine-tuning to Qwen models. It added rubric-aware input structuring: checklist-style reasoning, rubrics rewritten as decision trees, injected background knowledge and ensemble voting.
  • Other systems.
    • RETUYT-INCO used meta-prompting to optimise its prompts.
    • AMATI fed ILP-derived rules into an LLM.
    • HFT ran a zero-shot LLM and had humans selectively review its predictions.

Interesting ideas

  • I1. Rewriting a free-text rubric as a decision tree or checklist turns vague criteria into explicit yes/no tests that can be checked one at a time, which makes each verdict traceable.
  • I2. Evaluating separately on new answers to known questions and on entirely new questions exposes how much a system memorises item-specific patterns instead of general skill.
  • I3. Deriving a coarse 2-way label from a 3-way scale by merging levels gives two difficulty settings from one annotation effort and shows where fine distinctions break down.
  • I4. Injecting symbolic rules induced by ILP into an LLM trades some flexibility for controllability and inspectable decisions.
  • I5. Letting a cheap zero-shot model handle most cases while humans review only selected predictions keeps overall quality high at low cost.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course test readers could compile answer criteria into small checklists evaluated over the world graph, so every grade comes with the list of satisfied and failed conditions.
  • I2 (learn): Evaluation could hold out whole fables or problem templates, not just sentences, as a check that repair rules and trained parsers generalise beyond the items they were tuned on.
  • I3 (learn): Silver-data evaluation could report both a collapsed and a fine-grained version of a label set, for example coarse versus full dependency relations, to locate where witness parsers disagree.
  • I4 (learn): Self-induced repair rules could be learned in a similar ILP style from parser errors, keeping each correction a readable rule rather than an opaque weight change.
  • I5 (learn): LLM teacher tiers could be queried only for low-confidence trees or frames, and the cheaper witnesses would handle the rest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany75 works · 41,081 citations🔬DIPF | Leibniz Institute for Research and Information in EducationFrankfurt am Main · Germany9 works · 2,209 citations🔬Leibniz Institute for Science and Mathematics EducationKiel · Germany4 works · 261 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Sebastian Gombert, Zhifan Sun, Fabian Zehner, Jannik Lossjew, Tobias Wyrwich, Berrit Czinczel, David Bednorz, Sascha Bernholt, Knut Neumann, Ute Harms, Aiso Heinze and Hendrik Drachsler. 2026. Report on the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.85

@inproceedings{gombert2026report,
  title = {{Report on the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German}},
  author = {Sebastian Gombert, Zhifan Sun, Fabian Zehner, Jannik Lossjew, Tobias Wyrwich, Berrit Czinczel, David Bednorz, Sascha Bernholt, Knut Neumann, Ute Harms, Aiso Heinze and Hendrik Drachsler},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.85},
  url = {https://doi.org/10.18653/v1/2026.bea-1.85}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiaoyu Bai and Manfred Stede. 2023. A survey of current machine learning approaches to student free-text evaluation for intelligent tutoring. International Journal of Artificial Intelligence in Education, 33(4):992– 1030. ↗
  2. David Bednorz, Kristin Litteck, Daniel Sommerhoff, and Aiso Heinze. 2024. Erfassung individueller Lern-trajektorien in einer digitalen Lernumgebung zum Ableitungsbegriff. Universitätsbibliothek Dortmund. find
  3. Kate Rebecca Belcher, Marius De Kuthy Meurers, Kordula De Kuthy, and Detmar Meurers. 2026. IWM-DKM at BEA 2026 shared task 2: Supplementing supervised fine-tuning for rubric-based short answer scoring. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics. find
  4. Marie Bexte, Yuning Ding, Josef Ruppenhofer, Nils-Jonathan Schaller, Daniel Mora Melanchthon, Torsten Zesch, and Andrea Horbach. 2026. ASLAN at BEA 2026 shared task 2: Voting across scoring paradigms. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics. find
  5. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2022. Similarity-based content scoring-how to make S-BERT keep up with BERT. In Proceedings of the 17th workshop on innovative use of nlp for building educational applications (bea 2022), pages 118–123. find
  6. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2023. Similarity-based content scoring - a more classroom-suitable alternative to instance-based scoring? In Findings of the Association for Computational Linguistics: ACL 2023, pages 1892–1903, Toronto, Canada. Association for Computational Linguistics. find
  7. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2024. Strengths and weaknesses of automated scoring of free-text student answers. Informatik Spektrum, pages 1–9. ↗
  8. Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The eras and trends of automatic short answer grading. International journal of artificial intelligence in education, 25:60–117. ↗
  9. Leon Camus and Anna Filighera. 2020. Investigating transformers for automatic short answer grading. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21, pages 43–48. Springer. find
  10. Aubrey Condor, Zachary Pardos, and Marcia Linn. 2022. Representing scoring rubrics as graphs for automatic short answer grading. In International Conference on Artificial Intelligence in Education, pages 354–365. Springer. find
  11. Longwei Cong, Leon Hammerla, Sonja Hahn, Sebastian Gombert, Hendrik Drachsler, and Ulf Kroehne. 2026. Automatic short answer grading with llms: From memorization to reasoning. In Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference, pages 75–84. find
  12. Berrit K Czinczel, Daniela Fiedler, and Ute Harms. 2025. How do species change over time? designing a hybrid teaching unit on five factors of evolution. The American Biology Teacher, 87(2):78–83. ↗
  13. Myroslava Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. SemEval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 263–274, Atlanta, Georgia, USA. Association for Computational Linguistics.
  14. Anna Filighera, Siddharth Parihar, Tim Steuer, Tobias Meuser, and Sebastian Ochs. 2022. Your answer is incorrect... would you like to know why? introducing a bilingual short answer feedback dataset. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8577–8591. find
  15. Scott Frohn, Tyler Burleigh, and Jing Chen. 2025. Automated scoring of short answer questions with large language models: Impacts of model, item, and rubric design. In International Conference on Artificial Intelligence in Education, pages 44–51. Springer. find
  16. Sebastian Gombert, Daniele Di Mitri, Onur Karademir, Marcus Kubsch, Hannah Kolbe, Simon Tautz, Adrian Grimm, Isabell Bohm, Knut Neumann, and Hendrik Drachsler. 2023. Coding energy knowledge in constructed responses with explainable NLP models. Journal of Computer Assisted Learning, 39(3):767– 786. ↗
  17. Sebastian Gombert, Sonja Hahn, Nico Andersen, Leon Camus, Zhifan Sun, Ngoc Nhu Hao Nguyen, Fabian Zehner, Longwei Cong, Alexander Mehler, and Hendrik Drachsler. 2026a. Rubrics as semantic subspaces: A unified approach to rubric-based constructed response scoring across short answers and essays. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics. Sebastian Gombert, Zhifan Sun, Fabian Zehner, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Marcus Kubsch, Daniele Di Mitri, Knut Neumann, and 1 others. 2026b. Are rubrics all you need? towards rubric-based automatic short answer scoring via guided rubric-answer alignment. In Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference, pages 272– 282. find
  18. Jonas Gwozdz and Andreas Both. 2026. WSE research at BEA 2026 shared task 2: Multi-strategy rubric-based short answer scoring for german: Multi-strategy rubric-based short answer scoring for german. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics. find
  19. Ben Hamner, Jaison Morgan, lynnvandev, Mark Shermis, and Tom Vander Ark. 2012. The hewlett foundation: Automated essay scoring. https://kaggle.com/competitions/asap-aes. Kaggle. ↗
  20. Marcus Kubsch, Berrit Czinczel, Jannik Lossjew, Tobias Wyrwich, David Bednorz, Sascha Bernholt, Daniela Fiedler, Sebastian Strauß, Ulrike Cress, Hendrik Drachsler, and 1 others. 2022. Toward learning progression analytics—developing learning environments for the automated analysis of learning using evidence centered design. In Frontiers in education, volume 7, page 981910. Frontiers Media SA. find
  21. Zhaohui Li, Susan Lloyd, Matthew Beckman, and Rebecca J Passonneau. 2023. Answer-state recurrent relational network (AsRRN) for constructed response assessment and feedback grouping. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3879–3891. find
  22. Zhexiong Liu and Jing Zhang. 2026. SDPA at BEA 2026 shared task 2: Efficient LLM fine-tuning for rubric-based short answer scoring. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics.
  23. Jannik Lossjew and Sascha Bernholt. 2025. Digital support custom-made or in moderation–design and evaluation of a digitally supported teaching unit on reaction kinetics and chemical equilibrium. CHEMKON, 32(4):122–131. ↗
  24. OECD. 2023. PISA 2022 Assessment and Analytical Framework. PISA. OECD Publishing. ↗
  25. OECD. 2024. PISA 2022 Technical Report. PISA. OECD Publishing. ↗
  26. Ulrike Padó. 2026. HFT at BEA 2026 shared task 2: Blunt-edge models for hybrid grading. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics. ↗
  27. Ulrike Padó, Yunus Eryilmaz, and Larissa Kirschner. 2023. Short-answer grading for german: Addressing the challenges. International Journal of Artificial Intelligence in Education, 34(4):1321–1352. ↗
  28. Ernesto Panadero and Anders Jonsson. 2013. The use of scoring rubrics for formative assessment purposes revisited: A review. Educational research review, 9:129–144. ↗
  29. Y Malini Reddy and Heidi Andrade. 2010. A review of rubric use in higher education. Assessment & evaluation in higher education, 35(4):435–448. ↗
  30. Fabian Zehner, Christine Sälzer, and Frank Goldhammer. 2016. Automatic coding of short text responses via clustering in educational assessment. Educational and psychological measurement, 76(2):280– 303. ↗
  31. A Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch, and Chong Min Lee. 2017. Investigating neural architectures for short answer scoring. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 159–168, Copenhagen, Denmark. Association for Computational Linguistics. find
  32. Ignacio Sastre, Ignacio Remersaro, Facundo Díaz, Nicolás de Horta, Luis Chiruzzo, Aiala Rosá, and Santiago Góngora. 2026. RETUYT-INCO at BEA 2026 shared task 2: Meta-prompting in rubric-based scoring for german. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics. ↗
  33. Shashank Sonkar, Kangqi Ni, Lesa Tran Lu, Kristi Kincaid, John S Hutchinson, and Richard G Baraniuk. 2024. Automated long answer grading with ricechem dataset. In International Conference on Artificial Intelligence in Education, pages 163–176. Springer. find
  34. Tianqi Wang, Naoya Inoue, Hiroki Ouchi, Tomoya Mizumoto, and Kentaro Inui. 2019. Inject rubrics into short answer grading system. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 175–182, Hong Kong, China. Association for Computational Linguistics. find
  35. Yuchen Wei, Dennis Pearl, Matthew Beckman, and Rebecca J Passonneau. 2025. Concept-based rubrics improve LLM formative assessment and data synthesis. arXiv preprint arXiv:2504.03877. ↗
  36. Alistair Willis and Aisling Third. 2026. AMATI at BEA 2026 shared task 2: Automatic short answer grading with inductive logic programming and a large language model. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026). Association for Computational Linguistics.
  37. Tobias Wyrwich, Marcus Kubsch, Hendrik Drachsler, and Knut Neumann. 2025. Tracking students’ progression in developing understanding of energy using ai technologies. Physical Review Physics Education Research, 21(1). ↗
  38. Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard, Wei-Wei Tu, Quanming Yao, Huan Zhao, and Isabelle Guyon. 2022. Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform. Patterns, 3(7). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gombert-2026-report-bea-2026-shared-task/