Elamur Research⌕
☰
home / papers / 2026 / AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model

AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model

Paper · 2026 · Alistair Willis and Aisling Third

Summary

The paper presents AMATI's system for the BEA 2026 shared task, which targets rubric-based short answer scoring in German under the unseen-answers setting. The authors build a neuro-symbolic grader. Its aim is to pair interpretable grading rules with the tolerance that LLMs show for messy student language. Questions, rubrics and responses are lemmatised with spaCy and encoded as logic facts. Grading rules are then induced per question in an Inductive Logic Programming style, with the clingo Answer Set Programming solver choosing among a small set of rule templates. A greedy covering loop then assembles the selected rules into an ordered theory. This theory is placed in the prompt for Mistral-large, alongside the training answers. Adding the rules improved the 3-way (correct / partially correct / incorrect) results, raising QWK from 0.721 to 0.749. The 2-way results did not change. The authors stress that their main goal is explainable, editable grading for educators, not leaderboard performance.

Key points

  • Encoding: Lemmas come from spaCy de_core_news_md and are stored as lemma_in_doc(Doc, Position, Lemma) facts. Each response is linked to its gold grade by a label fact.

  • Rule templates: Six templates are used. They cover single lemmas, lemma pairs (optionally ordered) and "key information" (Pandey 2022), meaning terms found in the rubric but absent from the question.

  • Rule induction: clingo optimises compression to pick each rule, and covered responses are removed after every step. When applied, the first matching rule sets the grade, and the default is incorrect.

  • Prompt variants and results: Four prompt variants were tried on trial data, and the "Test ILP" variants performed worse. In the final evaluation, QWK was:

    TaskRules onlyMistralMistral + Theory
    2-way0.4890.6440.644
    3-way0.6140.7210.749

  • Ranking, limitations and code: The team placed 6th of 9 in the 2-way task and 5th of 8 in the 3-way task. The authors note that English prompts were used for German data, each experiment was run once, and the templates are lexically limited. Code is on GitHub (AMATI_BEA_submission).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The Open UniversityMilton Keynes · United Kingdom10 works · 338 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Alistair Willis and Aisling Third. 2026. AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.89

@inproceedings{willis2026amati,
  title = {{AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model}},
  author = {Alistair Willis and Aisling Third},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.89},
  url = {https://doi.org/10.18653/v1/2026.bea-1.89}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dishank Aggarwal, Pritam Sil, Bhaskaran Raman, and Pushpak Bhattacharyya. 2025. “I Understand Why I Got This Grade”: Automatic Short Answer Grading (ASAG) with Feedback. In Artificial Intelligence in Education, pages 304–318, Cham. Springer Nature Switzerland. find
  2. Bikram Pratim Bhuyan, Amar Ramdane-Cherif, Ravi Tomar, and T. P. Singh. 2024. Neuro-symbolic artificial intelligence: a survey. Neural Computing and Applications, 36(21):12809–12844. find
  3. Ian Clark. 2012. Formative Assessment: Assessment Is for Self-regulated Learning. Educational Psychology Review, 24(2):205–249. find
  4. Andrew Cropper, Sebastijan Dumančić, Richard Evans, and Stephen H. Muggleton. 2022. Inductive logic programming at 30. Machine Learning, 111(1):147– 172. ArXiv: 2102.10556 ISBN: 0123456789. — [arXiv:2102.10556](https://arxiv.org/abs/2102.10556) find
  5. Jiaxin Cui and Rohan Alexander. 2026. Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs. ArXiv: 2602.14349. — [arXiv:2602.14349](https://arxiv.org/abs/2602.14349) find
  6. Marius-Constantin Dinu, Claudiu Leoveanu-Condrei, Markus Holzleitner, Werner Zellinger, and Sepp Hochreiter. 2024. SymbolicAI: A framework for logic-based approaches combining generative models and solvers. _eprint: 2402.00854. find
  7. Myroslava O. Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Second Joint Conference on Lexica…
  8. Martin Gebser, Roland Kaminski, Benjamin Kaufmann, and Torsten Schaub. 2019. Multi-shot ASP solving with clingo. Theory and Practice of Logic Programming, 19(1):27–82. find
  9. Sebastian Gombert, Zhifan Sun, Fabian Zehner, Jannik Lossjew, Tobias Wyrwich, Berrit Katharina Czinczel, David Bednorz, Sascha Bernholt, Knut Neumann, Ute Harms, Aiso Heinze, and Hendrik Drachsler. 2026. Report on the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German. In Proceedin…
  10. Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. 2024. Evaluating the Elementary Multilingual Capabilities of Large Language Models with \<span style="font-variant:smallcaps;"\>MultiQ\</span\>. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4476–4494. find
  11. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  12. Mark Law, Alessandra Russo, and Krysia Broda. 2014. Inductive Learning of Answer Set Programs. In European Conference on Logics in Artificial Intelligence (JELIA). ArXiv: 1608.01946. — [arXiv:1608.01946](https://arxiv.org/abs/1608.01946) find
  13. Joan Lu, Bhavya Krishna Balasubramanian, Mike Joy, and Qiang Xu. 2025. Survey and Analysis for the Challenges in Computer Science to the Automation of Grading Systems. ACM Computing Surveys, 58(1). find
  14. Mistral AI. 2024. Mistral Large 3 Model Card. find
  15. Stephen Muggleton. 1995. Inverse entailment and Progol. New Generation Computing, 13(3-4):245–286. find
  16. Stephen Muggleton and Cao Feng. 1990. Efficient Induction of Logic Programs. New Generation Computing, pages 368–381. ISBN: 354063875X. find
  17. Suraj Jung Pandey. 2022. Modelling Alignment and Key Information for Automatic Grading. PhD Thesis, The Open University. find
  18. J R Quinlan. 1990. Learning logical definition from Relations. Machine Learning, 5(1990):239–266. find
  19. Ayush Thakur. 2025. Eisvogel: Evaluating German Language Proficiency. find
  20. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482.
  21. Alistair Willis. 2015. Using NLP to Support Scalable Assessment of Short Free Text Responses. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 243–253, Denver, Colorado. Association for Computational Linguistics. find
  22. Xiao-Wen Yang, Jie-Jing Shao, Lan-Zhe Guo, Bo-Wen Zhang, Zhi Zhou, Lin-Han Jia, Wang-Zhou Dai, and Yu-Feng Li. 2025. Neuro-Symbolic Artificial Intelligence: Towards Improving the Reasoning Abilities of Large Language Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artif… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/willis-2026-amati-bea-2026-shared-task/