Elamur Research⌕
☰
home / papers / 2026 / Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring

Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring

Paper · 2026 · Jonas Kubesch, Lena Huber and Clemens Havas

Summary

The study asks whether open-weight LLMs, run locally, can grade Austrian A-level ("Zentralmatura") German exams using the official nationwide rubric. The SRDP exam committee supplied scanned exams. After removing records with incomplete grades and handwritten scripts, 101 typed, anonymised exams remained. They cover three text types: Literary Interpretation, Letter to the Editor and Commentary.

Four models were compared: DeepSeek-R1 32b, Qwen3 30b, Mixtral 8x7b and LLama3.3 70b. The authors tested:

  • zero-shot and few-shot prompting, each with and without Chain-of-Thought;
  • RAG retrieval of comparable graded essays;
  • fixed example sets chosen to cover the whole grade range.

Few-shot prompting beat both the zero-shot baseline and RAG, mainly because the models then used the full grading scale. Even so, agreement with the human grader stayed low. The best result was 40.6% on rubric sub-dimensions and 32.8% on final grades. The authors conclude that the models can follow a standardised rubric but are not reliable enough to grade real exams. They see current systems as aids for teachers, not replacements.

Key points

  • Data pipeline:
    • The SRDP provided 173 exam sets from the 2023 and 2024 main dates.
    • 28 sets lacked complete grades, and 44 handwritten scripts could not be extracted, which left 101 exams.
    • The redaction process made the PDFs non-machine-readable, so OCR artifacts were introduced.
  • Rubric structure:
    • Grades run from 1 (best) to 5 (fail).
    • K1 and K2 score Content and Structure for each of the two texts.
    • K3/1 and K3/2 score Style and Expression and Language Conventions; these two may be combined into K3.
    • The final grade is the average of the K-sections, provided all of them are positive.
  • RAG strategies tested: Best-Average-Worst, Most-similar-matches and Range-of-examples. All improved on the zero-shot baseline but still compressed grades into a narrow band.
  • Context length interacted with essay length. Task 1 (longer essays) worked better with fewer examples, and Task 2 (shorter essays) needed more. A mixed, task-specific setup gave the best scores.
  • Practical limits:
    • Adding CoT raised exact agreement but not QWK.
    • Injecting random-word "noise" documents gave no gain.
    • Full runs on one A100 sometimes took up to 24 hours, and a single grading could take over 10 minutes.
    • The ground truth came from a single human grader.

Interesting ideas

  • I1. Exact-match accuracy and QWK can move in opposite directions: CoT produced more exact hits but larger misses, so both metrics should always be reported together.
  • I2. Few-shot examples deliberately chosen to cover the whole output range worked better than retrieving the most similar items, because similarity retrieval collapsed the predictions toward the middle.
  • I3. The useful amount of context depended on input length: long inputs needed fewer examples and short inputs needed more, so a per-task context budget beat any single fixed setting.
  • I4. Injecting random-word documents, a trick reported to help elsewhere, gave no benefit here, which is a reminder to re-test borrowed tricks on one's own task.
  • I5. OCR artifacts were misread as the student's own spelling errors and distorted the scores, showing that input noise and real mistakes must be separated before judging.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluate repair rules and teacher tiers with both a hit rate and a severity-weighted score, so that gains in accuracy which hide worse errors get flagged.
  • I2 (learn): When prompting an LLM teacher for silver labels, select demonstrations that cover all label classes, not just the nearest neighbours, to avoid label collapse.
  • I3 (answer): Readers answering school tests or fable QA could scale the number of worked examples to the passage length rather than use one global setting.
  • I4 (learn): Treat random-noise context as an explicit negative control in evaluations to check that any gains come from content and not from prompt length or shape.
  • I5 (tokens): Tag tokens that look like OCR or encoding damage as a distinct noise category in the tokenizer, so later stages do not treat them as genuine errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Fachhochschule Salzburg.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Jonas Kubesch, Lena Huber and Clemens Havas. 2026. Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring. arXiv:2603.06066.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3748522.3779879 · arXiv 2603.06066

@misc{kubesch2026evaluating,
  title = {{Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring}},
  author = {Jonas Kubesch, Lena Huber and Clemens Havas},
  year = {2026},
  doi = {10.1145/3748522.3779879},
  eprint = {2603.06066},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3748522.3779879}
}

References

2 references, 2 of them with a card of ours. ↗ leads to the work.

  1. Tom Brown et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33, 1877–1901. Bundesministerium für Bildung, Wissenschaft und Forschung. 2024. Standardisierte kompetenzorientierte schriftliche reifeprüfung / reife- und diplomprü-fung / berufsreifeprüfung - 2. mai 2024. (2024). Retrieved August 27, 2025 from https://www.matura.gv.at/index.php?eID=dumpFile&t=f&f=6941&token=42 e50976a60ff1c075d870c400d6794f161edd7c. Bundesministerium für Bildung, Wissenschaft und Forschung. 2023. Standardisierte kompetenzorientierte schriftliche reifeprüfung / reife- und diplomprü-fung / berufsreifeprüfung - 5. mai 2023. (2023). Retrieved August 27, 2025 from https://www.matura.gv.at/index.php?eID=dumpFile&t=f&f=5836&token=cc 7ef226aa5da503dd0d7966e4dc901cb6ba9c35. Bundesministerium für Bildung, Wissenschaft und Forschung. 2020. Textsortenkat-alog zur srdp in der unterrichtssprache (deutsch, kroatisch, slowenisch, un-garisch). (2020). Retrieved August 27, 2025 from https://www.matura.gv.at/ind ex.php?eID=dumpFile&t=f&f=4525&token=950c7f2b86f0ebc3459c5f0aa0e04 013ab99c572. Jill Burstein. 2003. The e-rater scoring engine: automated essay scoring with natural language processing. Automated essay scoring: A cross-disciplinary perspective, 113121. Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 719–729. Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American society for information science, 41, 6, 391–407. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association ↗
  2. for computational linguistics: human language technologies, volume 1 (long and short papers), 4171–4186. Fei Dong and Yue Zhang. 2016. Automatic features for essay scoring–an empirical study. In Proceedings of the 2016 conference on empirical methods in natural language processing, 1072–1077. Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st conference on computational natural language learning (CoNLL 2017), 153–162. Scott Elliot. 2003. Intellimetric: from here to validity. Automated essay scoring: A cross-disciplinary perspective, 71–86. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. (2020). Tahereh Firoozi, Hamid Mohammadi, and Mark J Gierl. 2025. Using automated procedures to score educational essays written in three languages. Journal of Educational Measurement, 62, 1, 33–56. Ben Hamner, Jaison Morgan, Iynnvandev, Mark Shermis, and Tom Vander Ark. 2012. The hewlett foundation: automated essay scoring. (2012). Retrieved August 27, 2025 from — Attention Is All You Need https://www.kaggle.com/c/asap-aes. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. (2021). Gyeong-Geon Lee, Ehsan Latif, Xuansheng Wu, Ninghao Liu, and Xiaoming Zhai. 2024. Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence, 6, 100213. Sabrina Ludwig, Christian Mayer, Christopher Hansen, Kerstin Eilers, and Steffen Brandt. 2021. Automated essay scoring using transformer models. Psych, 3, 4, 897–915. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. (2013). Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2, 2, 100050. Frank Mußmann, Martin Riethmüller, Thomas Hardwig, Ilka Charlotte Ohms, and Stefan Klötzer. 2016. Niedersächsische arbeitszeitstudie lehrkräfte an öffentlichen schulen 2015/2016. Unter Mitarbeit von Stefan Peters, Marcel Parciak, Ilka Charlotte Ohms und Stefan Klötzer. Göttingen: Georg-August-Universität Göttingen, Kooperationsstelle Hochschulen und Gewerkschaften. Ollama. 2025. Ollama: run large language models locally. Retrieved August 27, 2025 from https://ollama.com/. Austin Pack, Alex Barrett, and Juan Escalante. 2024. Large language models and automated essay scoring of english language learner writing: insights into validity and reliability. Computers and Education: Artificial Intelligence, 6, 100234. Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47, 5, 238–243. Nils-Jonathan Schaller, Yuning Ding, Andrea Horbach, Jennifer Meyer, and Thorben Jansen. 2024. Fairness in automated essay scoring: a comparative analysis of algorithms on german learner essays from secondary education. In Proceedings of the 19th workshop on innovative use of nlp for building educational applications (bea 2024), 210–221. Kathrin Seßler, Maurice Fürstenberg, Babette Bühler, and Enkelejda Kasneci. 2025. Can ai grade your essays? a comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, 462–472. Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024. Exploring llm prompting strategies for joint essay scoring and feedback generation. (2024). Maja Stahl, Nadine Michel, Sebastian Kilsbach, Julian Schmidtke, Sara Rezat, and Henning Wachsmuth. 2024. A school student essay corpus for analyzing interactions of argumentative structure and quality. (2024). Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 conference on empirical methods in natural language processing, 1882–1891. Hugo Touvron et al. 2023. Llama: open and efficient foundation language models. (2023). Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. Changrong Xiao, Wenxing Ma, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Qi Fu. 2024. From automation to augmentation: large language models elevating essay scoring landscape. (2024). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kubesch-2026-evaluating-austrian-level-german-essays/