Classifying German Language Proficiency Levels Using Large Language Models

Paper · 2025 · Elias-Leander Ahlers; Witold Brunsmann; and Malte Schilling

Summary

The paper looks at whether large language models can automatically assign German learner texts to the six CEFR proficiency levels (A1–C2). German CEFR data is scarce and skewed toward the middle levels, so the authors first build a balanced set of 1,567 texts. It merges three Falko subcorpora and the MERLIN corpus, and fills the missing A1 range with synthetic texts written by Claude 3.5 Sonnet. They then compare three ways of using LLaMA-3-8B-Instruct:

several prompting strategies,
supervised fine-tuning,
a probing classifier trained on the model's final-layer activations.

Fine-tuning works best. It reaches a weighted F1 of 0.769, above the 0.702 reported earlier by Caines & Buttery with hand-engineered features. Its errors are almost always one level off, which counts as near-misses on an ordinal scale. Probing helps compared with prompting alone but stays behind fine-tuning. The work matters because it replaces expert feature engineering with a scalable pipeline. The classifier could also check generated texts that are meant to sit at a given level.

Key points

Dataset: 1,567 German texts covering A1–C2, drawn from four sources.
- Falko EssayL2 (B2–C2).
- Falko SummaryL1, native-speaker summaries treated as C2.
- Falko SummaryL2.
- MERLIN, which is strongest at A2–B2.
- 122 synthetic A1 texts, each checked by hand.
Falko label mapping: C-test scores were converted to CEFR levels using the Falko authors' scheme: 60–79 → B2, 80–89 → C1, 90–100 → C2.
Synthetic data prompts: These were written in German, included Goethe Institute A1 criteria, and steered away from childish or primary-school topics. Earlier drafts had produced texts that were too simple and thematically biased.
Fine-tuned LLaMA-3-8B-Instruct results:
- Weighted F1 of 0.769.
- Mean classification distance of 0.233.
- 100% "group accuracy," where an adjacent-level prediction counts as partially correct.
Per-level F1: A1 0.857, C2 0.837 (with precision 1.000), B2 0.681 (the hardest level).

Interesting ideas

I1. On an ordered label scale, the authors report how far wrong predictions land (mean level distance, adjacent-level accuracy) alongside exact-match scores, so near-misses are separated from gross errors.
I2. A missing region of the label space was filled with LLM-written samples. The prompts were in the target language, built on official level descriptors, revised to avoid stereotyped topics, and every output was reviewed manually.
I3. Corpora with different annotation schemes were merged by converting a continuous score into discrete labels, using the mapping published by the source corpus's own authors.
I4. A small classifier trained on frozen final-layer states serves as a middle option between prompting and full fine-tuning. Comparing all three reveals how much task information the untuned model already encodes.
I5. The authors suggest using the trained level classifier as an automatic validator for texts generated or rewritten to hit a target level.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

I1 (learn): Graded outputs such as confidence tiers or difficulty grades on school-course tests could be scored by error distance, so that small slips are not lumped together with serious failures.
I2 (learn): LLM teacher tiers could generate examples of rare constructions in the same way, with explicit criteria in the prompt, a check for topical bias, and a human or witness-parser review before the examples enter silver data.
I3 (tree): When combining treebanks or converting UD dialects, relying on documented, source-sanctioned mapping tables keeps the merged labels defensible and explainable.
I4 (tree): Probing gparse's hidden states for specific properties, then comparing against a fine-tuned head, could show whether a parsing error comes from missing information or from the decoding step.
I5 (back): A classifier like this could check whether text regenerated from the meaning graph meets a requested register or simplicity level before it is accepted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields, topics and concepts

📚 Shelves 4Computer Science · 16,667 works Artificial Intelligence · 16,194 works Arts and Humanities · 16,039 works Language and Linguistics · 15,891 works

Cite

Cite as Elias-Leander Ahlers; Witold Brunsmann; and Malte Schilling. 2025. Classifying German Language Proficiency Levels Using Large Language Models. arXiv:2512.06483.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2512.06483 · arXiv 2512.06483

@misc{brunsmann2025classifying,
  title = {{Classifying German Language Proficiency Levels Using Large Language Models}},
  author = {Elias-Leander Ahlers; Witold Brunsmann; and Malte Schilling},
  year = {2025},
  doi = {10.48550/arxiv.2512.06483},
  eprint = {2512.06483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2512.06483}
}

References

6 references, 5 of them with a card of ours. ↗ leads to the work.

Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/methods.html. Anthropic. Claude 3.5 sonnet model card and announcement. https://www.anthropic.com/news/claude-3-5-sonnet, June 2024. Model card with performance benchmarks and capabilities. Amos Azaria and Tom Mitchell. The internal state of an llm knows when it’s lying. arXiv preprint arXiv:2304.13734, 2023. Adriane Boyd, Jirka Hana, Lionel Nicolas, Detmar Meurers, Katrin Wisniewski, Andrea Abel, Karin Schöne, Barbora Stindlová, and Chiara Vettori. The MERLIN corpus: Learner Language and the CEFR. In LREC, pages 1281–1288. ↗
Reykjavik, Iceland, 2014. Andrew Caines and Paula Buttery. REPROLANG 2020: Automatic proficiency scoring of Czech, English, German, Italian, and Spanish learner essays. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5614–5623, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.689. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, ..., and the Llama Team, AI@Meta. The Llama 3 Herd of Models. CoRR, abs/2407.21783, July 2024. URL https://arxiv.org/abs/2407.21783. ↗
Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen. Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database(EFCamDat). In Proceedings from 31st Second Language Research Forum, 2014. URL https://api.semanticscholar.org/CorpusID:37833484. Goethe Institut. Deutschkurse und Prüfungen. https://www.goethe.de/de/spr/kur/stu.html, 2020. Accessed on 2024-09-09. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685, June 2021. doi: 10.48550/ARXIV.2106.09685. Zhen Huang, Zengzhi Wang, Shijie Xia, and Pengfei Liu. Olympic Arena Medal Ranks: Who Is the Most Intelligent AI So Far? CoRR, abs/2406.16772, June 2024. doi: 10.48550/arXiv.2406.16772. URL https://arxiv.org/abs/2406.16772. Hugging Face. ↗
Transformers, 2025. URL https://github.com/huggingface/transformers. Accessed on 11.09.2025. Sebastian Kahl, Felix Löffler, Martin Maciol, Fabian Ridder, Marius Schmitz, Jennifer Spanagel, Jens Wienkamp, Christopher Burgahn, and Malte Schilling. Evaluating the impact of advanced LLM techniques on AI-lecture tutors for a robotics course. arXiv preprint arXiv:2408.04645, 2024. Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, and Graham Neubig. Evaluating language models as synthetic data generators, 2024. URL https://arxiv.org/abs/2412.03679. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of a large language model. ↗
Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2025. URL https://arxiv.org/abs/2402.06196. Dojun Park, Jiwoo Lee, Seohyun Park, Hyeyun Jeong, Youngeun Koo, Soonha Hwang, Seonwoo Park, and Sungeun Lee. Multiprageval: Multilingual pragmatic evaluation of large language models, 2024. URL https://arxiv.org/abs/2406.07736. Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1493. URL https://aclanthology.org/P19-1493/. Marc Reznicek, Maik Walter, Karin Schmidt, Anke Lüdeling, Hagen Hirschmann, Cedric Krummes, and Torsten Andreas. Das Falko-Handbuch: Korpusaufbau und Annotationen. Institut für deutsche Sprache und Linguistik, Humboldt-Universität zu Berlin, Berlin, 2010. Fabian Ridder and Malte Schilling. The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM’s Internal States, 2024a. Fabian Ridder and Malte Schilling. The hallurag dataset: Detecting closed-domain hallucinations in rag applications using an llm’s internal states. arXiv preprint arXiv:2412.17056, 2024b. Weihang Su, Changyue Wang, Qingyao Ai, Yiran HU, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models, 2024. Edit Szuügyi, Soören Etler, Andrew Beaton, and Manfred Stede. Automated assessment of Language Proficiency on German Data. KONVENS 2019, 2019. Satoru Uchida, Yuki Arase, and Tomoyuki Kajiwara. Profiling English sentences based on CEFR levels. ITL-International Journal of Applied Linguistics (Belgium), pages 103–126, March 2024. doi: 10.1075/itl.22018.uch. Sowmya Vajjala and Taraka Rama. Experiments with universal CEFR classification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2018), pages 147–153, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-0515. URL https://aclanthology.org/W18-0515/. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023. URL https://arxiv.org/abs/2302.11382. ↗
Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. Retrieval-augmented generation with knowledge graphs for customer service question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, pages 2905–2909, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314. doi: 10.1145/3626772.3661370. URL https://doi.org/10.1145/3626772.3661370. ↗

Updated 30 October 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahlers-2025-classifying-german-language-proficiency-levels/