Elamur Research⌕
☰
home / papers / 2026 / BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

Paper · 2026 · Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach et al.

Summary

German legal analysis follows a fixed reasoning pattern called subsumption: state the norm to test, define its elements, map the facts onto each element, then draw the legal consequence. Existing legal NLP benchmarks mostly use short answers, common-law settings, or fixed gold labels, so they do not test this pattern. BenGER fills that gap with three German-law corpora:

  • a large set of published ZJS exam cases with expert reference solutions;
  • a small newly written Benchathon set, solved by law students and graduates either alone or with LLM help;
  • a set of short doctrinal questions.

The authors evaluated 12 LLM systems using an LLM judge guided by German grading rubrics. They checked that judge against three blind human reviews per solution and against five other LLM judges.

Closed flagship models scored best on every corpus, and human work with AI assistance beat unaided human work. The judge correlated reasonably well with human graders, and system rankings stayed stable across judges. The work matters because it treats noisy expert grading as a distribution rather than a single truth, which offers a template for evaluating open-ended reasoning.

Key points

  • The dataset holds 596 exam-style free-text case tasks (581 from ZJS, 15 from Benchathon) and 531 short Doctrinal Principles (Grundprinzipien) items. It is meant for evaluation only and has no train/test split.
  • Benchathon collected 220 human solutions, about 15 per task: 65 in the traditional condition and 155 in the human-AI co-creation condition.
  • Agreement between the judge and humans reached Pearson r=0.76 and Cohen's κ=0.60. Two judges from independent providers passed the Calderon single-reviewer replacement bar, but the authors call this a promising point estimate, not a robust result (the 95% CI is wide).
  • Case corpora are graded with a 10-dimension Falllösung rubric, whereas Doctrinal Principles is scored on four dimensions: subsumption, clarity, knowledge of the law, and whether the outcome is correct. For that reason only aggregate scores are compared across corpora.
  • The motivation comes from Hufeld's grader study: 23 graders scored 15 exams, and grades for the same answer differed by 6.47 points on average on the 0–18 scale.

Interesting ideas

  • I1. Grading is tied to a fixed reasoning scaffold, so each rubric dimension points to a specific part of the answer, and a fluent text that skips a step can still be caught.
  • I2. Human grades are treated as a spread of plausible expert opinions, and the automatic judge is acceptable if its deviation is comparable to the disagreement among human raters.
  • I3. The authors check whether the system ranking survives a swap of the judge across six LLM judges from different providers, which separates real differences from quirks of one judge.
  • I4. The authors admit that agreement between unpublished and public sets does not prove a lack of contamination, because contamination could affect all systems equally.
  • I5. To limit benchmark gaming, the authors release the rubric and judge prompts and ask for per-dimension breakdowns rather than one headline number.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Readers' explanations could be scored step by step (frame chosen, world fact cited, conclusion drawn), so an answer that is correct but missing its anchoring step is flagged.
  • I2 (learn): A parser's disagreement with silver data could be measured against the spread among the witness parsers themselves, rather than against a single "gold" tree.
  • I3 (learn): When using LLM teacher tiers, Mova's model comparisons could be rerun under several teachers, and only orderings that stay stable would be trusted.
  • I4 (learn): Negative controls should be designed so that a leak would change the ordering, not merely the absolute scores.
  • I5 (answer): School-test and fable-QA results could be reported per reasoning dimension, so that progress on one skill cannot hide a regression in another.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🎓University of KonstanzKonstanz · Germany17 works · 232 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 worksLaw · 77 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksLegal AI · 134 works

Cite

Cite as Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach, Aleyna Koçak, Anne Zettelmeier, Elly Breu, Angelina Greiner, Sofija Milijas and Matthias Grabmair. 2026. BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law. arXiv:2605.28183.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.28183 · arXiv 2605.28183

@misc{nagl2026benger,
  title = {{BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law}},
  author = {Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach, Aleyna Koccak, Anne Zettelmeier, Elly Breu, Angelina Greiner, Sofija Milijas and Matthias Grabmair},
  year = {2026},
  doi = {10.48550/arxiv.2605.28183},
  eprint = {2605.28183},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.28183}
}

References

36 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pawitsapak Akarajaradwong, Pirat Pothavorn, Chompakorn Chaksangchaichot, Panuthep Tasawong, Thi-tiwat Nopparatbundit, Keerakiat Pratai, and Sarana Nutanong. 2025. NitiBench: Benchmarking LLM frameworks on Thai legal question answering capabilities. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34304–34327, Suzhou, China. Association for Computational Linguistics. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  3. Bundesministerium der Justiz. 1981. Verordnung über eine Noten- und Punkteskala für die erste und zweite juristische Prüfung (JurPrNotSkV). BGBl. I S. 1243. Federal grading ordinance for the two German state law examinations. find
  4. Abdur-Rahman Butler and Umar Butler. 2026. Legal RAG Bench: An end-to-end benchmark for legal RAG. Preprint, arXiv:2603.01710. ↗
  5. Marius Büttner and Ivan Habernal. 2024. Answering legal questions from laymen in German civil law system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2015–2027, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  6. Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for LLM-as-a-judge: How to statistically justify replacing human annotators with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16051– 16081, Vienna, Austria. Association for Computational Linguistics.
  7. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics. ↗
  8. Odysseas S. Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androutsopoulos. 2025. GreekBar-Bench: A challenging benchmark for free-text legal reasoning and citations. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25099–25119, Suzhou, China. Association for Computational Linguistics. find
  9. Odysseas S. Chlapanis, Dimitrios Galanis, and Ion Androutsopoulos. 2024. LAR-ECHR: A new legal argument reasoning task and dataset for cases of the European Court of Human Rights. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 267–279, Miami, FL, USA. Association for Computational Linguistics. find
  10. Manan Roy Choudhury, Adithya Chandramouli, Mannan Anand, and Vivek Gupta. 2026. Better call CLAUSE: A discrepancy benchmark for auditing LLMs legal reasoning capabilities. In Findings of the Association for Computational Linguistics: EACL 2026, pages 5776–5818, Rabat, Morocco. Association for Computational Linguistics. find
  11. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  12. Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2026. LEXam: Benchmarking legal reasoning on 340 law exams. In Proceedings of the Fourteenth International Conference on Learning Representations. ArXiv:2505.12864. ↗
  13. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. LawBench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962, Miami, Florida, USA. Association for Computational Linguistics. ↗
  14. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 others. 2023. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems, volume 36, pages 44123–44279. Curran Associates, Inc.
  15. Emaan Hariri and Daniel E. Ho. 2025. AI for statutory simplification: A comprehensive state legal corpus and labor benchmark. In Proceedings of the Twentieth International Conference on Artificial Intelligence and Law, pages 177–187, Chicago, IL, USA. ACM. ArXiv:2508.19365. ↗
  16. Clemens Hufeld. 2024. Jede Korrektur eine andere Note: Quantitative Untersuchung der Objektivität juristischer Klausurbewertungen. Zeitschrift für Didaktik der Rechtswissenschaft, 11(1):59–83. ↗
  17. Zheng Jia, Shengbin Yue, Wei Chen, Siyuan Wang, Yidong Liu, Zejun Li, Yun Song, and Zhongyu Wei. 2026. Ready jurist one: Benchmarking language agents for legal intelligence in dynamic environments. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10351–10376. Association for Computational Linguistics. find
  18. Yeeun Kim, Youngrok Choi, Eunkyung Choi, JinHwan Choi, Hai Jin Park, and Wonseok Hwang. 2024. Developing a pragmatic benchmark for assessing Korean legal language understanding in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5573– 5595, Miami, Florida, USA. Association for Computational Linguistics.
  19. Terry K. Koo and Mae Y. Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2):155–163. ↗
  20. Karl Larenz and Claus-Wilhelm Canaris. 1995. Methodenlehre der Rechtswissenschaft, 3 edition. Springer, Berlin, Heidelberg. find
  21. Jihyung Lee, Daehui Kim, Seonjeong Hwang, Hyounghun Kim, and Gary Lee. 2025. KoBLEX: Open legal question answering with multi-hop reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4019–4053, Suzhou, China. Association for Computational Linguistics. find
  22. Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. 2024. LexEval: A comprehensive Chinese legal benchmark for evaluating large language models. In Advances in Neural Information Processing Systems, volume 37, pages 25061–25094. Curran Associates, Inc. find
  23. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  24. Thomas M. J. Möllers. 2023. Juristische Methodenlehre, 5 edition. C.H. Beck, München. find
  25. Sebastian Nagl and Matthias Grabmair. 2026. BenGER Platform: A collaborative web platform for end-to-end benchmarking of German legal tasks. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Law, Singapore. ACM. To appear. arXiv:2604.13583. ↗
  26. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems, volume 37, pages 68772–68802. Curran Associates, Inc. find
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics.
  28. Nicholas Pipitone and Ghita Houir Alami. 2024. LegalBench-RAG: A benchmark for retrieval-augmented generation in the legal domain. Preprint, arXiv:2408.10343. ↗
  29. Maja Popović. 2015. chrF: Character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. Association for Computational Linguistics.
  31. Yuzhen Shi, Huanghai Liu, Yiran Hu, Gaojie Song, Xinran Xu, Yubo Ma, Tianyi Tang, Li Zhang, Qingjing Chen, Di Feng, Wenbo Lv, Weiheng Wu, Kexin Yang, Sen Yang, Wei Wang, Rongyao Shi, Yuanyang Qiu, Yuemeng Qi, Jingwen Zhang, and 11 others. 2026. PLawBench: A rubric-based benchmark for evaluating LLMs in real-world legal practice. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10067–10116. Association for Computational Linguistics. find
  32. Patrick E. Shrout and Joseph L. Fleiss. 1979. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420–428. ↗
  33. Malte Weber, Balaramakrishna Paritala, Abhilash Reddy Rechu, Leila Feddoul, Suresh Kumar Bonagiri, Norman Klewer, Pirmin Mathias Karg, Christoph Unger, Marianne Mauch, and Birgitta König-Ries. 2025. GerLeRB – German legislative retrieval benchmark. In Fachtagung Rechts- und Verwaltungsinformatik (RVI 2025), Recht und Informatik, pages 157–168, Bonn. Gesellschaft für Informatik. ↗
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. ArXiv:1904.09675. ↗
  35. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578. Association for Computational Linguistics. ↗
  36. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. Curran Associates, Inc. Contents of the Appendix

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nagl-2026-benger-benchmarking-llm-systems-subsumption/