Elamur Research⌕
☰
home / papers / 2026 / ANNOTARES: A Dataset for Extracting Logical Structures from German Statutory Texts

ANNOTARES: A Dataset for Extracting Logical Structures from German Statutory Texts

Paper · 2026 · Ronja Schwarz and Jannik Strotgen

Summary

German statutes are written for human readers. Their nested, implicit phrasing makes it hard to automatically separate a norm's legal prerequisites (Tatbestand) from its legal effects (Rechtsfolge). The authors frame this separation as a three-label token tagging task: Tatbestand, Rechtsfolge, or none. They release ANNOTARES, a span-annotated corpus drawn from three federal codes.

The annotation guidelines come from the federal legislative drafting manual. They rely on recognisable linguistic cues, so annotators without legal training can apply them consistently. Two rules shape every annotation: each sentence carries one legal statement, and the two logical labels always appear together.

The authors benchmark six approaches:

  • a rule-based baseline
  • CRF
  • BiLSTM
  • BiLSTM-CRF
  • mBERT
  • LLM prompting

mBERT performs best. It also transfers well to statutes it was not trained on. Explicit POS and dependency features help the trained models considerably, but adding them to LLM prompts makes results worse. The resource matters because it turns statutory prose into condition/consequence structure. Knowledge graphs, RAG and neuro-symbolic legal reasoning systems could use that structure directly.

Key points

  • ANNOTARES covers three codes:
    • BDSG, the main corpus: over 400 sentences, of which 351 are used for training, plus a test split.
    • BAföG and BauGB: 50 sentences each, used as out-of-domain test sets.
  • The corpus has over 21,500 tokens, labelled roughly 43% Tatbestand, 44% Rechtsfolge and 13% none. Token-level inter-annotator agreement reaches Krippendorff's alpha 0.89.
  • mBERT reaches accuracy 0.854 and mF1 0.798 on BDSG. Its mF1 on the unseen statutes is higher still: 0.868 on BAföG and 0.812 on BauGB, with accuracy staying between 0.843 and 0.854.
  • Removing POS/DEP tags lowers mBERT's mF1 from 0.798 to 0.679. For BiLSTM-CRF, mF1 falls from 0.480 to 0.292 while accuracy barely changes (0.572 vs 0.582), because the models collapse toward the majority class.
  • LLMs score lower on token-level metrics but are the best at exact span matching for the none class. Prompts that include syntactic tags hurt LLM performance.

Interesting ideas

  • I1. Treat a norm as a pair of spans, condition and consequence, and make the labels mutually dependent: if one is present the other must be too, otherwise the whole sentence is none.
  • I2. Build annotation guidelines on surface linguistic cues rather than expert judgement, such as conditional conjunctions like wenn/falls/sofern and subject-position conditions.
  • I3. Report macro-F1 alongside accuracy. An ablation showed a model whose accuracy held steady while it stopped predicting the minority class altogether.
  • I4. Show that syntax matters by removing POS and dependency features and measuring the drop. The drop indicated that conditions are signalled by dependency structure more than by vocabulary.
  • I5. Train on one domain and test on separate held-out statutes. This tests whether the model learned general structural patterns rather than topic vocabulary.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A conditional frame with obligatory paired slots would let prag reject half-filled "if X then Y" readings and flag sentences with no rule at all.
  • I2 (learn): Cue-based guidelines could steer LLM teacher tiers or silver labelling of conditional structure so that disagreements stay explainable and reproducible.
  • I3 (learn): Evaluations should include per-class F1 and confusion matrices as a negative control, so that majority-class collapse cannot hide behind stable accuracy.
  • I4 (tree): This supports deriving condition/consequence boundaries from UD subtrees (e.g., advcl, mark) instead of from word lists, and offers an ablation recipe to test that choice.
  • I5 (answer): Holding out whole text genres, such as fables versus school tests, would show whether readers generalise across sources rather than memorising content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Karlsruhe University of Applied SciencesKarlsruhe · Germany4 works · 47 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPolitical Science and International Relations · 182 works

🧭 Topics 1Research subjects and languages the work is about.Legal AI · 134 works

Cite

Cite as Ronja Schwarz and Jannik Strotgen. 2026. ANNOTARES: A Dataset for Extracting Logical Structures from German Statutory Texts. arXiv:2608.03898.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.03898 · arXiv 2608.03898

@misc{schwarz2026annotares,
  title = {{ANNOTARES: A Dataset for Extracting Logical Structures from German Statutory Texts}},
  author = {Ronja Schwarz and Jannik Strotgen},
  year = {2026},
  doi = {10.48550/arxiv.2608.03898},
  eprint = {2608.03898},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.03898}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. The Claude 3 model family: Opus, Sonnet, Haiku. Technical report, Anthropic. find
  2. Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. ACM Comput. Surv., 58(6). ↗
  3. Arian Askari, Zihui Yang, Zhaochun Ren, and Suzan Verberne. 2024. Answer retrieval in legal community question answering. In Advances in Information Retrieval: 46th European Conference on Information Retrieval, ECIR 2024, Glasgow, UK, March 24–28, 2024, Proceedings, Part III, page 477–485, Berlin, Heidelberg. Springer-Verlag. find
  4. Ting Wai Terence Au, Vasileios Lampos, and Ingemar Cox. 2022. E-NER — an annotated named entity recognition corpus of legal text. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 246–255, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  5. Bundesministerium der Justiz. 2024. Handbuch der Rechtsförmlichkeit, 4 edition. Bundesministerium der Justiz. Empfehlungen des Bundesministeriums der Justiz für die rechtsförmliche Gestaltung von Gesetzen und Rechtsverordnungen nach § 42 Absatz 4 und § 62 Absatz 2 der Gemeinsamen Geschäftsordnung der Bundesministerien. Bekanntgemacht im BAnz AT 31.10.2024. find
  6. Marius Büttner and Ivan Habernal. 2024. Answering legal questions from laymen in German civil law system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2015–2027, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  7. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, Florence, Italy. Association for Computational Linguistics. on legal information, extraction/entailment (COL-IEE) 2023. The Review of Socionetwork Strategies, 18(1):27–47.
  8. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  9. Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2545–2568, Online. Association for Computational Linguistics.
  10. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  11. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  12. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904, Online. Association for Computational Linguistics. ↗
  13. Ambedkar Kanapala, Sukomal Pal, and Rajendra Pamula. 2019. Text summarization from legal documents: a survey. Artificial Intelligence Review, 51(3):371–402. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  15. Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal judgment prediction via event extraction with constraints. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 648–664, Dublin, Ireland. Association for Computational Linguistics. find
  16. Ingo Glaser, Sebastian Moser, and Florian Matthes. 2021. Sentence boundary detection in german legal documents. In Proceedings of the 13th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, pages 812–821. IN-STICC, SciTePress. find
  17. Randy Goebel, Yoshinobu Kano, Mi-Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. 2024. Overview and discussion of the competition Deepali Jain, Malaya Dutta Borah, and Anupam Biswas. 2024. A sentence is known by the company it keeps: Improving legal document summarization using deep clustering. Artificial Intelligence and Law, 32(1):165–200. find
  18. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 5–9, Santa Fe, New Mexico. Association for Computational Linguistics.
  19. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, page 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  20. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260–270, San Diego, California. Association for Computational Linguistics. ↗
  21. Elena Leitner, Georg Rehm, and Julian Moreno-Schneider. 2019. Fine-grained named entity recognition in legal documents. In Semantic Systems. The Power of AI and Knowledge Graphs. SEMANTiCS 2019, pages 272–287, Cham. Springer International Publishing. A literature review on current legal NLP approaches. IEEE Access, 13:18253–18276. find
  22. Elena Leitner, Georg Rehm, and Julian Moreno-Schneider. 2020. A dataset of German legal documents for named entity recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4478–4485, Marseille, France. European Language Resources Association. ↗
  23. Răzvan-Alexandru Smădu, Ion-Robert Dinică, Andrei-Marius Avram, Dumitru-Clementin Cercel, Florin Pop, and Mihaela-Claudia Cercel. 2022. Legal named entity recognition with multi-task domain adaptation. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 305–321, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  24. Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20):22266–22275.
  25. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. Software available from https://github.com/doccano/doccano. ↗
  26. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1659–1666, Portorož, Slovenia. European Language Resources Association (ELRA). ↗
  27. Vasile Pais, Maria Mitrofan, Carol Luca Gasan, Vlad Coneschi, and Alexandru Ianov. 2021. Named entity recognition in the Romanian legal domain. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 9–18, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  28. Subhash Chandra Pujari, Annemarie Friedrich, and Jannik Strötgen. 2021. A multi-task approach to neural multi-label hierarchical patent classification using transformers. In Advances in Information Retrieval, pages 513–528, Cham. Springer International Publishing. find
  29. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online. Association for Computational Linguistics. ↗
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  31. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. ↗
  32. Marco Siino, Mariana Falco, Daniele Croce, and Paolo Rosso. 2025. Exploring LLMs applications in law: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  33. Qiqi Wang, Kaiqi Zhao, Robert Amor, Benjamin Liu, and Ruofan Wang. 2022. D2GCLF: Document-to-graph classifier for legal document classification. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2208–2221, Seattle, United States. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schwarz-2026-annotares-dataset-extracting-logical-structures/