Elamur Research⌕
☰
home / papers / 2026 / Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law

Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law

Paper · 2026 · Ali Hamza Bashir, Muhammad Rehan Khalid, Kostadin Cvejoski et al.

Summary

General-purpose LLMs often answer German legal questions poorly, and the few existing German legal QA resources are scraped from forums, which makes their licensing uncertain. The authors avoid this by creating training data straight from statutes. Each provision of four core German codes (BGB, StGB, SGB, GG) is stored as a record of law, section number and text. GPT-4 then turns these records into German question–answer pairs. They compare two generators. One is a plain single-prompt baseline. The other is a four-level difficulty ladder that runs from simple clause lookups to scenarios combining several sections, followed by an LLM reviewer that removes weak pairs. LLaMA 3.1 (8B) and Gemma 3 (12B) are fine-tuned with LoRA. They are evaluated on four new test sets: multiple-choice and open-ended variants built on held-out BGB passages and on out-of-distribution LegalMC4 documents. The graded pipeline beats both the base models and the baseline on all of these sets, and general skills stay roughly intact. A key finding is that naive synthetic data gives no gain and can even hurt.

Key points

  • Sources: statute texts of BGB, StGB, SGB and GG, each normalized to law ID, section ID (e.g., § 433) and German text. The German part of LegalMC4 is used only to build out-of-distribution test sets.
  • The graded generator has four levels: L1 clause comprehension, L2 client-style paraphrase, L3 scenario application on one section, L4 integration across several input sections. A reviewer model drops unanswerable, unsupported or redundant pairs and deduplicates.
  • Fine-tuning uses LoRA on LLaMA 3.1 8B Instruct and Gemma 3 12B it. There are four evaluation sets: multiple-choice scored by exact accuracy, and open-ended scored by LLM-as-a-judge, each on BGB and on LegalMC4.
  • Checks for forgetting used ARC-E, ARC-C, ARC-DE and MMLU:
    • Gemma with graded data improved ARC-C (60.8 → 63.7) and ARC-E (83.7 → 85.8), with MMLU slightly down (71.6 → 71.0).
    • LLaMA shifted by under 2 points, e.g., ARC-DE 40.2 → 38.5.
  • Contamination is controlled by splitting at the passage level. Every QA pair stays traceable to a fixed source span.

Interesting ideas

  • I1. Generating questions from one source passage on an explicit difficulty ladder (lookup → paraphrase → scenario → multi-passage) yields far more useful supervision than a single open-ended "make some questions" prompt.
  • I2. A separate reviewer pass rejects generated items that the source cannot answer, that the source does not support, or that repeat earlier items.
  • I3. Every training item carries a canonical provenance record (document ID + section ID + exact text), so any example can be audited against its fixed input span.
  • I4. The evaluation pairs an in-distribution held-out split with an out-of-distribution set, uses both exact-scored multiple-choice and judge-scored open answers, and adds general benchmarks to detect forgetting.
  • I5. The finding that naive synthetic data can lower even in-distribution scores is a warning that more generated data is not automatically better.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): fable and tale QA test sets could be built per story at graded levels, so we can see exactly where the reader breaks, from single-sentence facts up to reasoning across several events.
  • I2 (learn): LLM teacher tiers producing silver questions or frames could be followed by a reviewer filter of this kind before anything enters training.
  • I3 (world): entities, states and events in the world of a text could keep pointers to their originating sentence span, so every answer can cite its evidence.
  • I4 (learn): when a component is retrained, we could always report an in-domain score, an off-domain score and an unrelated-skill control, which acts as a negative control against overfitting.
  • I5 (learn): self-play or self-induced repair data should be checked on held-out data before it is accepted, rather than added in bulk.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BonnBonn · Germany10 works · 5,751 citations🔬Fraunhofer Institute for Intelligent Analysis and Information SystemsSankt Augustin · Germany9 works · 3,875 citations🎓University of GöttingenGöttingen · Germany20 works · 158 citations🔬Lamarr Institute for Machine Learning and Artificial IntelligenceSankt Augustin · Germany6 works · 121 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPolitical Science and International Relations · 206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksLegal AI · 150 works

Cite

Cite as Ali Hamza Bashir, Muhammad Rehan Khalid, Kostadin Cvejoski, Jana Birr, Jule Berghaus, Armin Berger, Sandra Halscheidt, Christian Temath, Rafet Sifa and David Berghaus. 2026. Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law. arXiv:2601.14160.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2601.14160 · arXiv 2601.14160

@misc{bashir2026domain,
  title = {{Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law}},
  author = {Ali Hamza Bashir, Muhammad Rehan Khalid, Kostadin Cvejoski, Jana Birr, Jule Berghaus, Armin Berger, Sandra Halscheidt, Christian Temath, Rafet Sifa and David Berghaus},
  year = {2026},
  doi = {10.48550/arxiv.2601.14160},
  eprint = {2601.14160},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.14160}
}

References

2 references, 2 of them with a card of ours. ↗ leads to the work.

  1. [Bell et al., 2024] Thiago Bell, David Leonhard, Ali Hamza Bashir, Tim Dilmaghani, Mohamed Khaled, Ulrich Warning, Rüdiger Loitz, Sandra Halscheidt, Jana Birr, Armin Berger, Rafet Sifa, and David Berghaus. Fine-tuning large language models for compliance checks. In 2024 IEEE International Conference on Big Data (BigData), pages 8790–8792, 2024. [Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning (ICML), pages 41–48, 2009. [Berghaus et al., 2025a] David Berghaus, Kostadin Cvejoski, Patrick Seifner, César Ojeda, and Ramsés J. Sanchez. Foundation inference models for markov jump processes. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2025. [Berghaus et al., 2025b] David Berghaus, Patrick Seifner, Kostadin Cvejoski, César Ojeda, and Ramsés J. Sánchez. In-context learning of temporal point processes with foundation inference models. arXiv preprint arXiv:2509.24762, 2025. [Büttner and Habernal, 2024] Marius Büttner and Ivan Habernal. Answering legal questions from laymen in german civil law system. In Proceedings of the 18th Conference of the European Chapter of the ACL (EACL), pages 2015–2027, 2024. [Chalkidis et al., 2020] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. Legal-bert: The muppets straight out of law school, 2020. [Chalkidis et al., 2022] Ilias Chalkidis, Abhik Jana, Daniel Hartung, and Ion Androutsopoulos. Lexglue: A benchmark dataset for legal language understanding in english, 2022. [Chen et al., 2023] Zekun Chen, Zhengdong Liu, Chi Feng Ma, and Benfeng Chen. Lawyergpt: A legal large language model with enhanced retrieval ability. In Proceedings of the 36th Conference on Legal Knowledge and Information Systems (JURIX 2023), 2023. To appear. [Cheng et al., 2024] Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models to domains via reading comprehension. In International Conference on Learning Representations (ICLR), 2024. [Clark and others, 2018] Peter Clark et al. Ai2 reasoning challenge (arc), 2018. [Dahl et al., 2024] Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93, 2024. [European Language Resources Association (ELRA), 2023] European Language Resources Association (ELRA). Legalmc4: Filtered multilingual c4 corpus for legal nlp, 2023. Dataset. ↗
  2. [Gao et al., 2024] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. [Guo and et al., 2023] Zhipeng Guo and et al. Chatlaw: Open-source chinese legal large language model, 2023. [Hendrycks et al., 2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. [Hillebrand et al., 2024] Lars Hillebrand, Armin Berger, Daniel Uedelhoven, David Berghaus, Ulrich Warning, Tim Dilmaghani, Bernd Kliem, Thomas Schmid, Rüdiger Loitz, and Rafet Sifa. Advancing risk and quality assurance: A rag chatbot for improved regulatory compliance. In 2024 IEEE International Conference on Big Data (Big-Data), pages 8668–8670, 2024. [Hollmann et al., 2023] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations 2023, 2023. [Hollmann et al., 2025] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 01 2025. [Hu et al., 2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. [Hu et al., 2025] Yinghao Hu, Leilei Gan, Wenyi Xiao, Kun Kuang, and Fei Wu. Fine-tuning large language models for improving factuality in legal question answering. In Proceedings of the 31st International Conference on Computational Linguistics (COLING), pages 4410–4427, 2025. [Huang et al., 2023] Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. Lawyer llama: Empowering large language models in legal domain, 2023. [Liu et al., 2023] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. Association for Computational Linguistics, 2023. [Niklaus et al., 2023] Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. LEXTREME: A multi-lingual and multi-task benchmark for the legal domain. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3016–3054. Association for Computational Linguistics, 2023. [Seifner et al., 2025] Patrick Seifner, Kostadin Cvejoski, David Berghaus, Cesar Ojeda, and Ramses J Sanchez. In-context learning of stochastic differential equations with foundation inference models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Song, 2023] Pengxiao Song. Lawgpt: A domain-specific legal language model, 2023. [Taori and et al., 2023] Rohan Taori and et al. Alpaca: A strong, replicable instruction-following model. Technical report, Stanford Center for Research on Foundation Models (CRFM), 2023. [Tobin et al., 2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30, 2017. [Wang et al., 2023] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484– 13508. Association for Computational Linguistics, 2023. [Xu et al., 2023] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. [Zheng and others, 2023] Chaojun Zheng et al. Legalbench: A collaboratively built benchmark for legal reasoning in llms, 2023. [Zheng et al., 2021] Chaojun Zheng, Yixin Guo, Daniel M. Katz, Michael A. Livermore, and L. Thorne McCarty. When does pretraining help? assessing self-supervised learning for law and the case of caselaw, 2021. [Zheng et al., 2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-asa-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bashir-2026-domain-adaptation-through-synthetic-data/