Elamur Research⌕
☰
home / papers / 2025 / The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models

The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models

Paper · 2025 · Lukas Gienapp, Christopher Schroder, Stefan Schweter et al.

Summary

Most text used to pretrain large language models comes from web crawls whose copyright status is unclear. Very little German text is clearly open. The authors build the German Commons, a German pretraining corpus in which every document carries an explicit open license. They started from the German part of Common Corpus and applied stricter license and quality checks, which cut it from 112B to 70B tokens. They then added newer releases and further collections from institutional providers such as national libraries, research data platforms and Wikimedia. All sources were processed with one pipeline that filters for quality (including OCR-specific checks), removes duplicates, repairs text formatting and removes personal data. The result is 154.56B tokens in 35.78M documents from 41 sources. These cover seven domains, and news and cultural material make up most of the volume. Each document records its license and provenance. The processing code is released too, so others can rebuild or extend the corpus. This makes it possible to train German models that are open in both weights and data.

Key points

  • Size: 154.56B GPT-2 tokens across 35.78M documents from 41 sources and seven domains:
    • news: 72.67B tokens, 47.02%
    • cultural: 54.49B, 35.25%
    • web: 19.89B
    • political: 3.57B
    • legal: 2.99B
    • scientific: 0.84B
    • economics: 0.11B
  • Licensing rule: the corpus follows the Open Knowledge Foundation's Open Definition 2.1.
    • Share-alike licenses (e.g. CC-BY-SA-4.0) are allowed, unlike in Open License Corpus and KL3M. This keeps Wikipedia in.
    • A collection is dropped if only the aggregate or its metadata is openly licensed but the texts themselves are not.
  • Providers include Text+, Zenodo, Huggingface, DNB, ÖNB, DWDS, IDS and Wikimedia. On Zenodo and Huggingface only uploads from trusted parties were accepted. Data was collected up to 31 August 2025.
  • Release:
    • every document is tagged with a canonical SPDX license URL;
    • the deduplication bloom filters are published;
    • the processing library llmdata is open;
    • the corpus comes with a Gebru-style datasheet.
  • Profiling with off-the-shelf classifiers:
    • Text complexity: about 65% everyday language, 30% special language, 2% plain and 3% easy.
    • Sentiment: about 80.5% neutral, 16.4% negative and 3.1% positive.
  • Stated limitations:
    • a skew toward historical text (much cultural material dates from the 18th–20th centuries);
    • OCR noise, especially with German diacritics;
    • few dialects;
    • only surface-level removal of personal data (regex and Presidio).

Interesting ideas

  • I1. Each document stores a machine-readable license identifier and its provenance, so subsets can be selected or removed later without re-auditing sources.
  • I2. The deduplication bloom filters are released so that outsiders can check whether their test data leaked into the corpus.
  • I3. A source is rejected when the license of the collection differs from, or hides, the license of its contents; open packaging alone is not enough.
  • I4. The authors profile the corpus with ready-made complexity and sentiment classifiers per domain to reveal register skew, instead of just counting tokens.
  • I5. They chose not to use LLM-based OCR correction, because generated fixes can hallucinate and misread historical text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Tag every silver sentence, treebank and teacher output with its source and license. Training mixes could then be filtered by permission, and a bad source could be removed in one step.
  • I2 (learn): Keep a bloom filter of all evaluation items (school tests, fables, word problems). Check every new silver or teacher batch against it, as a cheap negative control against contamination.
  • I3 (world): When compiling knowledge from WordNet, FrameNet and Wikidata, record whether each fact's own terms are clear. Don't assume the dump-level license covers it, so the knowledge graph stays redistributable.
  • I4 (answer): Profile reading tasks and training texts by register and complexity, so that weak answer accuracy can be traced to gaps in coverage (e.g. too little everyday narrative).
  • I5 (tokens): Keep text normalization conservative and rule-based, with each edit logged. Any fix that cannot be explained could otherwise distort tokens and lemmas without anyone noticing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Lukas Gienapp, Christopher Schroder, Stefan Schweter, Christopher Akiki, Ferdinand Schlatt, Arden Zimmermann, Phillipe Genet and Martin Potthast. 2025. The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models. arXiv:2510.13996.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.13996 · arXiv 2510.13996

@misc{gienapp2025german,
  title = {{The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models}},
  author = {Lukas Gienapp, Christopher Schroder, Stefan Schweter, Christopher Akiki, Ferdinand Schlatt, Arden Zimmermann, Phillipe Genet and Martin Potthast},
  year = {2025},
  doi = {10.48550/arxiv.2510.13996},
  eprint = {2510.13996},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.13996}
}

References

50 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Longpre et al. 2023. The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI. CoRR, abs/2310.16787. doi:10 . 48550 /ARXIV.2310.16787. ↗
  2. Kilgarriff and Grefenstette. 2003. Introduction to the Special Issue on the Web as Corpus. Comput. Linguistics, 29, 3, 333–348. doi:10.1162/089120103322711569. ↗
  3. Brin, Motwani, Page, and Winograd. 1998. What can you do with a Web in your Pocket? IEEE Data Eng. Bull., 21, 2, 37–47. find
  4. Gebru, Morgenstern, Vecchione, Vaughan, Wallach, III, and Crawford. 2021. Datasheets for datasets. Commun. ACM, 64, 12, 86–92. doi:10.1145/3458723. ↗
  5. Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li, and Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21, 140:1–140:67. ↗
  6. Xue, Constant, Roberts, Kale, Al-Rfou, Siddhant, Barua, and Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the ACL: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. ACL, 483–498. doi:10.1 8653/V1/2021.NAACL-MAIN.41. ↗
  7. Gao et al. 2021. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. CoRR, abs/2101.00027. ↗
  8. Abadji, Suárez, Romary, and Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. In Proc. of LREC. European Language Resources Association, 4344–4355.
  9. Laurençon et al. 2022. The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. ↗
  10. Weber et al. 2024. RedPajama: an Open Dataset for Training Large Language Models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.
  11. Soldaini et al. 2024. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Proceedings of the 62nd Annual Meeting of the ACL (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. ACL, 15725–15788. doi:10.18653/V1/2024.ACL-LONG.840. ↗
  12. De Gibert et al. 2024. A New Massive Multilingual Dataset for High-Performance Language Technologies. In Proc. of LREC/COLING. ELRA and ICCL, 1116–1128. ↗
  13. Penedo, Kydlícek, Allal, Lozhkov, Mitchell, Raffel, von Werra, and Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.
  14. Penedo et al. 2025. FineWeb2: One Pipeline to Scale Them All - Adapting Pre-Training Data Processing to Every Language. CoRR, abs/2506.20920. doi:10 .48550/ARXIV.2506.20920. ↗
  15. Habernal, Zayed, and Gurevych. 2016. C4Corpus: Multilingual Web-size Corpus with Free License. In Proc. of LREC. European Language Resources Association (ELRA). find
  16. II, Bommarito, and Katz. 2025. The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models. CoRR, abs/2504.07854. doi:10.4 8550/ARXIV.2504.07854. ↗
  17. Hong, Hutson, Agnew, Huda, Kohno, and Morgenstern. 2025. A Common Pool of Privacy Problems: Legal and Technical Lessons from a Large-Scale Web-Scraped Machine Learning Dataset. CoRR, abs/2506.17185. doi:10.48550 /ARXIV.2506.17185. ↗
  18. Longpre et al. 2024. A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity. In Proceedings of the 2024 Conference of the North American Chapter of the ACL: Human Language Technologies (Volume 1: Long Papers). ACL, 3245–3276. doi:10.18653/v1/2024.na acl-long.179. ↗
  19. Wang, Zhong, Wang, Zhu, Mi, Wang, Shang, Jiang, and Liu. 2024. Data Management For Training Large Language Models: A Survey. (2024). ↗
  20. Goldhahn, Eckart, and Quasthoff. 2012. Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages. In Proc. of LREC. European Language Resources Association (ELRA), 759–765.
  21. Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23-25, 2012. European Language Resources Association (ELRA), 2214–2218. find
  22. Kocetkov et al. 2023. The Stack: 3 TB of permissively licensed source code. Trans. Mach. Learn. ↗
  23. Res., 2023. Min, Gururangan, Wallace, Shi, Hajishirzi, Smith, and Zettlemoyer. 2024. SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  24. OpenReview.net. Kandpal et al. 2025. The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text. CoRR, abs/2506.05209. doi:10.48550/ARXIV.2506.05 209. [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] ↗
  25. Langlais et al. 2025. Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training. CoRR, abs/2506.01732. doi:10.48550/ARXIV.2506.01732. Nolda. Wikipedia-Korpus: Korpusquellen der deutschsprachigen Wikipedia im TEI-Format. Version 20250101. Zenodo. doi:10.5281/zenodo.14748605. Nolda. Wikivoyage-Korpus: Korpusquellen der deutschen Sprachversion von Wikivoyage im TEI-Format. Version 20250101. Zenodo. doi:10.5281/zenodo.14 748553. ↗
  26. Margaretha and Lüngen. 2014. Building Linguistic Corpora from Wikipedia Articles and Discussions. J. Lang. Technol. Comput. Linguistics, 29, 2, 59–82. ↗
  27. Schabus, Skowron, and Trapp. 2017. One Million Posts: A Data Set of German Online Discussions. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017. ACM, 1241–1244. doi:10.1145/3077136.3080711. Boenig, Haaf, Nolda, and Wiegand. Reichstagsprotokoll-Korpus. Version v1.0.2. Zenodo. doi:10.5281/zenodo.10225467. Barbaresi. German Political Speeches Corpus. Version v4.2019. Zenodo. doi:10 .5281/zenodo.3611246. Fobbe. Corpus der Drucksachen des Deutschen Bundestages (CDRS-BT). Version 2021-04-02. Zenodo. doi:10.5281/zenodo.4643066. Fobbe. Corpus der Plenarprotokolle des Deutschen Bundestages (CPP-BT). Version 2021-02-17. Zenodo. doi:10.5281/zenodo.4542662. Fobbe. Corpus des Deutschen Bundesrechts (C-DBR). Version 2025-01-07. Zenodo. doi:10.5281/zenodo.14592346. ↗
  28. Ostendorff, Blume, and Ostendorff. 2020. Towards an Open Platform for Legal Information. In JCDL ’20: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, Virtual Event, China, August 1-5, 2020. ACM, 385–388. doi:10.1145/3383583.3398616. Fobbe. Corpus der Entscheidungen des Bundesfinanzhofs (CE-BFH). Version 2025-01-14. Zenodo. doi:10.5281/zenodo.14622341. Fobbe and Swalve. Entscheidungen des Bundesgerichtshofs in Strafsachen aus dem 20. Jahrhundert (BGH-Strafsachen-20Jhd). Version 1.0.0. Zenodo. doi:10.5 281/zenodo.4540377. Fobbe. Corpus der Entscheidungen des Bundesgerichtshofs (CE-BGH). Version 2024-09-25. Zenodo. doi:10.5281/zenodo.12814022. Fobbe. Corpus der Entscheidungen des Bundesverfassungsgerichts (CE-BVerfG). Version 2024-07-24. Zenodo. doi:10.5281/zenodo.12705674. Fobbe. Corpus der Entscheidungen des Bundespatentgerichts (CE-BPatG). Version 2024-07-09. Zenodo. doi:10.5281/zenodo.10849977. Fobbe. Corpus der Entscheidungen des Bundesverwaltungsgerichts (CE-BVerwG). Version 2024-03-13. Zenodo. doi:10.5281/zenodo.10809039. Fobbe. Corpus der amtlichen Entscheidungssammlung des Bundesverfassungsgerichts (C-BVerfGE). Version 2024-03-08. Zenodo. doi:10.5281/zenodo.107831 77. Fobbe. Corpus der Entscheidungen des Bundesarbeitsgerichts (CE-BAG). Version 2020-08-28. Zenodo. doi:10.5281/zenodo.4006645. ↗
  29. Chalkidis, Fergadiotis, and Androutsopoulos. 2021. MultiEURLEX - A multilingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In ACL, 6974–6996. doi:10. 18653 / V1 /2021 . EMNLP -MAIN.559. Boenig and Hug. DiBiLit-Korpus. Version v3.0. Zenodo. doi:10.5281/zenodo.57 86725. ↗
  30. Labs. 2021. Digitised Books. c. 1510 - c. 1900. JSONL (OCR derived text + metadata). https://doi.org/10.23636/r7w6-zy15. (2021). ↗
  31. Gaido, Papi, Bentivogli, Brutti, Cettolo, Gretter, Matassoni, Nabih, and Negri. 2024. MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages. In Proc. of EMNLP. ACL, 13934– 13947. Labusch and Lehmann. Fulltexts of the Digitized Collections of the Berlin State Library (SBB). Version 1. Staatsbibliothek zu Berlin - Berlin State Library. doi:10.5281/zenodo.7716098. Nolda. Wikibooks-Korpus: Korpusquellen von deutschsprachigen Wikibooks im TEI-Format. Version 20250101. Zenodo. doi:10.5281/zenodo.14748586. ↗
  32. Hug, Kassung, and Meyer. 2010. Dingler-Online – The Digitized »Polytech-nisches Journal« on Goobi Digitization Suite. In Digital Humanities DH2010. Conference Abstracts, 311–313. find
  33. Priem, Piwowar, and Orr. 2022. OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. CoRR, abs/2205.01833. doi:1 0.48550/ARXIV.2205.01833. ↗
  34. Lee, Cooper, and Grimmelmann. 2024. Talkin’ ’Bout AI Generation: Copyright and the Generative-AI Supply Chain (The Short Version). In Proceedings of the Symposium on Computer Science and Law, CSLAW 2024, Boston, MA, USA, March 12-13, 2024. ACM, 48–63. doi:10.1145/3614407.3643696. ↗
  35. Tarkowski. 2025. Data Governance in Open Source AI. 2025. GROBID. https://github.com/kermitt2/grobid. (2025). ↗
  36. Poznanski, Borchardt, Dunkelberger, Huff, Lin, Rangapur, Wilhelm, Lo, and Soldaini. 2025. olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models. CoRR, abs/2502.18443. doi:10.48550/ARXIV.2502.18443. [56] [57] [58] [59] [60] [61] [62] [63] ↗
  37. Speer. 2019. ftfy. Zenodo. (2019). doi:10.5281/zenodo.2591652. ↗
  38. Joulin, Grave, Bojanowski, Douze, Jégou, and Mikolov. 2016. FastText.zip: Compressing text classification models. CoRR, abs/1612.03651. ↗
  39. Joulin, Grave, Bojanowski, and Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Proc. of EACL. ACL, 427–431. doi:10.18653/V1/E17-2068. ↗
  40. Radford, Wu, Child, Luan, Amodei, Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8, 9. find
  41. Scao et al. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. CoRR, abs/2211.05100. doi:10.48550/ARXIV.2211.05100. ↗
  42. Rae et al. 2021. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. CoRR, abs/2112.11446. ↗
  43. Nguyen, Nguyen, Lai, Man, Ngo, Dernoncourt, Rossi, and Nguyen. 2024. CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy. ELRA and ICCL, 4226–4237.
  44. Mendels, Peled, Vaisman Levy, Hart, Rosenthal, Lahiani, et al. 2018. Microsoft Presidio: Context aware, pluggable and customizable PII anonymization service for text and images. find
  45. Arnett, Jones, Yamshchikov, and Langlais. 2024. Toxicity of the Commons: Curating Open-Source Pre-Training Data. arXiv preprint arXiv:2410.22587. ↗
  46. Guhr, Schumann, Bahrmann, and Böhme. 2020. Training a broad-coverage german sentiment classification model for dialog systems. In Proceedings of The 12th Language Resources and Evaluation Conference. European Language Resources Association, 1620–1625. find
  47. Zhu, Chen, Gao, and Wang. 2024. Evaluating llms at evaluating temporal generalization. CoRR, abs/2405.08460. doi:10.48550/ARXIV.2405.08460. ↗
  48. Kanerva, Ledins, Käpyaho, and Ginter. 2025. OCR error post-correction with llms in historical documents: no free lunches. CoRR, abs/2502.01205. doi:10.485 50/ARXIV.2502.01205. ↗
  49. Kurpicz-Briki. 2020. Cultural differences in bias? origin and gender bias in pre-trained german and french word embeddings. In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing, SwissText/KONVENS 2020, Zurich, Switzerland, June 23-25, 2020 [online only] (CEUR Workshop Proceedings). Vol. 2624. find
  50. CEUR-WS.org. Bender and Friedman. 2018. Data statements for natural language processing: toward mitigating system bias and enabling better science. Trans. Assoc. Comput. Linguistics, 6, 587–604. doi:10.1162/TACL\_A\_00041. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gienapp-2025-german-commons-154-billion-tokens/