Elamur Research⌕
☰
home / papers / 2026 / GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training

GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training

Summary

Large Dutch pre-training datasets with clear legal standing are hard to find. Web-derived resources such as CulturaX, FineWeb-2 and HPLT offer volume but uncertain provenance, while curated INT corpora are small or limit commercial reuse. The authors built the GPT-NL Public Corpus, the publicly releasable part of the data used to train the first GPT-NL model. They selected material from existing sets like Common Corpus and Common Crawl and created new Dutch collections, mostly through partnerships with organisations or synthetic augmentation.

Every candidate dataset was judged against three criteria:

  • usefulness for a model meant to interpret verifiable information rather than store broad knowledge,
  • licensing that permits commercial use,
  • limited harmful or biased content.

The result is described as the largest permissively licensed Dutch resource. It is redistributed under CC-BY on the Hugging Face Hub. The corpus matters because it lets Dutch-focused LLMs, including commercial ones, be trained without relying on legally ambiguous web data. That is increasingly relevant under the EU AI Act and copyright rules.

Key points

  • The corpus has 21 new Dutch-only collections with about 36B preprocessed Dutch tokens that appear in no other LLM pre-training corpus. It adds roughly 207B English, 232B code and 48B German/Danish tokens, plus some Frisian.
  • The authors treat an LLM as a potential derivative work, so they exclude CC ND, NC and SA licences. Only CC-0, CC-BY and public-domain text qualify.
  • Code is limited to four licences: Apache 2.0, MIT, BSD (2/3-Clause) and Unlicense.
  • Common Corpus holds about 2T tokens but only around 8B Dutch tokens, mostly old Royal Library material. This gap motivated the new Dutch collections.
  • German and Danish data were added because related languages can help a lower-resource language like Dutch. Code was included for its possible boost to reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5fbtc336wwx2

@inproceedings{oort2026gpt,
  title = {{GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training}},
  author = {Oort, Jesse J. Van and Brinkkemper, Frank and Graaf, Erik de and Vanroy, Bram and Lensink, Saskia},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5fbtc336wwx2},
  url = {https://doi.org/10.63317/5fbtc336wwx2}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. 2024. Phi-4 technical report. ↗ our card
  2. Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. 2025. Hallulens: Llm hallucination benchmark. ↗
  3. BBC. 2025. Ai startup anthropic agrees to pay $1.5bn to settle book piracy lawsuit. BBC News. find
  4. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, Santa Clara, CA. USENIX Association. ↗ our card
  5. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. Multieurlex – a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. ↗
  6. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024. When is multilinguality a curse? language modeling for 250 high- and low-resource languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074–4096, Miami, Florida, USA. Association for Computational Linguistics. find
  7. Common Crawl. 2025. Common Crawl: Open repository of web crawl data. https://commoncrawl.org. Accessed: 2025-10-06. ↗
  8. Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. 2024. A new massive multilingual dataset for high-performance language technologies. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1116–1128, Torino, Italia. ELRA and ICCL. find
  9. Melissa Dell, Jacob Carlson, Tom Bryan, Emily Silcock, Abhishek Arora, Zejiang Shen, Luca D’Amico-Wong, Quan Le, Pablo Querubin, and Leander Heldring. 2023. American stories: A large-scale structured text dataset of historical u.s. newspapers. find
  10. Guust Franssens. 2025. Belgian journal. find
  11. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179. ↗ our card
  12. Ivan Habernal, Omnia Zayed, and Iryna Gurevych. 2016. C4Corpus: Multilingual web-size corpus with free license. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 914–922, Portorož, Slovenia. European Language Resources Association (ELRA). find
  13. Internet Archive. 2025. Internet archive: Digital library of free & borrowable texts, movies, music & wayback machine. https://archive.org/. Accessed: 2025-10-06. ↗
  14. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics. find our card
  15. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. find
  16. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. Madlad-400: A multilingual and document-level large audited dataset. find our card
  17. Pierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, and Ivan P. Yamshchikov. 2025. Common corpus: The largest collection of ethical data for llm pre-training. ↗
  18. Danny D. Leybzon and Corentin Kervadec. 2024. Learning, forgetting, remembering: Insights from tracking LLM memorization during training. In The 7th BlackboxNLP Workshop. find
  19. Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. 2023. At which training stage does code data help llms reasoning? ↗
  20. Nature Machine Intelligence. 2025. Transparency (in training data) is what we want. Nature Machine Intelligence, 7:329. Editorial. find
  21. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4226–4237, Torino, Italia. ELRA and ICCL. find our card
  22. Nelleke Oostdijk. 2000. Het corpus gesproken nederlands. Nederlandse Taalkunde, 5(3):280–285. find
  23. Hans Paulussen, Lieve Macken, Julia Trushkina, Piet Desmet, and Willy Vandeweghe. 2006. Dutch parallel corpus: a multifunctional and multilingual corpus. Cahiers de l’Institut de Linguistique de Louvain, 32:269–285. find
  24. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all – adapting pre-training data processing to every language. ↗
  25. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand. Association for Computational Linguistics. ↗
  26. Ralf Steinberger, Andreas Eisele, Szymon Klocek, Spyridon Pilos, and Patrick Schlüter. 2012. DGT-TM: A freely available translation memory in 22 languages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 454–459, Istanbul, Turkey. European Language Resources Association (ELRA). find our card
  27. S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. ↗ our card
  28. Gertjan van Noord, Ineke Schuurman, and Vincent Vandeghinste. 2006. Syntactic annotation of large corpora in STEVIN. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/oort-2026-gpt-nl-public-corpus-permissively/