Elamur Research⌕
☰
home / papers / 2026 / Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation

Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation

Paper · 2026 · Thomas F Burns, Letitia Parcalabescu, Stephan Waeldchen et al.

Summary

German has far less high-quality web text than English, and FineWeb2, the main heuristically filtered multilingual corpus, gives German only a small share. The authors build a German curation pipeline that combines rule-based and model-based filtering with LLM-generated text grounded in real web documents. The result is Aleph-Alpha-GermanWeb. The three parts add up to 642B words, though the abstract reports 628B:

  • Filtered Common Crawl: freshly processed Common Crawl not already in FineWeb2 (78B words).
  • Re-scored FineWeb2: FineWeb2 sorted into quality buckets (235B words).
  • Synthetic: a subset generated from organic web text (329B words).

To evaluate the data, they train two models from scratch on each subset separately, under the same conditions as FineWeb2 baselines:

  • a 1B Llama-style model;
  • an 8B tokeniser-free hierarchical autoregressive transformer (HAT).

On German benchmarks such as MMMLU, every subset beats FineWeb2. At 8B, the synthetic subset also beats a FineWeb2 mix enriched with curated sources like Wikipedia. This adds evidence that careful curation and grounded synthetic text can help languages with little data. The dataset is public on Hugging Face.

Key points

  • Source and scale: six Common Crawl dumps (September 2024 – February 2025) were reduced from about 10.88 billion documents to about 151.6 million.
  • Filtering steps:
    • URL blocklist;
    • Resiliparse text extraction instead of Trafilatura;
    • fastText language ID;
    • FineWeb2 repetition thresholds for German (for example, dropping documents with more than 28.2% duplicate lines);
    • exact and fuzzy deduplication.
  • Stricter line rules: instead of correcting bad lines, whole documents are dropped when numbers, uppercase lines, very short lines or boilerplate exceed set limits. All of this runs in NeMo Curator.
  • Evaluation models: a 1B Llama-style model and an 8B HAT, which splits text at byte level. At 8B, 1,000 steps correspond to roughly 3B words.
  • Caution on translation: the authors manually checked English→German machine translation, found many errors, and warn against large volumes of translated pre-training data. They also note open questions: model collapse, and whether synthetic data partly "games" benchmarks.

Interesting ideas

  • I1. Using a tokeniser-free HAT makes dataset comparisons fairer, because no learned tokeniser favours data resembling its own training corpus; any evaluator tied to one candidate's distribution biases the comparison.
  • I2. Trusted sources like Wikipedia are removed at the URL stage on purpose, so they can be mixed back later in deliberate, controlled ratios rather than leaking in uncontrolled.
  • I3. A cheap sanity test keeps a document only if at least two very common function words of the language appear, which catches wrong-language or garbled text at almost no cost.
  • I4. Each data subset trains its own model under identical settings, with no mixing, so any gain can be traced to a specific source.
  • I5. Manual inspection showed machine translation was too error-prone to trust in bulk, a reminder to audit generated text before training on it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when comparing silver corpora or treebank variants, score them with a parser or tagger not trained on either candidate, so the test does not favour one side.
  • I2 (learn): keep gold UD treebanks separate from crawled silver text, then add gold back at explicit proportions in training runs.
  • I3 (tokens): run a quick function-word check before tagging, to flag input that is not real English prose before it reaches the parser.
  • I4 (learn): ablate each witness parser's silver output or each teacher tier on its own, to see which source actually improves LAS.
  • I5 (back): check translations produced through the meta-world with a round-trip parse and graph comparison before reusing them as training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Thomas F Burns, Letitia Parcalabescu, Stephan Waeldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren and Björn Deiseroth. 2026. Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.58

@inproceedings{burns2026aleph,
  title = {{Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation}},
  author = {Thomas F Burns, Letitia Parcalabescu, Stephan Waeldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren and Björn Deiseroth},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.58},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.58}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. 2023. Semdedup: Data-efficient learning at web-scale through semantic deduplication. Preprint, arXiv:2303.09540. ↗
  2. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, and 1 others. 2024. Tokenizer choice for llm training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907–3924.
  3. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, and 3 others. 2025. Smollm2: When smol goes big – data-centric training of a small language model. Preprint, arXiv:2502.02737. ↗
  4. Aleph Alpha. 2024. Pharia-1-llm-7b-control. https://huggingface.co/Aleph-Alpha/ Pharia-1-LLM-7B-control. Accessed: 2025-04-17. ↗
  5. Adrien Barbaresi. 2021. Trafilatura: A web scraping library and command-line tool for text discovery and extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 122–131, Online. Association for Computational Linguistics. ↗
  6. Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2018. Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl. In Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018), Lecture Notes in Computer Science, Berlin Heidelberg New York. Springer. find
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. ↗
  8. A.Z. Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pages 21–29. find
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1. ↗
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300. ↗
  12. Harm De Vries. 2023. Go smol or go home, 2023. URL https://www.harmdevries. com/post/model-size-vs-compute-overhead. ↗
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  14. Peng Di, Jianguo Li, Hang Yu, Wei Jiang, Wenting Cai, Yang Cao, Chaoyu Chen, Dajun Chen, Hongwei Chen, Liang Chen, Gang Fan, Jie Gong, Zi Gong, Wen Hu, Tingting Guo, Zhichao Lei, Ting Li, Zheng Li, Ming Liang, and 19 others. 2024. Codefuse-13b: A pretrained multi-lingual code large language model. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’24, page 418–429, New York, NY, USA. Association for Computing Machinery. Common Crawl Foundation. Common crawl. find
  15. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, and 3 others. 2022. Training compute-optimal large language models. Preprint, arXiv:2203.15556. ↗
  16. Chandra Irugalbandara, Ashish Mahendra, Roland Day-nauth, Tharuka Kasthuri Arachchige, Jayanaka Dan-tanarayana, Krisztian Flautner, Lingjia Tang, Yiping Kang, and Jason Mars. 2024. Scaling down to scale up: A cost-benefit analysis of replacing openai’s llm with open source slms in production. In 2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 280–291. Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Shrimai Prabhumoye, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ryan Wolf, Sarah Yurick, and Varun Singh. NeMo-Curator: a toolkit for data curation. find
  17. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. Preprint, arXiv:2107.06499. ↗
  18. Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. 2024. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. Preprint, arXiv:2404.01413. ↗
  19. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, and 40 others. 2024. Datacomp-lm: In search of the next generation of training sets for language models. Preprint, arXiv:2406.11794. ↗
  20. Aristides Gionis, Piotr Indyk, Rajeev Motwani, and 1 others. 1999. Similarity search in high dimensions via hashing. In Vldb, volume 99, pages 518–529. find
  21. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. Preprint, arXiv:2109.07958. ↗
  22. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  23. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. Preprint, arXiv:2406.15126. ↗
  24. Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. 2024. The larger the better? improved LLM code-generation via budget reallocation. In First Conference on Language Modeling. ↗
  25. Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. 2023. When less is more: Investigating data pruning for pre-training llms at scale. Preprint, arXiv:2309.04564. ↗
  26. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. Preprint, arXiv:2401.16380. ↗
  27. Prasanna Mayilvahanan, Thaddäus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Brendel. 2025. Llms on the line: Data determines loss-to-loss scaling laws. arXiv preprint arXiv:2502.12120. ↗
  28. Björn Plüster. Germanbenchmark: Translating popular llm benchmarks to german. https://github.com/ bjoernpl/GermanBenchmark. Accessed: 2025-04-17. ↗
  29. Bettina Messmer, Vinko Sabolčec, and Martin Jaggi. 2025. Enhancing multilingual llm pretraining with model-based data selection. Preprint, arXiv:2502.10361. ↗
  30. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, and 61 others. 2022. Scaling language models: Methods, analysis & insights from training gopher. Preprint, arXiv:2112.11446. ↗
  31. Joss Moorkens, Andy Way, and Séamus Lankford. 2024. Automating Translation. Routledge. find
  32. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models. Preprint, arXiv:2305.16264. ↗
  33. Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg, and Lukas Balles. 2025a. Hierarchical autoregressive transformers for tokenizer-free language modelling. In The Thirteenth International Conference on Learning Representations. find
  34. Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg, and Lukas Balles. 2025b. Hierarchical autoregressive transformers: Combining byte- and word-level processing for robust, adaptable language models. Preprint, arXiv:2501.10322. ↗
  35. Binh-Nguyen Nguyen and Yang He. 2025. Swift cross-dataset pruning: Enhancing fine-tuning efficiency in natural language understanding. Preprint, arXiv:2501.02432. ↗
  36. Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, and 1 others. 2024. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09871. ↗
  37. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Second Conference on Language Modeling. ↗
  38. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, volume 37, pages 30811–30849. Curran Associates, Inc.
  39. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. Preprint, arXiv:2306.01116. ↗
  40. Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. 2023. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9.
  41. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755–759. ↗
  42. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. Preprint, arXiv:2201.11990. ↗
  43. Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. 2023. Beyond neural scaling laws: beating power law scaling via data pruning. Preprint, arXiv:2206.14486. ↗
  44. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024a. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. Preprint, arXiv:2412.02595. ↗
  45. Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. 2024b. Unraveling the mystery of scaling laws: Part i. Preprint, arXiv:2403.06563. ↗
  46. Richard Sutton. 2019. The bitter lesson. Incomplete Ideas (blog), 13(1):38. find
  47. Dixuan Wang, Yanda Li, Junyuan Jiang, Zepeng Ding, Guochao Jiang, Jiaqing Liang, and Deqing Yang. 2024. Tokenization matters! degrading large language models through challenging their tokenization. arXiv preprint arXiv:2405.17067. ↗
  48. Jiayi Wang, Yao Lu, Maurice Weber, Max Ryabinin, David Adelani, Yihong Chen, Raphael Tang, and Pontus Stenetorp. 2025. Multilingual language model pretraining using machine-translated data. Preprint, arXiv:2502.13252. ↗
  49. A Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Language Resources Association.
  50. Jin Yang, Zhiqiang Wang, Yanbin Lin, and Zunduo Zhao. 2024. Problematic tokens: Tokenizer bias in large language models. In 2024 IEEE International Conference on Big Data (BigData), pages 6387–6393. IEEE. find
  51. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. Limo: Less is more for reasoning. Preprint, arXiv:2502.03387. ↗
  52. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. Prompt for LLM-as-a-judge in quality filtering Educational filter prompt (1/2) – LLM-as-a-judge SYSTEM: Deine Aufgabe ist es zu bewerten, wie gut die deutsche Sprachqualität eines deutschen Textauszugs ist. Gebe deine Bewertung in folgendem JSON-Format: { "content_criticism": str (Falls es Schwächen der Antwort im Bezug auf Inhalt gibt, zum Beispiel plötzlicher Themenwechsel, unklarer Sinn, oder fehlende Informationen, nenne und belege sie anhand von Beispielen.), "content_grade": number (Eine Zahl zwischen 1 und 5, wobei 5 die beste und 1 die schlechteste Bewertung ist: 5: Sehr kohärent und informativ, sehr hohe Qualität. 4: Kohärent und informativ. 3: Kleine Schächen im Inhalt, aber der Sinn kann erfasst werden. 2: Begrenzte Relevanz oder Genauigkeit, der Sinn kann nicht ganz erfasst werden. 1: Völlig inkonsistent oder unsinnig. ), { "language_criticism": str (Falls es Schwächen der Antwort im Bezug auf Sprache gibt, zum Beispiel Mischung verschiedener Sprachen, Schimpfwörter, unübliche Wortwahl, nenne und belege sie anhand von Beispielen.), "language_grade": number (Eine Zahl zwischen 1 und 5, wobei 5 die beste und 1 die schlechteste Bewertung ist: 5: Sehr förmliche, objektive deutsche Sprache. 4: Wenige, vernachlässigbare Sprachfehler. Educational filter – prompt (2/2) continued LLM-as-a-judge ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/burns-2026-aleph-alpha-germanweb-improving-german/