Elamur Research⌕
☰
home / papers / 2026 / Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

Paper · 2026 · Ansar Aynetdinov, Patrick Haller and Alan Akbik

Summary

High-resource languages other than English, such as German, have web corpora of hundreds of billions of tokens, not trillions. Strict quality filtering therefore leaves a small pool. The authors ask whether it is better to train once over a large, lightly cleaned German corpus or to repeat a small, carefully selected one for several epochs. Starting from roughly 500M documents in the German part of FineWeb-2, they build three nested document classifiers: Coherence, Information Value and Educational Quality. Their intersection forms a "Dense Core" of about 28B tokens. They train German models from scratch at several sizes and token budgets and compare multi-epoch runs on the filtered sets with single passes over bigger, noisier data. Repeating the high-quality data wins consistently, and the advantage is still present after 7 epochs. The benefit also carries over to instruction-tuned assistants. The authors release the BOLDT models (up to 1B parameters) and corrected German benchmarks. They report state-of-the-art results for this size while using 10–360x fewer training tokens than comparable models.

Key points

  • Source pool: FW2-DE with 496M documents. Retention by tier:
    • Coherence keeps about 61% of documents (a 100B-token sample is used in experiments).
    • Information Value keeps 8.8% (65B tokens).
    • Educational Quality keeps 6.1% (33B tokens).
    • Dense Core keeps 5.1%, which is 24.5M documents and 28B tokens.
  • Documents get longer as filtering gets stricter. Dense Core averages about 1150 tokens per document versus about 786 for the random baseline, which is nearly 50% longer.
  • Each subset gets its own 32k-vocabulary BPE tokenizer. Cleaner tokenizer training data lowers fertility on train splits, on a shared FW2-DE test split and on benchmark prompts. Train-split fertility drops from 1.49 to 1.32.
  • Two existing German filtered sets serve as external baselines: FW2-MKC/FW HQ (Messmer et al.) and Aleph-Alpha-GermanWeb AA-High (Burns et al.).
  • The authors fixed broken machine-translation artifacts in the German versions of ARC-Challenge, HellaSwag, LAMBADA and OpenBookQA, and released the cleaned suite. The stated limits of the study are:
    • German only;
    • dense transformers only;
    • at most 1B parameters and 200B tokens;
    • no toxicity or bias audit.

Interesting ideas

  • I1. The filters form a ladder of increasing strictness (form, then content, then teaching value), and the strictest training set is the intersection of all three tiers rather than a single score threshold. Each property's effect can then be measured separately.
  • I2. A small, very clean set repeated over many epochs beat a large noisy set seen once, with no sign of saturation by 7 epochs. This challenges the usual fear of repeating data.
  • I3. Translated benchmarks were checked for items where translation broke the task logic, for example word-order changes that spoil a cloze target. Those items were then fixed or dropped, and the corrected set was released.
  • I4. Tokenizer quality was measured not only on held-out text but also on the evaluation prompts themselves, which ties preprocessing choices to the target task.
  • I5. The study checks whether pretraining data quality carries over to a later stage, instruction tuning, and finds that adding low-quality documents hurt the final assistant.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver parses could pass through similar tiers (well-formed sentence, witness parsers agree, semantically rich frame), and the intersection could serve as a small "dense core" training set.
  • I2 (tree): For the perceptron parser and gparse, it is worth testing many epochs over gold UD treebanks plus a strictly agreed silver core against adding large amounts of lower-agreement silver.
  • I3 (answer): School tests and fable QA items should be audited for questions whose answer hinges on surface artifacts rather than meaning, so that scores reflect actual understanding.
  • I4 (tokens): Lemma and morphology coverage of the en crate could be reported directly on test-question text, and lexicon tables could be built from the cleanest sources first.
  • I5 (frames): Frame and role extraction quality should be judged by how it affects downstream world-building and answers, not just in isolation, since noise upstream tends to propagate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Humboldt-Universität zu BerlinBerlin · Germany19 works · 41,895 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ansar Aynetdinov, Patrick Haller and Alan Akbik. 2026. Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling. arXiv:2604.28075.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.28075 · arXiv 2604.28075

@misc{aynetdinov2026repetition,
  title = {{Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling}},
  author = {Ansar Aynetdinov, Patrick Haller and Alan Akbik},
  year = {2026},
  doi = {10.48550/arxiv.2604.28075},
  eprint = {2604.28075},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.28075}
}

References

49 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022. Glam: Efficient scaling of language models with mixture-of-experts. Preprint, arXiv:2112.06905. ↗
  2. Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Alex Fang, Hadi Pouransari, Matt Jordan, Alexander T Toshev, Vaishaal Shankar, Ludwig Schmidt, and Tom Gunter. 2026. Datasets, documents, and repetitions: The practicalities of unequal data quality. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  3. Manuel Faysse, Patrick Fernandes, Nuno M Guerreiro, António Loison, Duarte Miguel Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro Henrique Martins, Antoni Bigata Casademunt, François Yvon, Andre Martins, Gautier Viaud, CE-LINE HUDELOT, and Pierre Colombo. 2025. CroissantLLM: A truly bilingual french-english language model. Transactions on Machine Learning Research. ↗
  4. Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, Ramya Raghavendra, Ruoxi Jia, and Carole-Jean Wu. 2025. Demystifying synthetic data in LLM pre-training: A systematic study of scaling laws, benefits, and pitfalls. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10739–10758, Suzhou, China. Association for Computational Linguistics. ↗
  5. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. ↗
  6. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361. ↗
  7. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. The language model evaluation harness. find
  8. Hele-Andra Kuulmets, Taido Purason, Agnes Luhtaru, and Mark Fishel. 2024. Teaching llama a new language through cross-lingual knowledge transfer. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3309–3325, Mexico City, Mexico. Association for Computational Linguistics. find
  9. Gemma 3 Team. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  10. Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. Preprint, arXiv:2307.16039. ↗
  11. GPT-3 Team. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  12. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. Preprint, arXiv:2309.05463. ↗
  13. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. 2023. Textbooks are all you need. Preprint, arXiv:2306.11644. ↗
  14. Llama 2 Team. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. ↗
  15. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, volume 35, pages 30016–30030. Curran Associates, Inc. find
  16. Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations. ↗
  17. Gemma 2 Team. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. ↗
  18. Llama 3 Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  19. LLM-jp Team. 2024. Llm-jp: A cross-organizational project for the research and development of fully open japanese llms. Preprint, arXiv:2407.03963. ↗
  20. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  21. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. Fineweb-edu: the finest collection of educational content. find
  22. Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. 2023. find
  23. Michael Hoffmann, Jophin John, Stefan Schweter, Gokul Ramakrishnan, Hoi-Fong Mak, Alice Zhang, Dmitry Gaynullin, and Nicolay J. Hammer. 2025. Llama-genba-10b: A trilingual large language model for german, english and bavarian. Preprint, arXiv:2509.05668. ↗
  24. FinGPT: Large generative models for a small language. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2710–2726, Singapore. Association for Computational Linguistics. find
  25. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data only. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. find
  26. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurollm: Multilingual language models for europe. Preprint, arXiv:2409.16235. ↗
  27. Jan Pfister, Julia Wunderle, and Andreas Hotho. 2025. LLäMmlein: Transparent, compact and competitive German-only language models from scratch. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2227–2246, Vienna, Austria. Association for Computational Linguistics.
  28. Bettina Messmer, Vinko Sabolčec, and Martin Jaggi. 2025. Enhancing multilingual LLM pretraining with model-based data selection. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  29. Qwen3 Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  30. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. ↗
  31. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  33. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2023. Scaling data-constrained language models. In Advances in Neural Information Processing Systems, volume 36, pages 50358–50376. Curran Associates, Inc. find
  34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. Preprint, arXiv:1910.10683. ↗
  35. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703– 1714, Online. Association for Computational Linguistics.
  36. Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, and André F. T. Martins. 2025. Tower+: Bridging generality and translation specialization in multilingual llms. Preprint, arXiv:2506.17080. ↗
  37. PaLM Team. 2022. Palm: Scaling language modeling with pathways. Preprint, arXiv:2204.02311. ↗
  38. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. Ai models collapse when trained on recursively generated data. Nature, 631:755 – 759. ↗
  39. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset. find
  40. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. 2025. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics. ↗
  41. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Second Conference on Language Modeling. ↗
  42. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, volume 37, pages 30811–30849. Curran Associates, Inc.
  43. Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. Preprint, arXiv:2402.06619. ↗
  44. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: an open corpus of three trillion tokens for language model pretraining research. Preprint, arXiv:2402.00159. ↗
  45. Gopher Team. 2022. Scaling language models: Methods, analysis and insights from training gopher. Preprint, arXiv:2112.11446. ↗
  46. Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. 2024. Arctic-embed 2.0: Multilingual retrieval without compromise. Preprint, arXiv:2412.04506. ↗
  47. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. ↗
  48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685. ↗
  49. Wenzhen Zheng, Wenbo Pan, Xu Xu, Libo Qin, Li Yue, and Ming Zhou. 2024. Breaking language barriers: Cross-lingual continual pre-training at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7725– 7738, Miami, Florida, USA. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aynetdinov-2026-repetition-over-diversity-high-signal/