Elamur Research⌕
☰
home / papers / 2024 / Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining

Paper · 2024 · Steven Feng, Shrimai Prabhumoye, Kezhi Kong et al.

Summary

Organizations that build large language models rarely disclose how they mix and order pretraining data, so there is little public guidance on how a data blend should be designed and whether it holds up at larger scale. This NVIDIA–Stanford study formalizes two-phase pretraining. In phase-1, the model trains on a diverse mixture dominated by Common Crawl data. In phase-2, the mixture shifts toward high-quality sources: math, code, Wikipedia, top-tier crawl and task/SFT data. The authors weight each source by its estimated quality and by how many epochs it would be repeated. They prototype blends cheaply by downsampling every source to 1/15 and training an 8B Megatron model on 1T tokens. They then test whether the best blends transfer to longer runs (1.7T and 15T tokens on full data) and to a 25B model. The two-phase recipe outperforms both a natural-distribution blend and randomly ordered data with the same weights. Its advantages persist at larger scale, which gives practitioners a reproducible procedure for designing data mixtures.

Key points

  • Average accuracy at 1T tokens was 45.17 for BASE-ND (natural token distribution), 51.12 for BASE-RO (random order) and 52.86 for two-phase. These correspond to relative gains of about 17% over BASE-ND and 3.4% over BASE-RO.
  • The best-performing combination was P1-Blend4 followed by P2-Blend1. Starting phase-2 for roughly the final 40% of training worked best, and extending it further brought diminishing returns.
  • The full corpus includes about 6.2T web crawl tokens, plus math, code, Wikipedia, books, papers, CC derivatives, multilingual data (9 languages) and FLAN/SFT task data.
  • The same blend design reached an average of 59.84 when an 8B model was trained on 15T tokens of full data. Scaling the model to 25B also improved accuracy.
  • The evaluation suite covered MMLU (5-shot), eight 0-shot reasoning tasks (e.g., ARC, HellaSwag, WinoGrande), HumanEval(+)/MBPP(+) for code and GSM8K (8-shot CoT) for math.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓Boston UniversityBoston · United States26 works · 4,022 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Steven Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi and Bryan Catanzaro. 2024. Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining. arXiv:2412.15285.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.15285 · arXiv 2412.15285

@misc{feng2024maximize,
  title = {{Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining}},
  author = {Steven Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi and Bryan Catanzaro},
  year = {2024},
  doi = {10.48550/arxiv.2412.15285},
  eprint = {2412.15285},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.15285}
}

References

62 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895– 4…
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  3. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. Preprint, arXiv:2001.08435. — [arXiv:2001.08435](https://arxiv.org/abs/2001.08435) find
  4. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. Preprint, arXiv:1911.11641.
  5. Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. 2024. Does your data spark joy? performance gains from domain upsampling at the end of training. Preprint, arXiv:2406.03476. — [arXiv:2406.03476](https://arxiv.org/abs/2406.03476) find
  6. Tom B. Brown et al. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165.
  7. Mark Chen et al. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374.
  8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  11. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.
  12. Together Computer. 2023. Redpajama: an open dataset for training large language models.
  13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024a. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Abhimanyu Dubey et al. 2024b. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  15. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Lin…
  16. Hugo Laurençon et al. 2022. The bigscience roots corpus: A 1.6tb composite multilingual dataset. In Advances in Neural Information Processing Systems, volume 35, pages 31809–31826. Curran Associates, Inc.
  17. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988, Online. Association for Computational Lin…
  18. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. 2024a. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794.
  19. Steven Y. Feng, Noah Goodman, and Michael Frank. 2024. Is child-directed speech effective training data for language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22055–22071, Miami, Florida, USA. Association for Computational Linguistics.
  20. Jeffrey Li et al. 2024b. Datacomp-LM: In search of the next generation of training sets for language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  21. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. find
  22. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. 2024. Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712. find
  23. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus . ↗
  24. Dirk Groeneveld et al. 2024. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand. Association for Computational Linguistics. find
  25. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. find
  26. Raymond Li et al. 2023. Starcoder: may the source be with you! Transactions on Machine Learning Research. Reproducibility Certification. find
  27. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. 2024. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887.
  28. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631–22648. PMLR.
  29. Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2024. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2024 Conference… find
  30. Richard Diehl Martinez, Hope McGovern, Zebulon Goriely, Christopher Davis, Andrew Caines, Paula Buttery, and Lisa Beinborn. 2023. CLIMB – curriculum learning for infant-inspired model building. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning,… find
  31. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  32. Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. 2022. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Proceedings of the 39th I… find
  33. Nvidia et al. 2024. Nemotron-4 340b technical report. Preprint, arXiv:2406.11704.
  34. OpenAI. 2023. tiktoken. https://github.com/openai/ tiktoken . Accessed: 2024-12-14. ↗
  35. OpenAI et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  36. Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557. — The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
  37. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. Preprint, arXiv:1907.10641. find
  38. Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Balanced data sampling for language model training with clustering. arXiv preprint arXiv:2402.14526. find
  39. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. find
  40. Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. 2023. Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818.
  41. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-lm: Training multi-billion parameter language models using model parallelism. Preprint, arXiv:1909.08053. find
  42. Snowflake. 2024. Snowflake arctic: The best llm for enterprise ai — efficiently intelligent, truly open. https://www.snowflake.com/en/blog/arctic-open-efficient-foundation-language-\\ models-snowflake/ . Accessed: 2024-12-14. ↗
  43. Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024a. Data, data everywhere: A guide for pre-training dataset construction. In Proceedings of the 2024 Conference on Empirical Methods in Natural L… find
  44. Luca Soldaini et al. 2024. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand. Association for Computational L… find
  45. Jupinder Parmar et al. 2024b. Nemotron-4 15b technical report. Preprint, arXiv:2402.16819.
  46. Stack Exchange. Accessed 2024. Stack exchange data dump. find
  47. Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2024. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations.
  48. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. Preprint, arXiv:2412.02595.
  49. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024a. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets…
  50. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024b. The fineweb datasets: Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:…
  51. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  52. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024a. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  53. Gemma Team et al. 2024b. Gemma: Open models based on gemini research and technology. Preprint, arXiv:2403.08295. find
  54. Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. 2024. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv:2402.10176. find
  55. Xin Wang, Yudong Chen, and Wenwu Zhu. 2022. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576. find
  56. Taylor Webb, Keith J. Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Preprint, arXiv:2212.09196. find
  57. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2023a. Doremi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems, volume 36, pages 69798–69818. Curran As… find
  58. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. 2023b. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201– 34227. find
  59. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  60. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. find
  61. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  62. Web Crawl - 31.0 49.0 High Quality Math Wiki Code 24.0 1.0 20.0 14.4 0.6 12.0 Medium Quality Books Papers CC dv 8.0 4.0 7.0 8.0 4.0 7.0 Multilingual - 3.7 4.2 Task Data - 1.3 0.8 *Table 15: Comparison of Blend1 and Blend6 (in %) used for scaling the token budget in §4.2 and Table 8.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/feng-2024-maximize-data-s-potential-enhancing/