Elamur Research⌕
☰
home / papers / 2026 / Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

Paper · 2026 · Yajiao Liu, Congliang Chen, Junchi Yang et al.

Summary

When an LLM is trained on data pooled from many sources, such as CommonCrawl, Wikipedia or Github, under a fixed compute budget, the share of each domain strongly affects downstream quality. This survey reviews methods that choose these domain weights automatically, a problem also called domain reweighting. It builds on the offline/online split from Albalak et al. (2024) and refines it. Offline methods set static weights before training the target model, usually with a smaller proxy model. The authors divide them into heuristic, algorithm-based and function fitting-based ("mixing law") approaches. Online methods adapt weights while the target model trains. The authors group them as online min-max, online mixing law and other approaches, mainly by showing they are online versions of the offline optimization frameworks. For each subtype, the paper sets out the formal problem and representative algorithms, then compares their strengths, weaknesses and open challenges. The paper is purely analytical and runs no experiments. Its value lies in a unified optimization-centred map of a fast-growing area that matters as LLM training costs keep rising.

Key points

  • Offline methods:
    • Heuristic methods (uniform/proportional sampling) are treated as degenerate cases.
    • Algorithm-based methods follow two frameworks borrowed from multi-task learning: min-max (DoReMi) and bi-level (DoGE, ScaleBiO).
    • Function fitting-based methods include DML, BiMix, D-CPT, AutoScale, RegMix and MDE.
  • Online methods:
    • Online min-max: MFTCoder, ShearedLLaMa, Velocitune.
    • Online mixing law approaches.
    • Others: ODM, ADO.
    • Only online mixing law methods work when performance is judged by out-of-domain loss.
  • Domain granularity: Natural domains are often coarse. For example, GLaM has 8 domains and the Pile has 22, and DoReMi works better on the Pile. For finding finer domains by clustering, loss-trajectory features beat embedding-based ones (Chen et al., 2023).
  • Weak gains over uniform sampling: Weights tuned on loss are often only marginally better than uniform sampling, or even worse. This raises doubts about loss as a proxy for task performance.
  • Stated limitation: The survey includes no aligned empirical comparison of the methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong Kong, ShenzhenShenzhen · China47 works · 2,596 citations🎓Shenzhen Research Institute of Big DataShenzhen · China17 works · 1,721 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yajiao Liu, Congliang Chen, Junchi Yang and Ruoyu Sun. 2026. Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.15

@inproceedings{liu2026rethinking,
  title = {{Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives}},
  author = {Yajiao Liu, Congliang Chen, Junchi Yang and Ruoyu Sun},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.15},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.15}
}

References

50 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. 2024. A survey on data selection for language models. Preprint, arXiv:2402.1682…
  2. Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. 2023. Efficient online data mixing for language model pre-training. Preprint, arXiv:2312.02406.
  3. Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. 2002. The nonstochastic multi-armed bandit problem. SIAM journal on computing, 32(1):48–77. find
  4. Lior Belenki, Alekh Agarwal, Tianze Shi, and Kristina Toutanova. 2025. Optimizing pre-training data mixtures with mixtures of data expert models. Preprint, arXiv:2502.15950. find
  5. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48.
  6. Mayee F. Chen, Michael Y. Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. 2024. Aioli: A unified optimization framework for language model data mixing. Preprint, arXiv:2411.05735. find
  7. Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. 2023. Skill-it! a data-driven skills framework for understanding and training language models. Preprint, arXiv:2307.14430. — [arXiv:2307.14430](https://arxiv.org/abs/2307.14430) find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  9. Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. 2018. Training gans with optimism. Preprint, arXiv:1711.00141. — [arXiv:1711.00141](https://arxiv.org/abs/1711.00141) find
  10. Constantinos Daskalakis and Ioannis Panageas. 2018. The limit points of (optimistic) gradient descent in min-max optimization. Preprint, arXiv:1807.03907. — [arXiv:1807.03907](https://arxiv.org/abs/1807.03907) find
  11. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas… — GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
  12. Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J. Zico Kolter. 2024. Adaptive data optimization: Dynamic sample selection with scaling laws. Preprint, arXiv:2410.11820. — [arXiv:2410.11820](https://arxiv.org/abs/2410.11820) find
  13. Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. 2025. Autoscale: Scale-aware data mixing for pre-training llms. Preprint, arXiv:2407.20177. — [arXiv:2407.20177](https://arxiv.org/abs/2407.20177) find
  14. Jean-Antoine Désidéri. 2012. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313–318. find
  15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  16. Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2023. DOGE: Domain reweighting with generalization estimation. In Second Agent Learning in Open-Endedness Workshop. find
  17. Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert Nowak. 2023. A fully first-order method for stochastic bilevel optimization. Preprint, arXiv:2301.10945. — [arXiv:2301.10945](https://arxiv.org/abs/2301.10945) find
  18. Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. 2017. Forward and reverse gradient-based hyperparameter optimization. Preprint, arXiv:1703.01785. — [arXiv:1703.01785](https://arxiv.org/abs/1703.01785) find
  19. Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Th… — Language models scale reliably with over-training and on downstream tasks
  20. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. find
  21. Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. 2024. Bimix: Bivariate data mixing law for language model pretraining. Preprint, arXiv:2405.14908. — [arXiv:2405.14908](https://arxiv.org/abs/2405.14908) find
  22. Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2023. Scaling expert language models with unsupervised domain discovery. Preprint, arXiv:2303.14177. find
  23. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… find
  24. Bingchang Liu, Chaoyu Chen, Cong Liao, Zi Gong, Huan Wang, Zhichao Lei, Ming Liang, Dajun Chen, Min Shen, Hailian Zhou, Hang Yu, and Jianguo Li. 2023a. Mftcoder: Boosting code llms with multitask fine-tuning. Preprint, arXiv:2311.02303. — [arXiv:2311.02303](https://arxiv.org/abs/2311.02303) find
  25. Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. 2023b. Famo: Fast adaptive multitask optimization. Preprint, arXiv:2306.03792. — [arXiv:2306.03792](https://arxiv.org/abs/2306.03792) find
  26. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. 2021a. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34. find
  27. Dong C Liu and Jorge Nocedal. 1989. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503–528. find
  28. Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021b. Towards impartial multi-task learning. In International Conference on Learning Representations. find
  29. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2024. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492.
  30. Jonathan Lorraine, Paul Vicol, and David Duvenaud. 2019. Optimizing millions of hyperparameters by implicit differentiation. Preprint, arXiv:1911.02590. — [arXiv:1911.02590](https://arxiv.org/abs/1911.02590) find
  31. Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Chen Qi, and Peng Cheng. 2024. Velocitune: A velocity-based dynamic domain reweighting method for continual pre-training. Preprint, arXiv:2411.14318. find
  32. Xingyou Song, Qiuyi Zhang, Chansoo Lee, Emily Fertig, Tzu-Kuo Huang, Lior Belenki, Greg Kochanski, Setareh Ariafar, Srinivas Vasudevan, Sagi Perel, and Daniel Golovin. 2024. The vizier gaussian process bandit algorithm. Preprint, arXiv:2408.11527. — [arXiv:2408.11527](https://arxiv.org/abs/2408.11527) find
  33. Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. Openelm: An efficient language model family with open training and inference framework. Preprint, arXiv:2404.146… — [arXiv:2404.14619](https://arxiv.org/abs/2404.14619) find
  34. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern,… — SciPy 1.0 — Fundamental Algorithms for Scientific Computing in Python
  35. Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. 2019. Distributionally robust language modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p… find
  36. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024a. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. Preprint, arXiv:2403.17919. find
  37. Rui Pan, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Xiaoyu Wang, and Tong Zhang. 2024b. Scalebio: Scalable bilevel optimization for llm data reweighting. arXiv preprint arXiv:2406.19976. find
  38. Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. 2024. D-cpt law: Domain-specific continual pre-training scaling law for large language models. Preprint, arXiv:… find
  39. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  40. Meisam Razaviyayn, Tianjian Huang, Songtao Lu, Maher Nouiehed, Maziar Sanjabi, and Mingyi Hong. 2020. Nonconvex min-max optimization: Applications, challenges, and recent theoretical advances. IEEE Signal Processing Magazine, 37(5):55–66. find
  41. Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. Preprint, arXiv:1911.08731. find
  42. Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. 2024. Slimpajama-dc: Understanding data combinations for llm training. Preprint, arXiv:2309.10818.
  43. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. Preprint, arXiv:2109.01652.
  44. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. Preprint, arXiv:2310.06694. find
  45. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. 2023. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305.10429. find
  46. A Yongxin Yang and Timothy M. Hospedales. 2015. A unified perspective on multi-domain and multi-task learning. Preprint, arXiv:1412.7489. — [arXiv:1412.7489](https://arxiv.org/abs/1412.7489) find
  47. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952. find
  48. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782.
  49. Jiawei Zhang, Peijun Xiao, Ruoyu Sun, and Zhi-Quan Luo. 2022. A single-loop smoothed gradient descent-ascent algorithm for nonconvex-concave min-max problems. Preprint, arXiv:2010.15768. — [arXiv:2010.15768](https://arxiv.org/abs/2010.15768) find
  50. Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. 2023. An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning. Preprint, arXiv:2308.00788. — [arXiv:2308.00788](https://arxiv.org/abs/2308.00788) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-rethinking-data-mixture-large-language/