Elamur Research⌕
☰
home / papers / 2026 / Scaling Laws for Code: A More Data-Hungry Regime

Scaling Laws for Code: A More Data-Hungry Regime

Summary

Scaling laws, which predict validation loss from model size (N), training tokens (D) and compute (C), have mostly been derived on natural language (NL). This paper asks whether they carry over to source code, given its rigid syntax, long-range dependencies and distinctive vocabulary. The authors trained 117 decoder-only Transformers spanning 0.2B–3.8B parameters and 2B–128B tokens on filtered GitHub code from the Opencoder corpus. Architecture shapes followed the Farseer procedure and learning rates and batch sizes followed StepLaw. They then fitted both the Chinchilla and the Farseer formulations. Farseer described code loss more accurately and predicted held-out points better. Code models kept improving as N grew, yet the compute-optimal D/N ratio for code is far larger than for NL, and the gap widens with budget. The authors link this to code's repetitiveness. They argue that today's relatively small Code LLMs reflect limited high-quality data and latency needs, not weak returns from scale. Two further sweeps with code–NL mixtures showed that NL helps only under tight budgets.

Key points

  • Experiment scale: The core grid has 117 runs, sampled log-uniformly over N (0.2B–3.8B) and D (2B–128B), with extreme D/N ratios removed. Training ran on 8–128 NVIDIA H100 GPUs per job, about 13,600 GPU-days in total.
  • Data:
    • The training set is the Opencoder code corpus, downsampled from 1216.93B to 895.51B tokens to balance programming languages.
    • Evaluation uses a separate 6.3M-token production codebase from Xuyang et al. (2025), checked for overlap with the training data.
  • Law comparison: The Farseer law, whose exponents depend on N, fit and extrapolated code loss better than the Chinchilla law with its fixed exponents.
  • Data hunger: For a given model size, code needs considerably more tokens than NL to reach optimal loss. Existing models such as Qwen2.5-Coder, starcoder2 and deepseek-coder-base already sit far above NL-predicted D/N ratios.
  • Mixtures: Two extra 117-run sweeps tested 70%/30% and 30%/70% code–NL blends (234 runs in total).
    • Adding NL improves code loss when code data or compute is scarce.
    • It hurts at larger scale or when plenty of pure code is available, with the NL-heavier blend performing worst.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Institutions

🎓Harbin Institute of TechnologyHarbin · China58 works · 25,250 citations🏛️Chinese Academy of SciencesBeijing · China51 works · 17,412 citations🎓Fudan UniversityShanghai · China45 works · 16,344 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China13 works · 2,509 citations

Cite

Cite as Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan and Wanxiang Che. 2026. Scaling Laws for Code: A More Data-Hungry Regime. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1101

@inproceedings{luo2026scaling,
  title = {{Scaling Laws for Code: A More Data-Hungry Regime}},
  author = {Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan and Wanxiang Che},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1101},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1101}
}

References

52 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, and 1 others. 2023. Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988. our card
  2. Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):1–37. find
  3. Casey Casalnuovo, Kenji Sagae, and Prem Devanbu. 2019. Studying the difference between natural and programming language corpora. Empirical Software Engineering, 24(4):1823–1868. find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… our card
  5. Zhengyu Chen, Siqi Wang, Teng Xiao, Yudong Wang, Shiqi Chen, Xunliang Cai, Junxian He, and Jingang Wang. 2025. Revisiting scaling laws for language models: The role of data quality and training strategies. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vo… find
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113. our card
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, and 1 others. 2024. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437. find
  8. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations. find
  9. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. find
  10. Md Mahade Hasan, Muhammad Waseem, Kai-Kristian Kemell, Jussi Rasku, Juha Ala-Rantala, and Pekka Abrahamsson. 2025. Assessing small language models for code generation: An empirical study with benchmarks. arXiv preprint arXiv:2507.03160. — [arXiv:2507.03160](https://arxiv.org/abs/2507.03160) find
  11. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… our card
  12. Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, and 1 others. 2025. Os agents: A survey on mllm-based agents for computer, phone and browser use. In Proceedings of the 63rd Annual Meeting of the Association for Computational Ling… find
  13. Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, Yuan-Tao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, and 7 others. 2025. OpenCoder: The open cookbook f… find
  14. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2.5-Coder technical report. arXiv preprint arXiv:2409.12186. our card
  15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. our card
  16. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.17192. find
  17. Houyi Li, Wenzheng Zheng, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. 2025a. Predictable scale: Part I, step law – optimal hyperparameter scaling law in large language model pretraining. In P… find
  18. Houyi Li and 1 others. 2025b. Predictable scale: Part II, Farseer: A refined scaling law in large language models. arXiv preprint arXiv:2506.10972. find
  19. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, and 48… find
  20. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, and 7 o… our card
  21. Jiayi Lin, Hande Dong, Yutao Xie, and Lei Zhang. 2024. Scaling laws behind code understanding model. arXiv preprint arXiv:2402.12813. — [arXiv:2402.12813](https://arxiv.org/abs/2402.12813) find
  22. Fei Liu, Yiming Yao, Ping Guo, Zhiyuan Yang, Zhe Zhao, Xi Lin, Xialiang Tong, Mingxuan Yuan, Zhichao Lu, Zhenkun Wang, and 1 others. 2024. A systematic survey on large language models for algorithm design. arXiv preprint arXiv:2410.14716. — [arXiv:2410.14716](https://arxiv.org/abs/2410.14716) find
  23. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2025. Regmix: Data mixture as regression for language model pre-training. In The Thirteenth International Conference on Learning Representations. find
  24. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). find
  25. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Tianyang Liu, Misha Laskin, and 1 others. 2024. StarCoder2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173. find
  26. Jan Ludziejewski, Maciej Pióro, Jakub Krajewski, Maciej Stefaniak, Michał Krutul, Jan Małaśnicki, Marek Cygan, Piotr Sankowski, Kamil Adamczewski, Piotr Miłoś, and 1 others. 2025. Joint moe scaling laws: Mixture of experts can be memory efficient. arXiv preprint arXiv:2502.05172. — [arXiv:2502.05172](https://arxiv.org/abs/2502.05172) find
  27. Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, and 1 others. 2024. Granite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:2405.04324. find
  28. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations. find
  29. OpenAI, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, and 6 others. 2025.… find
  30. Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R Fung, and 1 others. 2025. Scaling laws of synthetic data for language models. arXiv preprint arXiv:2503.19551. — [arXiv:2503.19551](https://arxiv.org/abs/2503.19551) find
  31. Nicholas Roberts, Niladri S. Chatterji, Sharan Narang, Mike Lewis, and Dieuwke Hupkes. 2025. Compute optimal scaling of skills: Knowledge vs reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 13295–13316, Vienna, Austria. Association for Computational Linguistic… find
  32. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, and 6 o… our card
  33. Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. 2024. Beyond chinchilla-optimal: accounting for inference in language model scaling laws. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. our card
  34. ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, and 1 others. 2025. Seed-coder: Let the code model curate data for itself. arXiv preprint arXiv:2506.03524. find
  35. Noam Shazeer. 2020. GLU variants improve transformer. arXiv preprint arXiv:2002.05202. find
  36. Ben Shi, Michael Tang, Karthik R Narasimhan, and Shunyu Yao. 2024. Can language models solve olympiad programming? In First Conference on Language Modeling. find
  37. Xian Shuai, Yiding Wang, Yimeng Wu, Xin Jiang, and Xiaozhe Ren. 2024. Scaling law for language models training considering batch size. arXiv preprint arXiv:2412.01505. — [arXiv:2412.01505](https://arxiv.org/abs/2412.01505) find
  38. Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. 2024. Unraveling the mystery of scaling laws: Part i. arXiv preprint arXiv:2403.06563. — [arXiv:2403.06563](https://arxiv.org/abs/2403.06563) find
  39. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864. our card
  40. Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, and Yueting Zhuang. 2025. A survey on (m)llm-based gui agents. Preprint, arXiv:2504.13865. — [arXiv:2504.13865](https://arxiv.org/abs/2504.13865) find
  41. CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A Choquette-Choo, Jingyue Shen, Joe Kelley, and 1 others. 2024. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409. find
  42. The Terminal-Bench Team. 2025. Terminal-bench: A benchmark for ai agents in terminal environments. find
  43. Team, Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534. our card
  44. Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, and Jun Zhou. 2025. Towards greater leverage: Scaling laws for efficient mixture-of-experts language models. arXiv preprint arXiv:2507.17702. — [arXiv:2507.17702](https://arxiv.org/abs/2507.17702) find
  45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  46. Glynn Winskel. 1993. The formal semantics of programming languages: an introduction. MIT press. find
  47. Shijie Xuyang, Xianzhen Luo, Tianhao Cheng, Zheng Chu, Houyi Li, ziqi wang, Siming Huang, Qingfu Zhu, Qiufeng Wang, Xiangyu Zhang, Shuigeng Zhou, and Wanxiang Che. 2025. Is compression really linear with code intelligence? Preprint, arXiv:2505.11441. — [arXiv:2505.11441](https://arxiv.org/abs/2505.11441) find
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. our card
  49. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32. find
  50. Guoliang Zhao, Yuhan Fu, Shuaipeng Li, Xingwu Sun, Ruobing Xie, An Wang, Weidong Han, Zhen Yang, Weixuan Sun, Yudong Zhang, and 1 others. 2025. Towards a comprehensive scaling law of mixture-of-experts. arXiv preprint arXiv:2509.23678. — [arXiv:2509.23678](https://arxiv.org/abs/2509.23678) find
  51. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, and 1 others. 2024a. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931. our card
  52. Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Jia Li, Zhi Jin, and Hong Mei. 2024b. Hot or cold? adaptive temperature sampling for code generation with large language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-scaling-laws-code-more-data/