Elamur Research⌕
☰
home / papers / 2024 / Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Summary

Two influential compute-optimal scaling laws disagree. Kaplan et al. found that, as compute grows, model size should grow faster than training tokens. Hoffmann et al. ("Chinchilla") found that tokens per parameter should stay roughly fixed. Porian et al. investigate why. They reproduce the Kaplan-style result with Llama-like decoder-only Transformers trained in OpenLM, using 16 models from 5M to 901M parameters. The data are RefinedWeb and OpenWebText2. They estimate optimal model sizes from IsoFLOP curves and fit power laws with bootstrap uncertainty, across more than 900 runs. Three factors close the gap:

  • counting the FLOPs of the output head,
  • shortening warmup for small models,
  • tuning learning rate, batch size and AdamW β2 per model size.

With these corrections, the fitted exponent matches Chinchilla. Contrary to Hoffmann et al.'s explanation, a learning-rate decay schedule matched to each token budget is not required, and even a constant learning rate works. The study shows how to get reliable scaling predictions on modest, academic-scale budgets.

Key points

  • The fitted exponent a in N⋆ ∝ C^a falls from 0.835 (Kaplan reproduction) to 0.497 (Chinchilla-like) once the factors are corrected. Intermediate values are 0.706 with head FLOPs counted and 0.602 with scaled warmup. Cosine decay alone gives 0.571, while per-size tuning with no decay reaches 0.497.
  • At Chinchilla's compute budget of 5.88e23 FLOPs, the predicted optimal model shrinks from about 3T to about 77B parameters.
  • Datasets: RefinedWeb (~600B tokens, CommonCrawl) and OpenWebText2 (~30B tokens, Reddit). Hyperparameters were tuned only on RefinedWeb and transferred well to OpenWebText2.
  • Secondary result: power-law fits describing how the best learning rate and best batch size scale. The common AdamW setting β2 = 0.95 performs poorly at batch sizes of 128 or below.
  • Code and data are released on GitHub (formll/resolving-scaling-law-discrepancies), with checkpoints on Hugging Face. Using a constant learning rate saves roughly as much compute as the hyperparameter sweep costs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Language models & pretraining

Cite

Cite as Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt and Yair Carmon. 2024. Resolving Discrepancies in Compute-Optimal Scaling of Language Models. arXiv:2406.19146.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.19146 · arXiv 2406.19146

@misc{porian2024resolving,
  title = {{Resolving Discrepancies in Compute-Optimal Scaling of Language Models}},
  author = {Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt and Yair Carmon},
  year = {2024},
  doi = {10.48550/arxiv.2406.19146},
  eprint = {2406.19146},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.19146}
}

References

63 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Common Crawl. https://commoncrawl.org. ↗
  2. New scaling laws for large language models, 2023. URL https://www.lesswrong.com/ posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-models. Accessed: 2024-08-19. ↗
  3. H. Akima. A new method of interpolation and smooth curve fitting based on local procedures. Journal of the ACM (JACM), 1970. find
  4. I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer. Getting ViT in shape: Scaling laws for compute-optimal model design. In Advances in Neural Information Processing Systems (NeurIPS), 2023. find
  5. A. Atanasov, B. Bordelon, S. Sainathan, and C. Pehlevan. The onset of variance-limited behavior for networks in the lazy and rich regimes. In International Conference on Learning Representations (ICLR), 2023. find
  6. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv:1607.06450, 2016. our card
  7. G. Bachmann, S. Anagnostidis, and T. Hofmann. Scaling mlps: A tale of inductive bias. Advances in Neural Information Processing Systems (NeurIPS), 2024. find
  8. M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta, et al. Stable lm 2 1.6 b technical report. arXiv:2402.17834, 2024. find
  9. T. Besiroglu, E. Erdil, M. Barnett, and J. You. Chinchilla scaling: A replication attempt. arXiv:2404.10102, 2024. our card
  10. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv:2204.06745, 2022. our card
  11. B. Bordelon, A. Atanasov, and C. Pand Pehlevan. A dynamical model of neural scaling laws. In International Conference on Machine Learning (ICML), 2024. find
  12. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 2020. our card
  13. M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on computer vision and pattern recognition (CVPR), 2023. our card
  14. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. our card
  15. DeepSeek. Deepseek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954, 2024. find
  16. M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML), 2023. find
  17. S. Y. Gadre, G. Smyrnis, V. Shankar, S. Gururangan, M. Wortsman, R. Shao, J. Mercat, A. Fang, J. Li, S. Keh, R. Xin, M. Nezhurina, I. Vasiljevic, J. Jitsev, A. G. Dimakis, G. Ilharco, S. Song, T. Kollar, Y. Carmon, A. Dave, R. Heckel, N. Muennighoff, and L. Schmidt. Language models scale reliably wi… find
  18. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027, 2020. find
  19. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv:1706.02677, 2017. find
  20. S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and J. Z. Kolter. Scaling laws for data filtering–data curation cannot be compute agnostic. In Conference on computer vision and pattern recognition (CVPR), 2024. find
  21. S. Gururangan, M. Wortsman, S. Y. Gadre, A. Dave, M. Kilian, W. Shi, J. Mercat, G. Smyrnis, G. Ilharco, M. Jordan, R. Heckel, A. Dimakis, A. Farhadi, V. Shankar, and L. Schmidt. OpenLM: a minimal but performative language modeling (lm) repository, 2023. https://github.com/ mlfoundations/open_lm. ↗
  22. A. Hägele, E. Bakouch, A. Kosson, L. B. Allal, L. Von Werra, and M. Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv:2405.18392, 2024. our card
  23. T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv:2010.14701, 2020. our card
  24. J. Hestness, S. Narang, N. Ardalani, G. F. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv:1712.00409, 2017. our card
  25. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), 2022. find
  26. S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024. our card
  27. M. Ivgi, Y. Carmon, and J. Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. find
  28. H. J. Jeon and B. Van Roy. arXiv:2407.01456, 2024. Information-theoretic foundations for neural scaling laws. — [arXiv:2407.01456](https://arxiv.org/abs/2407.01456) find
  29. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, F. B. Diego de las Casas, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B. arXiv:2310.06825, 2023. find
  30. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020. our card
  31. B. Lefaudeux, F. Massa, D. Liskovich, W. Xiong, V. Caggiano, S. Naren, M. Xu, J. Hu, M. Tintore, S. Zhang, P. Labatut, D. Haziza, L. Wehrstedt, J. Reizenstein, and G. Sizov. xformers: A modular and hackable transformer modelling library. https://github.com/ facebookresearch/xformers, 2022. ↗
  32. O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation, 2021. find
  33. L. Lin, J. Wu, S. M. Kakade, P. L. Bartlett, and J. D. Lee. Scaling laws in linear regression: Compute, parameters, and data. arXiv:2406.08466, 2024. — [arXiv:2406.08466](https://arxiv.org/abs/2406.08466) find
  34. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017. find
  35. A. Maloney, D. A. Roberts, and J. Sully. arXiv:2210.16859, 2022. A solvable model of neural scaling laws. — [arXiv:2210.16859](https://arxiv.org/abs/2210.16859) find
  36. S. McCandlish, J. Kaplan, D. Amodei, and The OpenAI Dota Team. An empirical model of large-batch training. arXiv:1812.06162, 2018. find
  37. N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems (NeurIPS), 2024. find
  38. E. Paquette, C. Paquette, L. Xiao, and J. Pennington. 4+3 phases of compute-optimal neural scaling laws. arXiv:2405.15074, 2024. — [arXiv:2405.15074](https://arxiv.org/abs/2405.15074) find
  39. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. 2019. our card
  40. T. Pearce and J. Song. Reconciling Kaplan and Chinchilla scaling laws. arXiv:2406.12907, 2024. — [arXiv:2406.12907](https://arxiv.org/abs/2406.12907) find
  41. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, H. Alobeidli, A. Cappelli, B. Pannier, E. Almazrouei, and J. Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Advances in Neural Information Processing Systems (NeurIPS), 2023. find
  42. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners, 2019. find
  43. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.11446, 2021. find
  44. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1–67, 2020. our card
  45. J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructive prediction of the generalization error across scales. arXiv:1909.12673, 2019. find
  46. N. Sardana and J. Frankle. Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv:2401.00448, 2023. our card
  47. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv:2211.05100, 2022. find
  48. R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems (NeurIPS), 2024. our card
  49. C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research (JMLR), 20(112):1–49, 2019. find
  50. N. Shazeer. Glu variants improve transformer. arXiv:2002.05202, 2020. find
  51. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053, 2019. find
  52. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model. rXiv:2201.11990, 2022. our card
  53. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. our card
  54. Y. Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. In International Conference on Learning Representations (ICLR), 2022. find
  55. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971, 2023. our card
  56. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… find
  57. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. 2017. our card
  58. M. Wortsman, P. J. Liu, L. Xiao, K. E. Everett, A. A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, J. Pennington, J. Sohl-Dickstein, K. Xu, J. Lee, J. Gilmer, and S. Kornblith. Small-scale proxies for large-scale transformer training instabilities. In International Conference on Lear… find
  59. X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In Conference on computer vision and pattern recognition (CVPR), 2022. find
  60. B. Zhang, I. Titov, and R. Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. find
  61. G. Zhang, L. Li, Z. Nado, J. Martens, S. Sachdeva, G. Dahl, C. Shallue, and R. B. Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems (NeuIPS), 2019. find
  62. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. OPT: Open pre-trained transformer language models. arXiv:2205.01068, 2022. our card
  63. Y. Zhang, C. Chen, N. Shi, R. Sun, and Z.-Q. Luo. Adam can converge without any modification on update rules. Advances in neural information processing systems, 35:28386–28399, 2022. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/