Elamur Research⌕
☰
home / papers / 2020 / Scaling Laws for Neural Language Models

Scaling Laws for Neural Language Models

Paper · 2020 · Jared Kaplan, Sam McCandlish, Tom Henighan et al.

Summary

The paper studies how the quality of a Transformer language model depends on how much is invested in it. Quality is measured as test cross-entropy loss, and the investment is split into three resources: parameter count (embeddings excluded), training tokens, and training compute. The authors trained a large family of autoregressive models on WebText2, ranging from about 10^3 to 10^9 non-embedding parameters. Across these runs they varied depth, width, attention heads, context length, batch size and training length.

Loss falls as a clean power law in each resource whenever the other two are not the bottleneck. Some trends span more than seven orders of magnitude. Within broad limits, architectural shape barely matters. One combined formula predicts overfitting from model and data size, and another fits learning curves over training steps.

From these fits the authors derive how to split a fixed compute budget. Most of it should go into bigger models, with only modest data growth and training stopped well before convergence. The work matters because it turns model planning into forecasting: cheap small runs can predict what expensive large ones will reach.

Key points

  • The fitted exponents are α_N ≈ 0.076, α_D ≈ 0.095 and α_C^min ≈ 0.050. Doubling the parameter count shrinks the loss by a factor of about 0.95.
  • The overfitting penalty depends on the ratio N^0.74/D. In practice, an 8x larger model needs only roughly 5x more data to avoid the penalty.
  • For compute-optimal training, the empirical fits are N ∝ C_min^0.73, B ∝ C_min^0.24 and S ∝ C_min^0.03, so data needs grow slowly, like D ∼ C^0.27.
  • The best batch size depends mainly on the current loss. For the largest models it is roughly 1–2 million tokens at convergence. Compute is estimated as C ≈ 6NBS and reported in PF-days.
  • On other text distributions, loss tracks training-distribution loss with a roughly constant offset. At about 1.5 billion parameters, depth has no effect on this transfer, and run-to-run noise is about 0.05 in loss.

Interesting ideas

  • I1. Fit a power law to results from a ladder of small, cheap runs and extrapolate it before committing to a big run; this replaces guesswork about whether scaling up pays off.
  • I2. Fit the early part of a learning curve to forecast the final loss, since curve shapes are roughly universal across model sizes.
  • I3. Measure run-to-run variation first and treat smaller gains as noise unless several seeds are averaged; this guards against celebrating random fluctuations.
  • I4. Break the average loss down by position in the context to see where capacity is spent and which patterns the model learns first.
  • I5. Out-of-distribution performance improves in step with in-distribution performance, with a fixed gap, so the in-domain dev score can serve as a cheap proxy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Fit gparse accuracy against the amount of silver treebank data to estimate how much more silver is worth producing.
  • I2 (learn): Stop unpromising training runs of the tagger or parser early and spend the GPU time on better configurations.
  • I3 (learn): Require any new self-induced repair rule or training change to beat the measured seed noise, alongside negative controls.
  • I4 (tree): Report LAS by dependency length and token position to show whether parser errors cluster on long-range attachments.
  • I5 (answer): Track the in-domain dev score during development and measure the gap on fables and school tests only occasionally.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations🏢OpenAISan Francisco · United States13 works · 18,282 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,344 works

💡 Concepts 2Key ideas and methods explained on our concept pages.Language model · 29 worksScaling laws · 30 works

Cite

Cite as Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2001.08361 · arXiv 2001.08361

@misc{kaplan2020scaling,
  title = {{Scaling Laws for Neural Language Models}},
  author = {Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei},
  year = {2020},
  doi = {10.48550/arxiv.2001.08361},
  eprint = {2001.08361},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.08361}
}

References

50 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eduardo G Altmann, Giampaolo Cristadoro, and Mirko Degli Esposti. On the origin of long-range correlations in texts. Proceedings of the National Academy of Sciences, 109(29):11582– 11587, 2012. 25 find
  2. Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv, 2017, 1710.03667. 11, 18, 22 find
  3. Michele Banko and Eric Brill. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th annual meeting on association for computational linguistics, pages 26–33. Association for Computational Linguistics, 2001. 18
  4. Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the bias-variance trade-off. arXiv, 2018, 1812.11118. 18 find
  5. GÊrard Biau. Analysis of a random forests model. Journal of Machine Learning Research, 13(Apr):1063–1095, 2012. 18 find
  6. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019, 1904.10509. URL http://arxiv.org/abs/1904.10509. 19 ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv:1810.04805. 2
  8. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. CoRR, abs/1807.03819, 2018, 1807.03819. URL http://arxiv.org/abs/1807.03819. 6, 9, 23, 24 ↗
  9. Werner Ebeling and Thorsten Pöschel. Entropy and long-range correlations in literary english. EPL (Europhysics Letters), 26(4):241, 1994. 25 find
  10. The Common Crawl Foundation. Common crawl. URL http://commoncrawl.org. 7 ↗
  11. Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. 2018, arXiv:1812.04754. 18 — [arXiv:1812.04754](https://arxiv.org/abs/1812.04754) find
  12. Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d’Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. arXiv, 2019, 1901.01608. 18 find
  13. Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. CoRR, abs/1901.10159, 2019, 1901.10159. URL http://arxiv.org/abs/1901.10159. 18 — [arXiv:1901.10159](https://arxiv.org/abs/1901.10159) ↗
  14. Joshua Goodman. A bit of progress in language modeling. CoRR, cs.CL/0108005, 2001. URL http://arxiv.org/abs/cs.CL/0108005. 18 ↗
  15. Scott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights. openai.com, 2017. 19 find
  16. Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: Computational challenges in deep learning. In Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming, PPoPP ’19, pages 1–14, New York, NY, USA, 2019. ACM. doi:10.1145/3293883.3295710. 18 — [doi:10.1145/3293883.3295710](https://doi.org/10.1145/3293883.3295710) find
  17. Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. CoRR, abs/1811.06965, 2018, 1811.06965. URL http://arxiv.org/abs/1811.06965. 19 ↗
  18. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017, 1712.00409. 18
  19. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571–8580, 2018. 18 find
  20. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014, 1412.6980. 7 find
  21. Aran Komatsuzaki. One epoch is all you need, 2019, arXiv:1906.06669. 18 — [arXiv:1906.06669](https://arxiv.org/abs/1906.06669) find
  22. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, pages 1097–1105, USA, 2012. Curran Associates Inc. URL http://dl… find
  23. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2019, 1909.11942. 9 find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019, 1907.11692. URL http://arxiv.org/abs/1907.11692. 2 ↗
  25. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv:1801.10198 [cs], 2018, 1801.10198. URL http://arxiv.org/abs/1801.10198. 2, 6 ↗
  26. Henry W Lin and Max Tegmark. Criticality in formal languages and statistical physics. arXiv preprint arXiv:1606.06737, 2016. 25 — [arXiv:1606.06737](https://arxiv.org/abs/1606.06737) find
  27. Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent, 2019, arXiv:1902.06720. 18 — [arXiv:1902.06720](https://arxiv.org/abs/1902.06720) find
  28. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training, 2018, arXiv:1812.06162. 3, 5, 6, 12, 13, 21 find
  29. Vardan Papyan. The full spectrum of deep net hessians at scale: Dynamics with sample size. CoRR, abs/1811.07062, 2018, 1811.07062. URL http://arxiv.org/abs/1811.07062. 18 — [arXiv:1811.07062](https://arxiv.org/abs/1811.07062) ↗
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf, 2018. 2, 6 ↗
  31. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, 1909.12673. 18 find
  32. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, arXiv:1909.12673. 18 find
  33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019, arXiv:1910.10683. 2
  34. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. openai.com, 2019. 2, 5, 6, 7, 8 find
  35. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers, 2018, 1811.02084. 19
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. CoRR, 2015, 1508.07909. 6
  37. Christopher J. Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training, 2018, arXiv:1811.03600. 12 — [arXiv:1811.03600](https://arxiv.org/abs/1811.03600) find
  38. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. CoRR, abs/1804.04235, 2018, 1804.04235. URL http://arxiv.org/abs/1804.04235.7 ↗
  39. Stefan Thurner, Rudolf Hanel, and Peter Klimek. Introduction to the theory of complex systems. Oxford University Press, 2018. 18 find
  40. Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.11946, 2019, 1905.11946. URL http://arxiv.org/abs/1905.11946. 18 ↗
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing S…
  42. Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks, 2016, arXiv:1605.06431. 8, 18 — [arXiv:1605.06431](https://arxiv.org/abs/1605.06431) find
  43. Larry Wasserman. All of nonparametric statistics. Springer Science & Business Media, 2006. 18 + find
  44. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2019, 1905.00537. 2
  45. Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Growing a brain: Fine-tuning by increasing model capacity. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi:10.1109/cvpr.2017.323. 19 — [doi:10.1109/cvpr.2017.323](https://doi.org/10.1109/cvpr.2017.323) find
  46. Wei Wen, Feng Yan, and Hai Li. Autogrow: Automatic layer growing in deep convolutional networks, 2019, 1906.02909. 19 find
  47. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2019, arXiv:1906.08237. 2
  48. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. Procedings of the British Machine Vision Conference 2016, 2016. doi:10.5244/c.30.87. 18 + — [doi:10.5244/c.30.87](https://doi.org/10.5244/c.30.87) find
  49. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi:10.1109/icc…
  50. Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, and Roger B. Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. CoRR, abs/1907.04164, 2019, 1907.04164. URL http://arxiv.org/abs/1907.0416… — [arXiv:1907.04164](https://arxiv.org/abs/1907.04164) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/