Elamur Research⌕
☰
home / papers / 2020 / Scaling Laws for Autoregressive Generative Modeling

Scaling Laws for Autoregressive Generative Modeling

Summary

The authors from OpenAI examine whether the scaling laws previously found for language models also hold for other data types. They trained series of autoregressive decoder-only Transformers of various sizes, ranging from roughly 10⁴ to several billion non-embedding parameters. The data cover four domains: YFCC100M images at various resolutions, video, caption–image pairs in both directions, and procedurally generated mathematical problems. In all domains, the cross-entropy loss decreases smoothly as model size and compute budget grow. This dependence is well described by a power law plus a constant. The authors interpret the constant as an estimate of the entropy of the data itself. They read the power-law term as the KL divergence separating the model from the real data distribution, i.e., as the information the model has not yet learned. The optimal model size for a given budget also grows as a power law, and its exponent is nearly the same for all modalities. The work makes it possible to plan model size, data volume, and compute in advance. In addition, it shows that improvements in generative training transfer to downstream tasks, in particular classification.

Key points

  • Form of the law: L(x) = L∞ + (x₀/x)^αₓ, where x is the number of parameters N, compute C, and in some cases also dataset size D. The exponent αₓ depends on the modality. A smaller exponent means that the data can be "compressed" further only by much larger models.
  • Optimal size: N_opt ∝ C^β, where β ≈ 0.7 across all domains. Hence, if each example is used only once during training, the dataset size can grow sublinearly: approximately D ∝ N^0.4.
  • YFCC100M 8×8: models with about a billion parameters model these images almost perfectly, since only a few nats per image remain unlearned. For higher resolutions, the laws make it possible to predict the required model size.
  • ImageNet 32×32: after fine-tuning generative models for classification, the loss and error rate decrease smoothly with increasing size. This happens even when the generative loss has almost stopped decreasing.
  • Multimodal models and mathematics: the authors introduce the metric Infogain = I(text, image) / L(text). It is always less than 1 and grows smoothly with model size. It is used to estimate how many words a single image is "worth". For mathematical problems, scaling laws were found even for problems harder than those the model was trained on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cite

Cite as Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei and Sam McCandlish. 2020. Scaling Laws for Autoregressive Generative Modeling. arXiv:2010.14701.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2010.14701 · arXiv 2010.14701

@misc{henighan2020scaling,
  title = {{Scaling Laws for Autoregressive Generative Modeling}},
  author = {Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei and Sam McCandlish},
  year = {2020},
  doi = {10.48550/arxiv.2010.14701},
  eprint = {2010.14701},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.14701}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  2. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019, 1904.10509. URL http://arxiv.org/abs/1904.10509. 6, 7, 20 ↗
  3. Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/1707.08819, 2017, 1707.08819. URL http://arxiv.org/abs/1707.08819. 4, 14, 15 — [arXiv:1707.08819](https://arxiv.org/abs/1707.08819) ↗
  4. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of Machine Learning and Systems 2020, pages 10466–10478. 2020. 3, 19 find
  5. Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music, 2020, 2005.00341. 8 find
  6. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise?, 2019, 1908.08351. 16 find
  7. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017, 1712.00409. 3, 19 our card
  8. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571–8580, 2018. 19 find
  9. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020, 2001.08361. 3, 4, 5, 6, 7, 9, 10, 11, 17, 18, 19, 20, 33 our card
  10. Aran Komatsuzaki. One epoch is all you need, 2019, arXiv:1906.06669. 19 — [arXiv:1906.06669](https://arxiv.org/abs/1906.06669) find
  11. Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism, 2020, 2003.02218. 19 find
  12. Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. abs/1711.05101, 2017, 1711.05101. URL http://arxiv.org/abs/1711.05101. 6 ↗
  13. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv:1801.10198 [cs], 2018, 1801.10198. URL http://arxiv.org/abs/1801.10198. 3, 20 ↗ our card
  14. Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers, 2020, 2002.11794. 3, 19 find
  15. Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent, 2019, arXiv:1902.06720. 19 — [arXiv:1902.06720](https://arxiv.org/abs/1902.06720) find
  16. Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning. CoRR, abs/1712.06559, 2017, 1712.06559. URL http://arxiv.org/abs/1712.06559. 10 — [arXiv:1712.06559](https://arxiv.org/abs/1712.06559) ↗
  17. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training, 2018, arXiv:1812.06162. 10 find
  18. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. Recipes for building an open-domain chatbot, 2020, 2004.13637. 3, 19 CoRR, our card
  19. Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales, 2020, 2006.10621. 19 find
  20. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, 1909.12673. 3, 19 find
  21. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. CoRR, abs/1904.01557, 2019, 1904.01557. URL http://arxiv.org/abs/1904.01557. 3, 8, 16, 20, 26, 27, 28, 29 ↗ our card
  22. Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold, 2020, 2004.10802. 3, 19 + find
  23. Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. Enhancing the transformer with explicit relational encoding for math problem solving, 2019, 1910.06611. 20 + find
  24. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Pub… our card
  25. Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. The new data and new challenges in multimedia research. CoRR, abs/1503.01817, 2015, 1503.01817. URL http://arxiv.org/abs/1503.01817. 3, 4, 5, 7, 13, 34 ↗
  26. Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. CoRR, abs/1601.06759, 2016, 1601.06759. URL http://arxiv.org/abs/1601.06759. 19 — [arXiv:1601.06759](https://arxiv.org/abs/1601.06759) ↗
  27. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018, 1711.00937. 6, 7, 8, 11 + find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing S… our card
  29. Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models, 2019, 1906.02634. 3, 20 find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/