Elamur Research⌕
☰
home / papers / 2021 / Scaling Laws for Transfer

Scaling Laws for Transfer

Summary

This paper asks how much language-model pre-training is worth when the target task has little data. It answers in units of data. The authors trained transformers across roughly four orders of magnitude in both non-embedding parameters and dataset size, using three curricula:

  • Python only, trained from scratch.
  • Natural text (WebText2, Common Crawl, English Wikipedia, Internet Books), then fine-tuned on Python.
  • A 50/50 mix of text and non-Python code, then fine-tuned on Python.

For each fine-tuned model they computed D_T, the "effective data transferred." This is the number of extra Python characters a same-sized from-scratch model would need to match its held-out loss. When fine-tuning data is scarce, D_T follows a simple power law in fine-tuning set size and model size. Pre-training therefore acts as a multiplier on the fine-tuning data, and the multiplier shrinks as data grows. From-scratch models stall once data-limited, while pre-trained ones keep gaining from scale. This offers a cheap way to decide between collecting more data and enlarging the model. It also quantifies how near one distribution is to another.

Key points

  • Fitted law: D_T = k·(D_F)^α·(N)^β. For text→Python, k = 1.9e4, α = 0.18, β = 0.38. For mixed text+code→Python, k = 2.1e5, α = 0.096, β = 0.38. The identical β suggests it depends on architecture and target, not on the source distribution.
  • Since β ≈ 2α for text→Python, a 10× larger model is roughly equivalent to 100× more fine-tuning data.
  • Example: for a 40M-parameter model fine-tuned on 3e5 characters, D_T is about 1000× D_F.
  • "Low-data regime" means having at most 10% of the data needed to reach 99% of infinite-data performance. Here, ignoring pre-training cost, fine-tuning is also more compute-efficient.
  • "Ossification": small pre-trained models never caught up with from-scratch models even at 10–100× D(N). The non-Python code corpus used for pre-training totals 240B characters (340 GB).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics Language models & pretraining

Cite

Cite as Danny Hernandez, Jared Kaplan, Tom Henighan and Sam McCandlish. 2021. Scaling Laws for Transfer. arXiv:2102.01293.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.01293 · arXiv 2102.01293

@misc{hernandez2021scaling,
  title = {{Scaling Laws for Transfer}},
  author = {Danny Hernandez, Jared Kaplan, Tom Henighan and Sam McCandlish},
  year = {2021},
  doi = {10.48550/arxiv.2102.01293},
  eprint = {2102.01293},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.01293}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei and Danny Hernandez. AI and Compute, May 2018. URL https://blog.openai.com/ai-and-compute/. 3 ↗
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv:1810.04805. 3, 5 our card
  4. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009. URL http://www.image-net.org/papers/ imagenet_cvpr09.pdf. 9 ↗
  5. Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning, 2015, 1511.01432. 3 find
  6. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017, 1703.03400. 9 find
  7. Danny Hernandez and Tom B. Brown. Measuring the algorithmic efficiency of neural networks, 2020, 2005.04305. 3 find
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021, 2009.03300. 9 our card
  9. Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4918–4927, 2019. 9 find
  10. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for auto… our card
  11. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017, 1712.00409. 3, 9 our card
  12. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification, 2018, 1801.06146. 3 find
  13. Diederik P. Kingma and Jimmy Ba. 1412.6980. 6 Adam: A method for stochastic optimization, 2014, find
  14. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020, 2001.08361. 3, 6, 8, 9, 10, 11, 18 our card
  15. Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015, https://science.sciencemag.org/content/350/6266/1332.full.pdf. doi:10.1126/science.aab3050. 9 ↗
  16. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining, 2018, 1805.00932. 9 find
  17. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt, 2019, 1912.02292. 19 find
  18. OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik’s cu… find
  19. OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/