Elamur Research⌕
☰
home / papers / 2023 / The Quantization Model of Neural Scaling

The Quantization Model of Neural Scaling

Summary

The paper tackles a puzzle about scaling. Average loss falls smoothly as a power law when models and datasets grow, yet specific abilities often seem to switch on abruptly. To reconcile the two, the authors propose the Quantization Hypothesis. It holds that prediction tasks rely on a countable set of discrete skills or knowledge units, called quanta. Each quantum is either acquired or missing, and models pick them up in order of how often they help on training data. If those usage frequencies follow a Zipf-like power law, the theory yields power-law loss curves in parameters and data. The smooth aggregate trend is then an average of many small, sudden jumps. The authors test this on synthetic toy tasks and then examine how scaling curves break down for real language models. They also introduce a gradient-clustering method, QDG, which automatically groups next-token predictions into coherent skills. The frequencies of the discovered clusters roughly match the power law implied by observed LLM scaling exponents, though the authors call this measurement imprecise. The work matters for forecasting new capabilities and for mechanistic interpretability.

Key points

  • Core model: the k-th quantum is useful on a fraction of samples proportional to k^−(α+1). A model that has learned n quanta has excess loss (L_n − L_∞) scaling as n^−α.
  • Parameter and data scaling:
    • If each quantum needs about C parameters, the parameter exponent α_N equals α.
    • If each quantum needs τ training examples, the multi-epoch data exponent is α_D = α/(α+1).
    • Together these predict α_D = α_N/(α_N + 1).
  • Monogenic vs. polygenic samples: the basic derivation assumes each sample depends on at most one quantum. The authors use this distinction to explain why some tasks improve gradually while others show breakthroughs.
  • Discovered skills: QDG, applied to text from The Pile, recovers clusters such as incrementing numerical sequences (Cluster 50) and newline prediction in line-length-limited text (Cluster 100).
  • Limitations and availability: the authors acknowledge that QDG is neither principled nor scalable. They also note that the quanta are modeled as independent, ignoring possible dependencies between skills. The paper appeared at NeurIPS 2023, and code is at github.com/ejmichaud/quantization-model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Efficient models & compression

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.13506 · arXiv 2303.13506

@misc{michaud2023quantization,
  title = {{The Quantization Model of Neural Scaling}},
  author = {Michaud, Eric J. and Liu, Ziming and Girit, Uzay and Tegmark, Max},
  year = {2023},
  doi = {10.48550/arxiv.2303.13506},
  eprint = {2303.13506},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.13506}
}

References

36 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. “Deep learning scaling is predictable, empirically”. In: arXiv preprint arXiv:1712.00409 (2017). Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, a… find our card
  2. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. “Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small”. In: arXiv preprint arXiv:2211.00593 (2022). find
  3. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… find
  4. Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. “Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla”. In: arXiv preprint arXiv:2307.09458 (2023). find our card
  5. Stephen Casper, Shlomi Hod, Daniel Filan, Cody Wild, Andrew Critch, and Stuart Russell. “Graphical clusterability and local specialization in deep neural networks”. In: ICLR 2022 Workshop on PAIR {\\textasciicircum} 2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structur… find
  6. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… find
  7. Jonathan Frankle and Michael Carbin. “The lottery ticket hypothesis: Finding sparse, trainable neural networks”. In: arXiv preprint arXiv:1803.03635 (2018). find
  8. Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. “Discovering Knowledge-Critical Subnetworks in Pretrained Language Models”. In: arXiv preprint arXiv:2310.03084 (2023). find
  9. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent Learning: Do different neural networks learn the same representations? arXiv:1511.07543 [cs]. Feb. 2016. URL : http://arxiv.org/abs/1511.07543 (visited on 08/19/2022). — [arXiv:1511.07543](https://arxiv.org/abs/1511.07543) ↗
  10. Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. arXiv:2010.15327 [cs]. Apr. 2021. URL : http://arxiv.org/abs/2010.15327. — [arXiv:2010.15327](https://arxiv.org/abs/2010.15327) ↗
  11. Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta Neurons: Mining the Common Units in a Model Zoo. arXiv:2306.09346 [cs]. June 2023. DOI : 10.48550/arXiv. 2306.09346. URL : http://arxiv.org/abs/2306.09346. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) ↗
  12. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… find
  13. Gwern Branwen. The scaling hypothesis. 2021. URL : https://gwern.net/scaling -hypothesis. ↗
  14. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. “Grokking: Generalization beyond overfitting on small algorithmic datasets”. In: arXiv preprint arXiv:2201.02177 (2022). find
  15. Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. “Hidden progress in deep learning: Sgd learns parities near the computational limit”. In: arXiv preprint arXiv:2207.08799 (2022). find
  16. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. “Pythia: A suite for analyzing large language models across training and scaling”. In: arXiv preprint arXiv:2304.01373 (2… find
  17. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. “The pile: An 800gb dataset of diverse text for language modeling”. In: arXiv preprint arXiv:2101.00027 (2020). find
  18. Utkarsh Sharma and Jared Kaplan. “Scaling Laws from the Data Manifold Dimension”. In: Journal of Machine Learning Research 23.9 (2022), pp. 1–34. URL : http://jmlr.org/ papers/v23/20-1111.html. ↗
  19. Eric J. Michaud, Ziming Liu, and Max Tegmark. “Precision Machine Learning”. In: Entropy 25.1 (2023). ISSN : 1099-4300. DOI : 10.3390/e25010175. URL : https://www.mdpi.com/ 1099-4300/25/1/175. — [doi:10.3390/e25010175](https://doi.org/10.3390/e25010175) ↗
  20. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. “Explaining neural scaling laws”. In: arXiv preprint arXiv:2102.06701 (2021). — [arXiv:2102.06701](https://arxiv.org/abs/2102.06701) find
  21. Alexander Maloney, Daniel A Roberts, and James Sully. “A Solvable Model of Neural Scaling Laws”. In: arXiv preprint arXiv:2210.16859 (2022). — [arXiv:2210.16859](https://arxiv.org/abs/2210.16859) find
  22. Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. “Spectrum dependent learning curves in kernel regression and wide neural networks”. In: International Conference on Machine Learning. PMLR. 2020, pp. 1024–1034. find
  23. Marcus Hutter. “Learning curve theory”. In: arXiv preprint arXiv:2102.04074 (2021). — [arXiv:2102.04074](https://arxiv.org/abs/2102.04074) find
  24. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models”. In: arXiv preprint arXiv:2206.04615 (… find our card
  25. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. “Are Emergent Abilities of Large Language Models a Mirage?” In: arXiv preprint arXiv:2304.15004 (2023). find our card
  26. Sanjeev Arora and Anirudh Goyal. “A theory for emergence of complex skills in language models”. In: arXiv preprint arXiv:2307.15936 (2023). — [arXiv:2307.15936](https://arxiv.org/abs/2307.15936) find
  27. Lorenza Saitta, Attilio Giordana, and Antoine Cornuejols. Phase transitions in machine learning. Cambridge University Press, 2011. find
  28. James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. “On the stepwise nature of self-supervised learning”. In: arXiv preprint arXiv:2303.15438 (2023). — [arXiv:2303.15438](https://arxiv.org/abs/2303.15438) find
  29. Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. “Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models”. In: arXiv preprint arXiv:2307.14430 (2023). — [arXiv:2307.14430](https://arxiv.org/abs/2307.14430) find
  30. Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. “Data distributional properties drive emergent in-context learning in transformers”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 18878–18891. find
  31. Marvin Minsky. Society of mind. Simon and Schuster, 1988. find
  32. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. “Scikit-learn: Machine Learning in Python”. In: Journal of Machine Learning Research 12 (201… find
  33. Ronen Eldan and Yuanzhi Li. “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?” In: arXiv preprint arXiv:2305.07759 (2023). find our card
  34. Pablo Villalobos. Scaling Laws Literature Review. https://epochai.org/blog/scaling-laws-literature-review. Accessed: 2023-01-31. 2023. ↗
  35. Jasha Droppo and Oguz Elibol. “Scaling laws for acoustic models”. In: arXiv preprint arXiv:2106.09488 (2021). — [arXiv:2106.09488](https://arxiv.org/abs/2106.09488) find
  36. Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. “Understanding Scaling Laws for Recommendation Models”. In: arXiv preprint arXiv:2208.08489 (2022). — [arXiv:2208.08489](https://arxiv.org/abs/2208.08489) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/