Elamur Research
☰
home / papers / 2022 / Beyond neural scaling laws: beating power law scaling via data pruning

Beyond neural scaling laws: beating power law scaling via data pruning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2206.14486

Summary

Neural scaling laws mean test error falls only as a slow power of training-set size, so each further gain needs far more data and compute. The authors ask whether choosing which examples to keep can beat this. They use statistical mechanics to analyse a perceptron in a student-teacher setup, where examples are ranked by their margin. The analysis yields two predictions. First, the right strategy depends on how much data you start with: keep hard examples when data is plentiful and easy ones when it is scarce. Second, if the pruning fraction is tuned well, error can fall exponentially with pruned dataset size. They see both effects with ResNets on SVHN, CIFAR-10 and ImageNet, and with fine-tuned Vision Transformers. They also benchmark ten pruning metrics on ImageNet and find that most transfer poorly. To address this, they propose a label-free metric that scores each example by its distance to k-means prototypes in a self-supervised embedding space. It performs about as well as the best expensive supervised metrics. The results suggest that small, carefully curated "foundation datasets" could cut the cost of large-scale training.

Key points

  • The theory is an analytic student-teacher perceptron model, and it closely matches simulations (N=200). It shows a Pareto-optimal pruning fraction that gives better-than-power-law scaling, and the same switch between keeping easy and keeping hard examples also appears for ResNet18 on CIFAR-10 with the EL2N score.
  • The ImageNet benchmark compares ten metrics, including EL2N, forgetting scores, memorization, ensemble active learning and DDD. Most do poorly at this scale, and only the most compute-heavy ones (such as memorization) hold up.
  • The self-supervised prototype metric removes 20% of ImageNet with no loss in accuracy, matching the best supervised metric while needing no labels or ensemble training. Its results stay stable when the number of clusters k is up to 10× above or below 1000.
  • A supervised version that uses one prototype per class performs about as well as memorization at a small fraction of the cost.
  • The paper names three limitations: exponential scaling depends on a high-quality metric, class balancing is needed to keep performance, and accuracy trades off against training time on the pruned data. It was published at NeurIPS 2022 (Stanford, Tübingen and Meta AI FAIR).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

100 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 96 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.14486 · arXiv 2206.14486

@misc{sorscher2022beyond,
  title = {{Beyond neural scaling laws: beating power law scaling via data pruning}},
  author = {Sorscher, Ben and Geirhos, Robert and Shekhar, Shashank and Ganguli, Surya and Morcos, Ari S.},
  year = {2022},
  doi = {10.48550/arxiv.2206.14486},
  eprint = {2206.14486},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.14486}
}

References

54 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. find
  2. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. find
  3. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. find
  4. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. International Conference on Learning Representations, 2020. find
  5. Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915–5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Comput… find
  6. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021. find
  7. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04560, 2021. — [arXiv:2106.04560](https://arxiv.org/abs/2106.04560) find
  8. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. find
  9. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019. find
  10. Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Adv. Neural Inf. Process. Syst., 34, December 2021. find
  11. Kashyap Chitta, José M Álvarez, Elmar Haussmann, and Clément Farabet. Training data subset search with ensemble active learning. IEEE Trans. Intell. Transp. Syst., pages 1–12, 2021. find
  12. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. find our card
  13. Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Adv. Neural Inf. Process. Syst., 33:2881–2891, 2020. find
  14. Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Estimating informativeness of samples with smooth unique information. In International Conference on Learning Representations, 2021. find
  15. Burr Settles. Active learning literature survey. Technical Report, 2009. find our card
  16. Antoine Bordes, Seyda Ertekin, Jason Weston, Léon Botton, and Nello Cristianini. Fast kernel classifiers with online and active learning. J. Mach. Learn. Res., 6(9), 2005. find
  17. Zeyad Ali Sami Emam, Hong-Min Chu, Ping-Yeh Chiang, Wojciech Czaja, Richard Leapman, Micah Goldblum, and Tom Goldstein. Active learning at the ImageNet scale. arXiv preprint arXiv:2111.12880, 2021. — [arXiv:2111.12880](https://arxiv.org/abs/2111.12880) find
  18. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017. find
  19. Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning. Mind your outliers! Investigating the negative impact of outliers on active learning for visual question answering. arXiv preprint arXiv:2107.02331, 2021. find
  20. Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Hal Daumé Iii and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6950–… find
  21. V Birodkar, H Mobahi, and S Bengio. Semantic redundancies in Image-Classification datasets: The 10% you don’t need. arXiv preprint arXiv:1901.11409, 2019. — [arXiv:1901.11409](https://arxiv.org/abs/1901.11409) find
  22. Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, and Felix A. Wichmann. Trivial or impossible—dichotomous data difficulty masks model differences (on ImageNet and beyond). In International Conference on Learning Representations, 2022. find
  23. Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1–34, 2022. find
  24. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. CoRR, abs/2102.06701, 2021. — [arXiv:2102.06701](https://arxiv.org/abs/2102.06701) find
  25. Jonathan S. Rosenfeld. Scaling Laws for Deep Learning. PhD thesis, Massachusetts Institute of Technology, USA, 2021. find
  26. A Engel and C V den Broeck. Statistical Mechanics of Learning. Cambridge Univ. Press, 2001. find
  27. Madhu Advani, Subhaneil Lahiri, and Surya Ganguli. Statistical mechanics of complex neural systems and high dimensional data. J. Stat. Mech: Theory Exp., 2013(03):P03014, 2013. find
  28. Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, March 2020. find
  29. Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: thresholds and algorithms. Adv. Phys., 65(5):453–552, September 2016. find
  30. E Gardner. The space of interactions in neural network models. J. of Physics A, 21:257–270, 1988. find
  31. H S Seung, H Sompolinsky, and N Tishby. Statistical mechanics of learning from examples. Phys. Rev. A, 45(8):6056, 1992. find
  32. Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby. Information, prediction, and query by committee. Adv. Neural Inf. Process. Syst., 5, 1992. find
  33. Hai-Jun Zhou. Active online learning in the binary perceptron problem. Commun. Theor. Phys., 71(2):243, February 2019. find
  34. Hugo Cui, Luca Saglietti, and Lenka Zdeborovà. Large deviations in the perceptron model and consequences for active learning, 2021. find
  35. M Mezard, G Parisi, and M A Virasoro. Spin glass theory and beyond. World scientific Singapore, 1987. find
  36. Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vo… find
  37. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 20… find our card
  38. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181–196, 2018. find
  39. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. find our card
  40. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  41. Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. VISSL. https://github.com/facebookresearch/vissl, 2021. ↗
  42. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015. find our card
  43. Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the ImageNet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 547–558, 2020. find
  44. Yuki M Asano, Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi. PASS: An ImageNet replacement for self-supervised pretraining without humans. arXiv preprint arXiv:2109.13228, 2021. — [arXiv:2109.13228](https://arxiv.org/abs/2109.13228) find
  45. Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019. ↗
  46. Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1–54, 2019. find
  47. Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885– 23899, 2021. find
  48. Felix A Wichmann, David HJ Janssen, Robert Geirhos, Guillermo Aguilar, Heiko H Schütt, Marianne Maertens, and Matthias Bethge. Methods and measurements to compare men against machines. Electronic Imaging, Human Vision and Electronic Imaging, 2017(14):36–45, 2017. find
  49. Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems, 2018. find
  50. Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. find
  51. Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019. find
  52. Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency. Advances in Neural Information Processing Systems, 33, 2020. find
  53. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026–1034, 2015. find
  54. John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning,… find