Elamur Research⌕
☰
home / papers / 2019 / A Constructive Prediction of the Generalization Error Across Scales

A Constructive Prediction of the Generalization Error Across Scales

Paper · 2019 · Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov et al.

Summary

Large networks trained on big datasets tend to win, but training them is expensive. Practitioners rarely know beforehand how much data or model capacity a target error will require. This paper looks for an explicit formula linking test error to both training-set size and parameter count. It assumes a fixed scaling policy, mainly width scaling with some depth scaling, so that every model size maps to one fully specified architecture. Hyper-parameters are held constant, so no search is needed at larger scales. The authors trained many subsampled-data and resized-model combinations across language modeling and image classification. Across these benchmarks, error follows saturating power laws in data size and in model size. From these patterns they built a joint function of both variables with a small number of interpretable parameters. This function fits measured errors closely and can be calibrated on cheap, small-scale runs to predict large-scale performance. That makes it useful for planning compute and data budgets and for architecture search.

Key points

  • Language modeling: PTB (0.9M words) and WikiText-2 (2M words) with AWD-LSTM (20M non-embedding parameters), and WikiText-103 (100M words) with Transformer-XL (41M).
  • Vision: ImageNet with ResNet-50 (25.5M parameters), plus CIFAR10, CIFAR100, DTD, Aircraft and UCF101 with WRN-44-16 (0.7M at base width).
  • Scaling grids:
    • Data was subsampled in factors of 2^-k. Vision classes were subsampled uniformly, and language corpora by sampling sentences.
    • Models were resized in factors of 4^-k.
    • Test sets were never modified.
  • Fit across all scales: the mean and standard deviation of relative error stay below 2%.
  • Extrapolation: calibrating on a model with 1/16 of the parameters and 1/8 of the data predicts full-scale error within about 5% on ImageNet and WikiText-103. Other datasets show comparable accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🎓Technion – Israel Institute of TechnologyHaifa · Israel54 works · 7,237 citations🎓York UniversityToronto · Canada15 works · 3,325 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov and Nir Shavit. 2019. A Constructive Prediction of the Generalization Error Across Scales. arXiv:1909.12673.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.12673 · arXiv 1909.12673

@misc{rosenfeld2019constructive,
  title = {{A Constructive Prediction of the Generalization Error Across Scales}},
  author = {Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov and Nir Shavit},
  year = {2019},
  doi = {10.48550/arxiv.1909.12673},
  eprint = {1909.12673},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.12673}
}

References

48 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. arXiv preprint arXiv:1811.04918, 2018a. find
  2. Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. arXiv preprint arXiv:1810.12065, 2018b. find
  3. Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296, 2018. — [arXiv:1802.05296](https://arxiv.org/abs/1802.05296) find
  4. Michele Banko and Eric Brill. Mitigating the paucity-of-data problem: Exploring the effect of training corpus size on classifier performance for natural language processing. In Proceedings of the first international conference on Human language technology research, pp. 1–5. Association for Computati… find
  5. Hakan Bilen, Basura Fernando, Efstratios Gavves, Andrea Vedaldi, and Stephen Gould. Dynamic image networks for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3034–3042, 2016. find
  6. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. In International Conference on Learning Representations, 2017. find
  7. Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, and Synho Do. How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? arXiv preprint arXiv:1511.06348, 2015. — [arXiv:1511.06348](https://arxiv.org/abs/1511.06348) find
  8. Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606–3613, 2014. find
  9. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978–2988, Florence, Italy, July 2019. As… find
  10. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019. find
  11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. pp. 770–778, 2016.
  12. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
  13. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  14. Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=S1Dh8Tg0-. ↗
  15. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  16. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. 1(2):3, 2017. find
  17. Roxana Istrate, Florian Scheidegger, Giovanni Mariani, Dimitrios Nikolopoulos, Costas Bekas, and A Cristiano I Malossi. Tapas: Train-less accuracy predictor for architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3927–3934, 2019. find
  18. Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy on large datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 450–455, Melbourne, Australia, July 2018. Association fo… find
  19. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
  20. Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial Intelligence and Statistics, pp. 528–536, 2017. find
  21. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. find
  22. Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the risk of minimum-norm interpolants and restricted lower isometry of kernels. arXiv preprint arXiv:1908.10292, 2019. — [arXiv:1908.10292](https://arxiv.org/abs/1908.10292) find
  23. Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018. find
  24. Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. — [arXiv:1306.5151](https://arxiv.org/abs/1306.5151) find
  25. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  26. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=SyyGPP0TZ. ↗
  27. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1489–1494, Copenhagen, Denmark, September 2017. Asso…
  28. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. find
  29. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pp. 5947–5956, 2017a. find
  30. Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564, 2017b. — [arXiv:1707.09564](https://arxiv.org/abs/1707.09564) find
  31. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017. find
  32. E Real, A Aggarwal, Y Huang, and QV Le. Aging evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence, 2019. find
  33. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018. find
  34. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506–516, 2017.
  35. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  36. Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. arXiv:1907.10597, 2019.
  37. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  38. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) find
  39. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843–852, 2017.
  40. Alon Talmor and Jonathan Berant. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4911–4921, Florence, Italy, July 2019. Association for Computational Linguisti…
  41. Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105–6114, 2019.
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sys…
  43. Green ai. arXiv preprint Zifeng Wu, Chunhua Shen, and Anton van den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. arXiv preprint arXiv:1611.10080, 2016. — [arXiv:1611.10080](https://arxiv.org/abs/1611.10080) find
  44. Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. arXiv preprint arXiv:1802.03620, 2018. — [arXiv:1802.03620](https://arxiv.org/abs/1802.03620) find
  45. Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016. — [arXiv:1605.07146](https://arxiv.org/abs/1605.07146) find
  46. Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.06906, 2018. — [arXiv:1807.06906](https://arxiv.org/abs/1807.06906) find
  47. Xiangxin Zhu, Carl Vondrick, Deva Ramanan, and Charless C Fowlkes. Do we need more training data or better models for object detection?. In BMVC, volume 3, pp. 5. Citeseer, 2012. find
  48. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697–8710, 2018.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosenfeld-2019-constructive-prediction-generalization-error-across/