Elamur Research⌕
☰
home / papers / 2017 / Deep Learning Scaling is Predictable, Empirically

Deep Learning Scaling is Predictable, Empirically

Summary

Practitioners widely assume that more data and bigger models improve accuracy, but there was no reliable way to forecast how much gain a given increase in training data would bring. Classical learning theory predicts power-law learning curves with exponents of −0.5 or −1, yet these values fail to describe real deep learning systems. The Baidu Research authors ran a large empirical study. Using ever-larger portions of the training data, they trained six deep neural network models spanning four application areas: speech recognition, image classification, language modeling and machine translation. At each data scale they searched for the best-fitting model size, then fitted curves for generalization error and model size. In every domain, error falls as a power law of training set size, but the slopes are much shallower than theory suggests. Better architectures and optimizers lower the curve without changing its steepness, and the parameter count needed grows sublinearly with data. This predictability helps teams estimate the data and compute needed to reach a target accuracy, debug models, decide whether to grow datasets, and plan computing systems.

Key points

  • Empirical learning-curve exponents fall roughly between −0.07 and −0.35, while prior theory predicts −0.5, −1 or −2. The paper says these observed values are still unexplained.
  • The study covers six models in four domains and finds that the power-law behaviour holds across many models, optimizers, regularizers and loss functions.
  • Within one domain, model improvements shift only the intercept of the learning curve, not its exponent.
  • Learning curves have three regions:
    • a small-data region dominated by best guessing,
    • a power-law region,
    • a final plateau set by irreducible error (e.g., Bayes error).
  • Required model size grows sublinearly with data. The authors also re-estimate an exponent of about 0.72 for the Banko and Brill (2001) models. Appendix B proves that a simple counting model for a coin flip converges with exponent −0.5.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1712.00409 · arXiv 1712.00409

@misc{hestness2017deep,
  title = {{Deep Learning Scaling is Predictable, Empirically}},
  author = {Hestness, Joel and Narang, Sharan and Ardalani, Newsha and Diamos, Gregory and Jun, Heewoo and Kianinejad, Hassan and Patwary, Md. Mostofa Ali and Yang, Yang and Zhou, Yanqi},
  year = {2017},
  doi = {10.48550/arxiv.1712.00409},
  eprint = {1712.00409},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1712.00409}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Amari. A Universal Theorem on Learning Curves. Neural Networks, 6:161–166, 1993. find
  2. S. Amari and N. Murata. Statistical Theory of Learning Curves under Entropic Loss Criterion. Neural Computation, 5(1):140–153, 1993. find
  3. S. Amari, N. Fujita, and S. Shinomoto. Four Types of Learning Curves. Neural Computation, 4(4):605–618, 1992. find
  4. D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, et al. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin. In Proceedings of The International Conference on Machine Learning (ICML), pages 173–182, 2016. find our card
  5. D. Arpit, S. Jastrz˛ebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, and S. Lacoste-Julien. A Closer Look at Memorization in Deep Networks. In Proceedings of the International Conference on Machine Learning, August 2017. find
  6. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end Attention-based Large Vocabulary Speech Recognition. arXiv preprint arXiv:1508.04395v2, 2016. — [arXiv:1508.04395](https://arxiv.org/abs/1508.04395) find
  7. M. Banko and E. Brill. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In Proceedings of Association of Computational Linguistics (ACL), January 2001. find our card
  8. P. L. Bartlett and S. Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. In Journal of Machine Learning Research 3, pages 463–482, November 2002. find
  9. E. Battenberg, J. Chen, R. Child, A. Coates, Y. Gaur, Y. Li, H. Liu, S. Satheesh, D. Seetapun, A. Sriram, and Z. Zhu. Exploring Neural Transducers for End-to-end Speech Recognition. arXiv preprint arXiv:1707.07413, 2017. — [arXiv:1707.07413](https://arxiv.org/abs/1707.07413) find
  10. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the Vapnik-Chervonenkis Dimension. Journal of the ACM (JACM), 36(4):929–965, October 1989. find
  11. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. arXiv preprint arXiv:1312.3005, 2013. find our card
  12. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-based Models for Speech Recognition. In Advances in Neural Information Processing Systems (NIPS), pages 577–585, 2015. find our card
  13. J. Collins, J. Sohl-Dickstein, and D. Sussillo. Capacity and Trainability in Recurrent Neural Networks. In Proceedings of The International Conference on Learning Representations (ICLR), 2017. find
  14. G. K. Dziugaite and D. M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In Proceedings of The International Conference on Machine Learning (ICML), 2017. find
  15. A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant. A General Lower Bound on the Number of Examples Needed for Learning. Information and Computation, 82:247–261, 1989. Workshop on Computing Learning Theory, 1988. find
  16. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceedings of the International Conference on Machine Learning (ICML), 2006. find
  17. G. Györgyi and N. Tishby. Statistical Theory of Learning a Rule. Neural Networks and Spin Glasses, pages 3–36, 1990. find
  18. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, et al. Deep Speech: Scaling Up End-to-End Speech Recognition. arXiv preprint arXiv:1412.5567, 2014. find our card
  19. N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension Bounds for Piecewise Linear Neural Networks. In Proceedings of Machine Learning Research, volume 65, pages 1––12, 2017. find
  20. D. Haussler. Quantifying Inductive Bias: AI Learning Algorithms and Valiant’s Learning Framework. Artificial Intelligence, 36(2):177–221, 1988. find
  21. D. Haussler, M. Kearns, H. S. Seung, and N. Tishby. Rigorous Learning Curve Bounds from Statistical Mechanics. Machine Learning, 25(2):195–236, November 1996. find
  22. K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, June 2016. find
  23. R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the Limits of Language Modeling. arXiv preprint arXiv:1602.02410v2, 2016. find
  24. K. Kawaguchi, L. P. Kaelbling, and Y. Bengio. arXiv:1710.05468v1, October 2017. — [arXiv:1710.05468](https://arxiv.org/abs/1710.05468) find
  25. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of the Association for Computational Linguistics (ACL), 2017. find our card
  26. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the Association of Computational Linguistics, Interactiv… find
  27. M. Luong, E. Brevdo, and R. Zhao. https://github.com/tensorflow/nmt, 2017. ↗
  28. T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, 2015. find our card
  29. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv preprint arXiv:1409.0575, January 2015. find our card
  30. H. Schwarze and J. Hertz. Statistical Mechanics of Learning in a Large Committee Machine. In Advances in Neural Information Processing Systems, (NIPS), pages 523–530, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc. find
  31. R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. arXiv preprint arXiv:1508.07909, 2016a. find our card
  32. R. Sennrich, B. Haddow, and A. Birch. Edinburgh Neural Machine Translation Systems for WMT 16. arXiv preprint arXiv:1606.02891, 2016b. find our card
  33. H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical Mechanics of Learning from Examples. Physical Review A, 45:6056–6091, April 1992. find
  34. S. L. Smith and Q. V. Le. A Bayesian Perspective on Generalization and Stochastic Gradient Descent. arXiv preprint arXiv:1710.06451v2, October 2017. — [arXiv:1710.06451](https://arxiv.org/abs/1710.06451) find
  35. C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. In Proceedings of the International Conference on Computer Vision (ICCV), 2017. find
  36. H. G. Tucker. A Generalization of the Glivenko-Cantelli Theorem. The Annals of Mathematical Statistics, 30 (3):828–830, September 1959. find
  37. V. Vapnik. An Overview of Statistical Learning Theory. In IEEE Transactions on Neural Networks, volume 10, pages 988–999, September 1998. find
  38. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding Deep Learning Requires Rethinking Generalization. arXiv preprint arXiv:1611.03530v2, 2017. find
  39. J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent Highway Networks. In Proceedings of The International Conference on Machine Learning (ICML), 2017. find
  40. Generalization in Deep Learning. arXiv preprint Neural Machine Translation (seq2seq) Tutorial. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/