Elamur Research⌕
☰
home / papers / 2013 / Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Paper · 2013 · Andrew M. Saxe, James L. McClelland and Surya Ganguli

Summary

Why deep networks train slowly, stall on long plateaus and then suddenly improve, and benefit from certain initializations has lacked a quantitative explanation. Saxe, McClelland and Ganguli address this using deep linear networks. Their input-output function reduces to a single matrix, but gradient descent on the weights is a coupled, non-convex, nonlinear system whose behavior changes as layers are added. The authors take a continuous-time limit and rotate the weights into the singular-vector basis of the input-output correlation matrix. This yields exact solutions showing how each statistical mode of the data is learned, with timescales set by the mode's singular value, the initial weights and the depth. The theory reproduces the plateau-then-jump pattern and roughly matches simulations of nonlinear networks. Greedy unsupervised pretraining (whose conditions MNIST approximately meets) or random orthogonal weights make learning time nearly independent of depth, while scaled Gaussian weights do not. This property, called dynamical isometry, largely survives in deep tanh networks tuned near the edge of chaos, which gives principled guidance for initializing very deep models.

Key points

  • In a three-layer linear network with whitened inputs, training converges to the best rank-N₂ approximation of the input-output correlation matrix. The error surface has saddle points but no spurious local minima.
  • Connectivity modes tied to the same singular value reinforce each other, while modes tied to different singular values repel each other. Stronger modes are learned first, which produces stage-like drops in error.
  • Greedy layer-wise unsupervised pretraining speeds up optimization when the data meet specific conditions. The authors show these conditions hold approximately for MNIST.
  • Random orthogonal initialization lets arbitrarily deep linear networks learn with only a finite, depth-independent delay compared with shallow ones. Norm-preserving scaled Gaussian initialization fails to do this.
  • In nonlinear tanh networks with 100 layers of 1000 neurons, the end-to-end Jacobian stays well-conditioned at gain g ≈ 1, the edge of chaos. The spectrum collapses for g < 1 and becomes only mildly anisotropic for g slightly above 1.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Andrew M. Saxe, James L. McClelland and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1312.6120 · arXiv 1312.6120

@misc{saxe2013exact,
  title = {{Exact solutions to the nonlinear dynamics of learning in deep linear neural networks}},
  author = {Andrew M. Saxe, James L. McClelland and Surya Ganguli},
  year = {2013},
  doi = {10.48550/arxiv.1312.6120},
  eprint = {1312.6120},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1312.6120}
}

References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Krizhevsky, I. Sutskever, and G.E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, 2012.
  2. Q.V. Le, M.A. Ranzato, R. Monga, M. Devin, K. Chen, G.S. Corrado, J. Dean, and A.Y. Ng. Building high-level features using large scale unsupervised learning. In 29th International Conference on Machine Learning, 2012. find
  3. D. Ciresan, U. Meier, and J. Schmidhuber. Multi-column Deep Neural Networks for Image Classification. In IEEE Conf. on Computer Vision and Pattern Recognition, pages 3642–3649, 2012.
  4. A. Mohamed, G.E. Dahl, and G. Hinton. Acoustic Modeling Using Deep Belief Networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):14–22, January 2012. find
  5. R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In Proceedings of the 25th International Conference on Machine Learning, 2008. find
  6. R. Socher, J. Bauer, C.D. Manning, and A.Y. Ng. Parsing with Compositional Vector Grammars. In Association for Computational Linguistics Conference, 2013. find
  7. S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. PhD thesis, TU Munich, 1991. find
  8. Y. Bengio, P. Simard, and P. Frasconi. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994. find
  9. Y. LeCun, L. Bottou, G.B. Orr, and K.R. Müller. Efficient BackProp. Neural networks: Tricks of the trade, 1998. find
  10. Y. Bengio and Y. LeCun. Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. De-Coste, and J. Weston, editors, Large-Scale Kernel Machines, number 1, pages 1–41. MIT Press, 2007. find
  11. D. Erhan, P.A. Manzagol, Y. Bengio, S. Bengio, and P. Vincent. The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training. In 12th International Conference on Artificial Intelligence and Statistics, volume 5, 2009. find
  12. Y. Bengio. Learning Deep Architectures for AI. 2009. find
  13. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. 13th International Conference on Artificial Intelligence and Statistics, 2010. find
  14. D. Erhan, Y. Bengio, A. Courville, P.A. Manzagol, and P. Vincent. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11:625–660, 2010. find
  15. Y.N. Dauphin and Y. Bengio. Big Neural Networks Waste Capacity. In International Conference on Learning Representations, 2013. find
  16. A.M. Saxe, J.L. McClelland, and S. Ganguli. Learning hierarchical category structure in deep neural networks. In Proceedings of the 35th Annual Conference of the Cognitive Science Society, 2013. find
  17. G.E. Hinton and R.R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–7, July 2006. find
  18. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy Layer-Wise Training of Deep Networks. Advances in Neural Information Processing Systems 20, 2007. find
  19. P. Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):53–58, January 1989. find
  20. K. Fukumizu. Effect of Batch Learning In Multilayer Neural Networks. In Proceedings of the 5th International Conference on Neural Information Processing, pages 67–70, 1998. find
  21. J. Martens. Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning, 2010. find
  22. O. Chapelle and D. Erhan. Improved Preconditioner for Hessian Free Optimization. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. find
  23. I. Sutskever, J. Martens, G. Dahl, and G.E. Hinton. On the importance of initialization and momentum in deep learning. In 30th International Conference on Machine Learning, 2013. find
  24. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. Technical report, Universite de Montreal, 2012. find
  25. P. Lamblin and Y. Bengio. Important gains from supervised fine-tuning of deep architectures on large labeled sets. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saxe-2013-exact-solutions-nonlinear-dynamics-learning/