Elamur Research⌕
☰
home / papers / 2016 / Deep Networks with Stochastic Depth

Deep Networks with Stochastic Depth

Paper · 2016 · Gao Huang, Yu Sun, Zhuang Liu et al.

Summary

Very deep convolutional networks are hard to train. Gradients weaken on their way back to early layers, input features fade as they pass forward, and training cost grows with every added layer. Huang et al. propose stochastic depth, a way to train a short network while keeping a deep one for inference. Starting from the ResNet architecture of He et al., each residual block is switched off at random for every mini-batch, so only its identity shortcut remains. The chance that a block stays active falls linearly from 1 near the input to a final value p_L (0.5 by default) at the last block. At test time the complete network is used.

Because the expected depth during training is much smaller, training is faster and gradients reaching early layers are stronger. The method also works as a regularizer, behaving much like a hidden combination of many networks that vary in depth. On CIFAR-10 it made a ResNet with over 1200 layers trainable, reaching 4.91% test error. This matters because it is simple, works alongside Batch Normalization, and pushes usable depth further.

Key points

  • Each ResBlock gets a Bernoulli gate. Survival probability decays linearly to p_L, and the linear schedule consistently beat a uniform one on a 110-layer ResNet (CIFAR-10).
  • Results stay robust for p_L between roughly 0.4 and 0.8. Even at p_L = 0.2, accuracy remains competitive while training time drops by about 40%.
  • A ResNet deeper than 1200 layers still improved, reaching 4.91% test error on CIFAR-10.
  • Gradient magnitudes at the first convolutional layer were consistently larger than in constant-depth training, especially after learning-rate drops. By contrast, Dropout gave practically no gain on a 110-layer ResNet with Batch Normalization.
  • On ImageNet with a 152-layer ResNet there was no immediate benefit. The authors suggest that larger models may be needed there.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Cornell UniversityIthaca · United States134 works · 65,159 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra and Kilian Weinberger. 2016. Deep Networks with Stochastic Depth. arXiv:1603.09382.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.09382 · arXiv 1603.09382

@misc{huang2016deep,
  title = {{Deep Networks with Stochastic Depth}},
  author = {Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra and Kilian Weinberger},
  year = {2016},
  doi = {10.48550/arxiv.1603.09382},
  eprint = {1603.09382},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.09382}
}

References

35 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009) find
  2. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE (2009) 248–255 find
  3. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. (2012) 1097–1105
  4. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 (2013)
  5. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  6. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014) — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) find
  7. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 1–9 find
  8. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
  9. Håstad, J., Goldmann, M.: On the power of small-depth threshold circuits. Computational Complexity 1(2) (1991) 113–129 find
  10. Håstad, J.: Computational limitations of small-depth circuits. (1987) find
  11. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on 5(2) (1994) 157–166 find
  12. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: International conference on artificial intelligence and statistics. (2010) 249–256 find
  13. Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. arXiv preprint arXiv:1409.5185 (2014) find
  14. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015) find
  15. Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)
  16. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1) (2014) 1929–1958
  17. Fahlman, S.E., Lebiere, C.: The cascade-correlation learning architecture. (1989) find
  18. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research 11 (2010) 625–660 find
  19. Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10). (2010) 807–814 find
  20. Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In Dasgupta, S., Mcallester, D., eds.: Proceedings of the 30th International Conference on Machine Learning (ICML-13). Volume 28., JMLR Workshop and Conference Proceedings (May 2013) 1058–1066 find
  21. Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint arXiv:1302.4389 (2013) — [arXiv:1302.4389](https://arxiv.org/abs/1302.4389) find
  22. Smith, L.N., Hand, E.M., Doster, T.: Gradual dropin of layers to train very deep neural networks. CVPR (2016) find
  23. Zagoruyko, S.: 92.45% on cifar-10 in torch (2015) find
  24. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013) — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) find
  25. Graham, B.: Fractional max-pooling. arXiv preprint arXiv:1412.6071 (2014) — [arXiv:1412.6071](https://arxiv.org/abs/1412.6071) find
  26. Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014) — [arXiv:1412.6830](https://arxiv.org/abs/1412.6830) find
  27. Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 3367–3375 find
  28. Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Ali, M., Adams, R.P., et al.: Scalable bayesian optimization using deep neural networks. arXiv preprint arXiv:1502.05700 (2015) — [arXiv:1502.05700](https://arxiv.org/abs/1502.05700) find
  29. Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems. (2015) 2368–2376 find
  30. Lee, C.Y., Gallagher, P.W., Tu, Z.: Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985 (2015) — [arXiv:1509.08985](https://arxiv.org/abs/1509.08985) find
  31. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS workshop on deep learning and unsupervised feature learning. Volume 2011., Granada, Spain (2011) 4 find
  32. Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: A matlab-like environment for machine learning. In: BigLearn, NIPS Workshop. (2011) find
  33. Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th international conference on machine learning (ICML-13). (2013) 1139–1147 find
  34. Gross, S., Wilber, M.: Training and investigating residual nets (2016) find
  35. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027 (2016) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2016-deep-networks-stochastic-depth/