Elamur Research⌕
☰
home / papers / 2014 / Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Paper · 2014 · Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky et al.

Summary

Large neural networks with many parameters easily memorise sampling noise in limited training data, which leads to overfitting. Averaging many separately trained big nets would help, but training and running all of them is too costly. The authors propose dropout. During training, each unit (hidden or input) is temporarily removed at random, together with its connections, so every training case updates a different "thinned" sub-network. All these sub-networks share weights. At test time a single full network is used, with each unit's outgoing weights multiplied by its retention probability. This cheaply approximates a geometric-mean ensemble over exponentially many models. The method is tested on vision (MNIST, SVHN, CIFAR-10/100), speech (TIMIT), document classification (Reuters RCV1) and computational biology (alternative splicing). It beats other regularisers and reaches state-of-the-art results on many benchmarks. The paper also extends the idea to Restricted Boltzmann Machines and shows that, in simple cases, the noise can be marginalised into a deterministic regulariser. Dropout matters because it gives ensemble-like robustness for roughly the price of one model.

Key points

  • A net with n units implicitly contains 2^n weight-sharing thinned networks. A retention probability p = 0.5 for hidden units is near-optimal across many tasks, while input units should be kept with p closer to 1.
  • Denoising autoencoders typically work best with about 5% noise. Dropout with test-time weight scaling tolerates far more: dropping 20% of inputs and 50% of hidden units was often optimal.
  • On SVHN (about 600,000 training and 26,000 test images), the best convnet used per-layer retention p = (0.9, 0.75, 0.75, 0.5, 0.5, 0.5), a max-norm constraint with c = 4 and momentum 0.95.
  • TIMIT frame classification used windows of 21 frames, with p = 0.8 for inputs and 0.5 for hidden layers. Features were prepared with the Kaldi toolkit.
  • On a Reuters RCV1 subset (402,738 articles, 2000-word vocabulary, 50 categories), dropout helped every architecture tried. The gain was smaller than on images and speech, which the authors attribute to the large data set limiting overfitting.

Interesting ideas

  • I1. Rescaling weights at test time by the retention probability makes the deterministic output match the expected training-time output, so a whole implicit ensemble collapses into one cheap model.
  • I2. Random removal stops units from co-adapting, so each feature must be useful alone or with a few partners, like "small conspiracies" that survive changed conditions.
  • I3. Noise levels should differ by layer: raw inputs are dropped far less than hidden representations, since losing input information is more harmful.
  • I4. Combining models helps most when the members truly differ, either in architecture or in training data, and dropout manufactures that diversity inside one model.
  • I5. The benefit of a regulariser shrinks when data are abundant, as the Reuters result shows, so gains should be measured against data size.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could train with dropout and still run as a single deterministic network at parse time, so every parse stays reproducible and explainable.
  • I2 (tree): the perceptron parser could randomly drop feature templates during training, so it does not depend on one brittle lexical feature and generalises better to unseen domains and UD dialect conversions.
  • I3 (tokens): when the tagger or morphology model is trained with noise, the word-form and character inputs should get mild masking and the internal layers stronger masking.
  • I4 (learn): when silver data is built from several witness parsers, deliberately diverse witnesses (different architectures or data slices) should give more trustworthy agreement than near-copies.
  • I5 (learn): evaluations should report improvements across small and large training sizes, so a trick that only helps in the low-data regime is not mistaken for a general win.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada110 works · 179,061 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR 15(56):1929–1958.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{srivastava2014dropout,
  title = {{Dropout: A Simple Way to Prevent Neural Networks from Overfitting}},
  author = {Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov},
  booktitle = {JMLR 15(56):1929–1958},
  year = {2014},
  url = {https://jmlr.org/papers/v15/srivastava14a.html}
}

References

37 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Chen, Z. Xu, K. Weinberger, and F. Sha. Marginalized denoising autoencoders for domain adaptation. In Proceedings of the 29th International Conference on Machine Learning, pages 767–774. ACM, 2012. find
  2. G. E. Dahl, M. Ranzato, A. Mohamed, and G. E. Hinton. Phone recognition with the mean-covariance restricted Boltzmann machine. In Advances in Neural Information Processing Systems 23, pages 469–477, 2010. find
  3. O. Dekel, O. Shamir, and L. Xiao. Learning to classify with missing and corrupted features. Machine Learning, 81(2):149–178, 2010. find
  4. A. Globerson and S. Roweis. Nightmare at test time: robust learning by feature deletion. In Proceedings of the 23rd International Conference on Machine Learning, pages 353–360. ACM, 2006. find
  5. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In Proceedings of the 30th International Conference on Machine Learning, pages 1319– 1327. ACM, 2013. find
  6. G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504 – 507, 2006. find
  7. G. E. Hinton, S. Osindero, and Y. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006. find
  8. K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In Proceedings of the International Conference on Computer Vision (ICCV’09). IEEE, 2009. find
  9. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. find
  10. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pages 1106–1114, 2012.
  11. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541–551, 1989. find
  12. Y. Lin, F. Lv, S. Zhu, M. Yang, T. Cour, K. Yu, L. Cao, Z. Li, M.-H. Tsai, X. Zhou, T. Huang, and T. Zhang. Imagenet classification: fast descriptor coding and large-scale svm training. Large scale visual recognition challenge, 2010. find
  13. A. Livnat, C. Papadimitriou, N. Pippenger, and M. W. Feldman. Sex, mixability, and modularity. Proceedings of the National Academy of Sciences, 107(4):1452–1457, 2010. find
  14. V. Mnih. CUDAMat: a CUDA-based matrix class for Python. Technical Report UTML TR 2009-004, Department of Computer Science, University of Toronto, November 2009. find
  15. A. Mohamed, G. E. Dahl, and G. E. Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 2010. find
  16. R. M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag New York, Inc., 1996. find
  17. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. find
  18. S. J. Nowlan and G. E. Hinton. Simplifying neural networks by soft weight-sharing. Neural Computation, 4(4), 1992. find
  19. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely. The Kaldi Speech Recognition Toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. find
  20. IEEE Signal Processing Society, 2011. find
  21. R. Salakhutdinov and G. Hinton. Deep Boltzmann machines. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 5, pages 448–455, 2009. find
  22. R. Salakhutdinov and A. Mnih. Bayesian probabilistic matrix factorization using Markov chain Monte Carlo. In Proceedings of the 25th International Conference on Machine Learning. ACM, 2008. find
  23. J. Sanchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, pages 1665–1672, 2011. find
  24. P. Sermanet, S. Chintala, and Y. LeCun. Convolutional neural networks applied to house numbers digit classification. In International Conference on Pattern Recognition (ICPR 2012), 2012. find
  25. P. Simard, D. Steinkraus, and J. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, volume 2, pages 958–962, 2003. find
  26. J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25, pages 2960–2968, 2012. find
  27. N. Srebro and A. Shraibman. Rank, trace-norm and max-norm. In Proceedings of the 18th annual conference on Learning Theory, COLT’05, pages 545–560. Springer-Verlag, 2005. find
  28. N. Srivastava. Improving Neural Networks with Dropout. Master’s thesis, University of Toronto, January 2013. find
  29. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B. Methodological, 58(1):267–288, 1996. find
  30. A. N. Tikhonov. On the stability of inverse problems. Doklady Akademii Nauk SSSR, 39(5): 195–198, 1943. find
  31. L. van der Maaten, M. Chen, S. Tyree, and K. Q. Weinberger. Learning with marginalized corrupted features. In Proceedings of the 30th International Conference on Machine Learning, pages 410–418. ACM, 2013. find
  32. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pages 1096–1103. ACM, 2008. find
  33. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. In Proceedings of the 27th International Conference on Machine Learning, pages 3371–3408. ACM, 2010. find
  34. S. Wager, S. Wang, and P. Liang. Dropout training as adaptive regularization. In Advances in Neural Information Processing Systems 26, pages 351–359, 2013. find
  35. S. Wang and C. D. Manning. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning, pages 118–126. ACM, 2013. find
  36. H. Y. Xiong, Y. Barash, and B. J. Frey. Bayesian prediction of tissue-regulated splicing using RNA sequence and cellular context. Bioinformatics, 27(18):2554–2562, 2011. find
  37. M. D. Zeiler and R. Fergus. Stochastic pooling for regularization of deep convolutional neural networks. CoRR, abs/1301.3557, 2013. — [arXiv:1301.3557](https://arxiv.org/abs/1301.3557) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srivastava-2014-dropout-simple-way-prevent/