Elamur Research
☰
home / papers / 2015 / A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1512.05287

Summary

Recurrent models such as LSTM and GRU overfit easily. Earlier studies concluded that dropout should only be applied to inputs and outputs, because noise on recurrent connections seemed to swamp the signal over long sequences. Gal and Ghahramani revisit this question using their earlier view of dropout as approximate variational inference in a Bayesian neural network. They treat the RNN weights as random variables and approximate the posterior with a mixture of two Gaussians, one of them centred at zero. The resulting training objective is equivalent to a new dropout scheme. One mask is sampled per sequence and reused at every time step, and it covers inputs, outputs and the recurrent weights. Word embeddings are also dropped, so whole word types disappear from a sentence. The authors test this on Penn Treebank language modelling and on sentiment regression with the Cornell film reviews corpus. The resulting Variational LSTM/GRU overfits less than naive-dropout and unregularised baselines. It reaches 73.4 test perplexity on Penn Treebank, which the authors report as the best single-model result at the time. The work gives a principled way to regularise recurrent layers instead of relying on trial and error.

Key points

  • Method: the dropout mask stays fixed across time steps and is also applied to recurrent connections. This differs from the earlier baseline by Zaremba et al., which samples new masks at each step and only drops non-recurrent connections. The closest related approach is a parallel variant by Moon et al., which drops elements of the LSTM cell state.
  • Penn Treebank: a single Variational LSTM reaches 73.4 test perplexity. An average of 10 Variational LSTMs gets 68.7, compared with 69.5 for Zaremba et al.'s ensemble of the same size and the same 68.7 their 38-model ensemble needed.
  • Sentiment data: the sentiment experiments use 5000 reviews from the corpus of Pang and Lee, cut into 200-word segments. The model has a 128-dimensional embedding, uses the Adam optimiser and averages 10 Monte Carlo dropout samples at test time. Here the Variational LSTM was the only model that did not overfit.
  • Ablations: strong dropout on both the embeddings and the recurrent layers is needed. Without embedding dropout, raising recurrent dropout actually increased overfitting.
  • Weight decay: weight decay, which corresponds to the prior over the weights, still matters with this dropout variant. Practitioners usually remove it when using naive dropout.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Treebanks

Cited by

235 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 216 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1512.05287 · arXiv 1512.05287

@misc{gal2015theoretically,
  title = {{A Theoretically Grounded Application of Dropout in Recurrent Neural Networks}},
  author = {Gal, Yarin and Ghahramani, Zoubin},
  year = {2015},
  doi = {10.48550/arxiv.1512.05287},
  eprint = {1512.05287},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1512.05287}
}

References

30 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM neural networks for language modeling. In INTERSPEECH, 2012. find
  2. Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In EMNLP, 2013. (a) Combinations of p E = 0, 0.5 with p U = 0, 0.5. (b) p U = 0, ..., 0.5 with fixed p E = 0. Figure 4: Test error for Variational LSTM with various settings on the sentiment analysis task. Different dropout… find
  3. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In NIPS, 2014. find
  4. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. find our card
  5. Geoffrey E others Hinton. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012. find
  6. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014. find
  7. Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? arXiv preprint arXiv:1301.5650, 2013. — [arXiv:1301.5650](https://arxiv.org/abs/1301.5650) find
  8. Justin Bayer, Christian Osendorfer, Daniela Korhammer, Nutan Chen, Sebastian Urban, and Patrick van der Smagt. On fast dropout and its applicability to recurrent networks. arXiv preprint arXiv:1311.0701, 2013. — [arXiv:1311.0701](https://arxiv.org/abs/1311.0701) find
  9. Vu Pham, Theodore Bluche, Christopher Kermorvant, and Jerome Louradour. Dropout improves recurrent neural networks for handwriting recognition. In ICFHR. IEEE, 2014. find
  10. Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Where to apply dropout in recurrent neural networks for handwriting recognition? In ICDAR. IEEE, 2015. find
  11. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014. find our card
  12. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In ICML, 2015. find
  13. Jose Miguel Hernandez-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In ICML, 2015. find
  14. Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv:1506.02158, 2015. — [arXiv:1506.02158](https://arxiv.org/abs/1506.02158) find
  15. Diederik Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In NIPS. Curran Associates, Inc., 2015. find
  16. Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling. Bayesian dark knowledge. In NIPS. Curran Associates, Inc., 2015. find
  17. Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. arXiv:1506.02142, 2015. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997. find
  19. Kyunghyun Cho et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In EMNLP, Doha, Qatar, October 2014. ACL. find
  20. Taesup Moon, Heeyoul Choi, Hoshik Lee, and Inchul Song. RnnDrop: A Novel Dropout for RNNs in ASR. In ASRU Workshop, December 2015. find
  21. David JC MacKay. A practical Bayesian framework for backpropagation networks. Neural computation, 4 (3):448–472, 1992. find
  22. Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995. find
  23. Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In COLT, pages 5–13. ACM, 1993. find
  24. David Barber and Christopher M Bishop. Ensemble learning in Bayesian neural networks. NATO ASI SERIES F COMPUTER AND SYSTEMS SCIENCES, 168:215–238, 1998. find
  25. Alex Graves. Practical variational inference for neural networks. In NIPS, 2011. find
  26. Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In ICASSP. IEEE, 2013. find
  27. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL. ACL, 2005. find
  28. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  29. James Bergstra et al. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Presentation. find
  30. fchollet. Keras. https://github.com/fchollet/keras, 2015. ↗