Elamur Research
☰
home / papers / 2014 / Recurrent Neural Network Regularization

Recurrent Neural Network Regularization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1409.2329

Summary

Dropout works well for regularizing feedforward networks but had largely failed for recurrent ones. Earlier work argued that noise injected into the recurrence builds up over time and disrupts learning. Without effective regularization, practitioners kept RNNs small to avoid overfitting. Zaremba, Sutskever and Vinyals propose a simple fix for multilayer LSTMs: apply dropout only to the vertical, non-recurrent connections (input to layer, layer to layer, and top layer to output). The hidden-to-hidden connections through time are left untouched. A signal is then corrupted only L+1 times, where L is the number of layers, no matter how many timesteps it travels. This leaves the LSTM's long-range memory intact. The authors test the method on word-level language modeling (Penn Tree Bank), acoustic modeling (Icelandic speech), English→French translation and image captioning. On PTB, it allows much larger networks and clearly lowers perplexity. The recipe became a standard baseline for regularizing LSTMs, and the code was released publicly (github.com/wojzaremba/lstm).

Key points

  • Method: dropout is applied only to non-recurrent connections, so memory carried through time is never dropped. Pham et al. (2013) independently found the same technique for handwriting recognition.
  • Penn Tree Bank (929k/73k/82k words, 10k vocabulary): the best non-regularized LSTM (2×200 units) reaches a test perplexity of 114.5. The medium regularized LSTM (2×650 units, 50% dropout) reaches 82.7, and the large one (2×1500 units, 65% dropout) reaches 78.4. An ensemble of 38 large models gets 68.7.
  • Training cost: on one NVIDIA K20 GPU, the medium model trains in about half a day and the large model in about a full day. Gradient norms are clipped at 5 and 10 respectively.
  • Speech (Icelandic Speech Dataset, 93k utterances): dropout lowers training frame accuracy from 71.6 to 69.4 but raises validation accuracy from 68.9 to 70.5, a sign of less overfitting.
  • English→French MT: test perplexity falls from 5.8 to 5.0 and BLEU rises from 25.9 to 29.03. This is still below the phrase-based LIUM system at 33.30.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

256 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 233 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1409.2329 · arXiv 1409.2329

@misc{zaremba2014recurrent,
  title = {{Recurrent Neural Network Regularization}},
  author = {Zaremba, Wojciech and Sutskever, Ilya and Vinyals, Oriol},
  year = {2014},
  doi = {10.48550/arxiv.1409.2329},
  eprint = {1409.2329},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1409.2329}
}

References

33 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bayer, Justin, Osendorfer, Christian, Chen, Nutan, Urban, Sebastian, and van der Smagt, Patrick. On fast dropout and its applicability to recurrent networks. arXiv preprint arXiv:1311.0701, 2013. — [arXiv:1311.0701](https://arxiv.org/abs/1311.0701) find
  2. Bourlard, H. and Morgan, N. Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, 1993. find
  3. Cheng, Wei-Chen, Kok, Stanley, Pham, Hoai Vu, Chieu, Hai Leong, and Chai, Kian Ming A. Language modeling with sum-product networks. Cho, Kyunghyun, van Merrienboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder… find
  4. Chow, Y, Dunham, M, Kimball, O, Krasner, M, Kubala, G, Makhoul, J, Price, P, Roucos, S, and Schwartz, R. Byblos: The bbn continuous speech recognition system. In Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP’87., volume 12, pp. 89–92. IEEE, 1987. find
  5. Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. Fast and robust neural network joint models for statistical machine translation. In ACL, 2014. find
  6. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  7. Graves, Alex, Liwicki, Marcus, Fernández, Santiago, Bertolami, Roman, Bunke, Horst, and Schmidhuber, Jürgen. A novel connectionist system for unconstrained handwriting recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 31(5):855–868, 2009. find
  8. Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. find
  9. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  10. Jaeger, Herbert, Lukoševičius, Mantas, Popovici, Dan, and Siewert, Udo. Optimization and applications of echo state networks with leaky-integrator neurons. Neural Networks, 20(3):335–352, 2007. find
  11. Kalchbrenner, N. and Blunsom, P. Recurrent continuous translation models. In EMNLP, 2013. find
  12. Koutnı́k, Jan, Greff, Klaus, Gomez, Faustino, and Schmidhuber, Jürgen. A clockwork rnn. arXiv preprint arXiv:1402.3511, 2014. — [arXiv:1402.3511](https://arxiv.org/abs/1402.3511) find
  13. Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollár, Piotr, and Zitnick, C Lawrence. Microsoft coco: Common objects in context. arXiv preprint arXiv:1405.0312, 2014. find our card
  14. Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993. find our card
  15. Mikolov, Tomáš. Statistical language models based on neural networks. PhD thesis, Ph. D. thesis, Brno University of Technology, 2012. find
  16. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In SLT, pp. 234–239, 2012. find
  17. Test perplexity 8.47 7.99 7.5 Test BLEU score 23.5 24.3 24.4 Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH, pp. 1045–1048, 2010. find
  18. Mikolov, Tomas, Deoras, Anoop, Povey, Daniel, Burget, Lukas, and Cernocky, Jan. Strategies for training large scale neural network language models. In Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on, pp. 196–201. find
  19. Mikolov, Tomas, Le, Quoc V, and Sutskever, Ilya. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013. find
  20. Pachitariu, Marius and Sahani, Maneesh. Regularization and nonlinearities for neural language models: when are they needed? arXiv preprint arXiv:1301.5650, 2013. — [arXiv:1301.5650](https://arxiv.org/abs/1301.5650) find
  21. Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, and Bengio, Yoshua. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026, 2013. find
  22. Pham, Vu, Kermorvant, Christopher, and Louradour, Jérôme. Dropout improves recurrent neural networks for handwriting recognition. arXiv preprint arXiv:1312.4569, 2013. — [arXiv:1312.4569](https://arxiv.org/abs/1312.4569) find
  23. Robinson, Tony, Hochberg, Mike, and Renals, Steve. The use of recurrent neural networks in continuous speech recognition. In Automatic speech and speaker recognition, pp. 233–258. Springer, 1996. find
  24. Sak, H., Vinyals, O., Heigold, G., Senior, A., McDermott, E., Monga, R., and Mao, M. Sequence discriminative distributed training of long short-term memory recurrent neural networks. In Interspeech, 2014. find
  25. Schwenk, Holger. University le mans, 2014. http://www-lium.univ-lemans.fr/˜schwenk/cslm\_joint/paper. ↗
  26. Schwenk, Holger, Lambert, Patrik, Barrault, Loı̈c, Servan, Christophe, Afli, Haithem, Abdul-Rauf, Sadaf, and Shah, Kashif. Lium’s smt machine translation systems for wmt 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 464–469. Association for Computational Linguist… find
  27. Srivastava, Nitish. Improving neural networks with dropout. PhD thesis, University of Toronto, 2013. find
  28. Sundermeyer, Martin, Schlüter, Ralf, and Ney, Hermann. Lstm neural networks for language modeling. In INTERSPEECH, 2012. find
  29. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014. find
  30. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  31. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014. find our card
  32. Wan, Li, Zeiler, Matthew, Zhang, Sixin, Cun, Yann L, and Fergus, Rob. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1058–1066, 2013. find
  33. Wang, Sida and Manning, Christopher. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 118–126, 2013. find