Elamur Research⌕
☰
home / papers / 2013 / How to Construct Deep Recurrent Neural Networks

How to Construct Deep Recurrent Neural Networks

Paper · 2013 · Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho et al.

Summary

In feedforward networks, depth means stacking nonlinear layers between input and output. In recurrent networks the idea is ambiguous: an RNN unrolled over time is already deep, yet each single step uses only an affine map followed by an elementwise nonlinearity. Pascanu, Gulcehre, Cho and Bengio examine one step of an RNN and identify three shallow components that could be deepened: the input-to-hidden mapping, the hidden-to-hidden transition and the hidden-to-output mapping.

They introduce two models. The deep transition RNN (DT-RNN) places an MLP between consecutive hidden states. The deep output, deep transition RNN (DOT-RNN) also adds intermediate layers before the output. Variants with shortcut connections, DT(S) and DOT(S), shorten gradient paths through time. These designs complement the older approach of stacking recurrent layers (sRNN).

Experiments covered polyphonic music prediction and word- and character-level language modeling. Every deep variant beat the shallow baseline, and DOT(S)-RNN gave the best word-level language modeling result reported at the time. The work offers a systematic way to think about where depth can be added to recurrent architectures.

Key points

  • The evaluated models are DT-RNN, DT(S)-RNN, DOT-RNN and DOT(S)-RNN. Their baselines are a conventional single-layer RNN and an RNN built from multiple recurrent layers, as proposed by Schmidhuber (1992) and by El Hihi and Bengio (1996).
  • On word-level language modeling, both DOT(S)-RNN and sRNN outperformed earlier bests from a 1000-unit LSTM (Graves, 2013) and a larger shallow RNN (Mikolov et al., 2011). Those earlier systems used dynamic evaluation, while the new models did not.
  • On character-level modeling, the deep RNNs beat the 1.41 reported for a shallow Hessian-free-trained RNN. They still trailed mRNN and adaptive weight noise.
  • For polyphonic music prediction, a different deep variant won on each dataset. The shallow RNN never outperformed the deep ones.
  • DOT(S)-RNN and sRNN were hard to train. The authors used shortcut connections and pretraining from a conventional RNN or DT(S)-RNN. A preliminary test combining non-saturating activations with dropout clearly improved the music results. The models were implemented in Theano.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🎓Aalto UniversityEspoo · Finland15 works · 950 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho and Yoshua Bengio. 2013. How to Construct Deep Recurrent Neural Networks. arXiv:1312.6026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1312.6026 · arXiv 1312.6026

@misc{pascanu2013how,
  title = {{How to Construct Deep Recurrent Neural Networks}},
  author = {Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho and Yoshua Bengio},
  year = {2013},
  doi = {10.48550/arxiv.1312.6026},
  eprint = {1312.6026},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1312.6026}
}

References

49 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements.
  2. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. find
  3. Bayer, J., Osendorfer, C., Korhammer, D., Chen, N., Urban, S., and van der Smagt, P. (2013). On fast dropout and its applicability to recurrent networks. arXiv:1311.0701 [cs.NE]. — [arXiv:1311.0701](https://arxiv.org/abs/1311.0701) find
  4. Bengio, Y. (2009). Learning deep architectures for AI. Found. Trends Mach. Learn., 2(1), 1–127. find
  5. Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. find
  6. Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. (2013). Better mixing via deep representations. In ICML’13. find
  7. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation. find
  8. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. In ICML’2012. find
  9. Chen, J. and Deng, L. (2013). A new method for learning deep recurrent neural networks. arXiv:1311.6091 [cs.LG]. — [arXiv:1311.6091](https://arxiv.org/abs/1311.6091) find
  10. Delalleau, O. and Bengio, Y. (2011). Shallow vs. deep sum-product networks. In NIPS. find
  11. El Hihi, S. and Bengio, Y. (1996). Hierarchical recurrent neural networks for long-term dependencies. In NIPS 8. MIT Press. find
  12. Glorot, X., Bordes, A., and Bengio, Y. (2011a). Deep sparse rectifier neural networks. In AISTATS. find
  13. Glorot, X., Bordes, A., and Bengio, Y. (2011b). Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML’2011. find
  14. Goodfellow, I., Le, Q., Saxe, A., and Ng, A. (2009). Measuring invariances in deep networks. In NIPS’09, pages 646–654. find
  15. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout networks. In ICML’2013. find
  16. Graves, A. (2011). Practical variational inference for neural networks. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348–2356. find
  17. Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs.NE]. find
  18. Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., and Schmidhuber, J. (2009). A novel connectionist system for improved unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  19. Graves, A., Mohamed, A., and Hinton, G. (2013). Speech recognition with deep recurrent neural networks. ICASSP. find
  20. Gulcehre, C., Cho, K., Pascanu, R., and Bengio, Y. (2013). Learned-norm pooling for deep feedforward and recurrent neural networks. arXiv:1311.1780 [cs.NE]. find
  21. Hermans, M. and Schrauwen, B. (2013). Training and analysing deep recurrent neural networks. In Advances in Neural Information Processing Systems 26, pages 190–198. find
  22. Hinton, G., Deng, L., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. (2012a). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6), 82–97. find
  23. Hinton, G. E. and Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. find
  24. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012b). Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580. find
  25. Hornik, K., Stinchcombe, M., and White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2, 359–366. find
  26. Jaeger, H. (2007). Discovering multiscale dynamical features with hierarchical echo state networks. Technical report, Jacobs University. find
  27. Ko, J. and Dieter, F. (2009). Gp-bayesfilters: Bayesian filtering using gaussian process prediction and observation models. Autonomous Robots. find
  28. Larochelle, H. and Murray, I. (2011). The Neural Autoregressive Distribution Estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AIS-TATS’2011), volume 15 of JMLR: W&CP. find
  29. Le Roux, N. and Bengio, Y. (2010). Deep belief networks are compact universal approximators. Neural Computation, 22(8), 2192–2207. find
  30. Marcus, M. P., Marcinkiewicz, M. A., and Santorini, B. (1993). Building a large annotated corpus of english: The Penn Treebank. Computational Linguistics, 19(2), 313–330.
  31. Martens, J. (2010). Deep learning via Hessian-free optimization. In L. Bottou and M. Littman, editors, Proceedings of the Twenty-seventh International Conference on Machine Learning (ICML-10), pages 735–742. ACM. find
  32. Martens, J. and Sutskever, I. (2011). Learning recurrent neural networks with Hessian-free optimization. In Proc. ICML’2011. ACM. find
  33. Mikolov, T. (2012). Statistical Language Models based on Neural Networks. Ph.D. thesis, Brno University of Technology. find
  34. Mikolov, T., Karafiát, M., Burget, L., Cernocky, J., and Khudanpur, S. (2010). Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH 2010), volume 2010, pages 1045–1048. International Speech Com… find
  35. Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., and Khudanpur, S. (2011). Extensions of recurrent neural network language model. In Proc. 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP 2011). find
  36. Mikolov, T., Sutskever, I., Deoras, A., Le, H., Kombrink, S., and Cernocky, J. (2012a). Subword language modeling with neural networks. unpublished. find
  37. Mikolov, T., Sutskever, I., Deoras, A., Le, H.-S., Kombrink, S., and Cernocky, J. (2012b). Subword language modeling with neural networks. preprint (http://www.fit.vutbr.cz/ imikolov/rnnlm/char.pdf). find
  38. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013a). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119.
  39. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013b). Efficient estimation of word representations in vector space. In International Conference on Learning Representations: Workshops Track.
  40. Pascanu, R. and Bengio, Y. (2013). Revisiting natural gradient for deep networks. Technical report, arXiv:1301.3584. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) find
  41. Pascanu, R., Mikolov, T., and Bengio, Y. (2013a). On the difficulty of training recurrent neural networks. In ICML’2013. find
  42. Pascanu, R., Montufar, G., and Bengio, Y. (2013b). On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv:1312.6098[cs.LG]. — [arXiv:1312.6098](https://arxiv.org/abs/1312.6098) find
  43. Pinheiro, P. and Collobert, R. (2014). Recurrent convolutional neural networks for scene labeling. In Proceedings of The 31st International Conference on Machine Learning, pages 82–90. find
  44. Raiko, T., Valpola, H., and LeCun, Y. (2012). Deep learning made easier by linear transformations in perceptrons. In Proceedings of the Fifteenth Internation Conference on Artificial Intelligence and Statistics (AISTATS 2012), volume 22 of JMLR Workshop and Conference Proceedings, pages 924–932. JML… find
  45. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by backpropagating errors. Nature, 323, 533–536. find
  46. Schmidhuber, J. (1992). Learning complex, extended sequences using the principle of history compression. Neural Computation, (4), 234–242. find
  47. Sutskever, I., Martens, J., and Hinton, G. (2011). Generating text with recurrent neural networks. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pages 1017–1024, New York, NY, USA. ACM. find
  48. Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). On the importance of initialization and momentum in deep learning. In ICML. find
  49. Valpola, H. and Karhunen, J. (2002). An unsupervised ensemble learning method for nonlinear dynamic state-space models. Neural Comput., 14(11), 2647–2692. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pascanu-2013-construct-deep-recurrent-neural-networks/