Elamur Research⌕
☰
home / papers / 2016 / Recurrent Highway Networks

Recurrent Highway Networks

Paper · 2016 · Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik et al.

Summary

Recurrent networks usually update their hidden state with a single nonlinear step per time step. Making this step-to-step transition deeper should add modeling power, but such models have very long gradient paths and are hard to train, even when built from LSTM cells. Zilly et al. study this problem using Geršgorin's circle theorem. The theorem bounds where the eigenvalues of the recurrent Jacobian can lie. This shows how weight initialization and gating affect vanishing and exploding gradients, and why LSTM-style gates help keep training stable. From this analysis the authors propose Recurrent Highway Networks (RHNs). An RHN stacks several gated Highway layers inside each recurrent transition, so the recurrence depth can exceed one while training stays manageable. They test RHNs on language modeling with Penn Treebank (word level) and text8 and enwik8 (character level). On these benchmarks, deeper transitions give clear gains without adding parameters. RHNs matter because they show that depth within each time step is a practical and efficient way to make recurrent models stronger.

Key points

  • On Penn Treebank, increasing recurrence depth from 1 to 10 with a fixed parameter budget lowers word-level test perplexity from 90.6 to 65.4.
  • A large RHN with recurrence depth 10 and 1500 units reaches 1.27 BPC on the test sets of both text8 (about 45M parameters) and enwik8 (about 46M parameters). This beat earlier published results that used fewer than 1.5 BPC and no dynamic evaluation.
  • The Geršgorin analysis shows that large off-diagonal recurrent weights widen the possible spread of eigenvalues and risk exploding gradients. Identity initialization (Le et al., 2015) helps at first but can still drift during training.
  • On JSB Chorales, gate inspection shows that a trained depth-6 RHN uses all of its layers. Lesioning experiments show the first transition layer contributes most to performance, but removing any layer hurts noticeably.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ETH ZurichZurich · Switzerland84 works · 9,181 citations🎓Università della Svizzera italianaLugano · Switzerland10 works · 8,004 citations🔬Dalle Molle Institute for Artificial Intelligence ResearchLugano · Switzerland10 works · 8,825 citations🎓University of Applied Sciences and Arts of Southern SwitzerlandManno · Switzerland8 works · 7,374 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik and Jurgen Schmidhuber. 2016. Recurrent Highway Networks. arXiv:1607.03474.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1607.03474 · arXiv 1607.03474

@misc{zilly2016recurrent,
  title = {{Recurrent Highway Networks}},
  author = {Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik and Jurgen Schmidhuber},
  year = {2016},
  doi = {10.48550/arxiv.1607.03474},
  eprint = {1607.03474},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1607.03474}
}

References

53 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, Yoshua and LeCun, Yann. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1–41, 2007. find
  2. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994. find
  3. Bianchini, Monica and Scarselli, Franco. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Networks, 2014. find
  4. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. ArXiv e-prints, June 2012. find
  5. Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  6. Chung, J., Ahn, S., and Bengio, Y. Hierarchical Multiscale Recurrent Neural Networks. ArXiv e-prints, September 2016.
  7. Chung, Junyoung, Gulcehre, Caglar, Cho, Kyunghyun, and Bengio, Yoshua. Gated feedback recurrent neural networks. In International Conference on Machine Learning, 2015. find
  8. Cooijmans, T., Ballas, N., Laurent, C., Gülçehre, Ç., and Courville, A. Recurrent Batch Normalization. ArXiv e-prints, March 2016. find
  9. Gal, Yarin. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287, 2015.
  10. Gers, Felix A., Schmidhuber, Jürgen, and Cummins, Fred. Learning to forget: Continual prediction with lstm. Neural Computation, 12(10):2451–2471, 2016/02/18 2000. find
  11. Geršgorin, S. Über die Abgrenzung der Eigenwerte einer Matrix. Bulletin de l’Acadèmie des Sciences de l’URSS. Classe des sciences mathèmatiques, no. 6:749–754, 1931. find
  12. Graves, A. Generating sequences with recurrent neural networks. ArXiv e-prints, August 2013. find
  13. Graves, A. Adaptive Computation Time for Recurrent Neural Networks. ArXiv e-prints, March 2016. find
  14. Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R, and Schmidhuber, J. LSTM: A Search Space Odyssey. arXiv preprint arXiv:1503.04069, 2015.
  15. Greff, Klaus, Srivastava, Rupesh K, and Schmidhuber, Jürgen. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016. — [arXiv:1612.07771](https://arxiv.org/abs/1612.07771) find
  16. Ha, D., Dai, A., and Le, Q. V. HyperNetworks. ArXiv e-prints, September 2016. find
  17. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
  18. Hochreiter, S. Untersuchungen zu dynamischen neuronalen Netzen. Master’s thesis, Institut f. Informatik, Technische Univ. Munich, 1991. find
  19. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. find
  20. Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In Kremer, S. C. and Kolen, J. F. (eds.), A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001. find
  21. Hutter, M. The human knowledge compression contest. http://prize.hutter1.net/, 2012. ↗
  22. Inan, H., Khosravi, K., and Socher, R. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. ArXiv e-prints, November 2016.
  23. Inan, Hakan and Khosravi, Khashayar. Improved learning through augmenting the loss, 2016. find
  24. Jozefowicz, Rafal, Zaremba, Wojciech, and Sutskever, Ilya. An empirical exploration of recurrent network architectures. 2015. find
  25. Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  26. Kalchbrenner, Nal, Danihelka, Ivo, and Graves, Alex. Grid long short-term memory. CoRR, abs/1507.01526, 2015. URL http://arxiv.org/abs/1507.01526. ↗
  27. Kim, Yoon, Jernite, Yacine, Sontag, David, and Rush, Alexander M. Character-aware neural language models. arXiv preprint arXiv:1508.06615, 2015.
  28. Krause, B., Lu, L., Murray, I., and Renals, S. Multiplicative LSTM for sequence modelling. ArXiv e-prints, September 2016. find
  29. Le, Q. V., Jaitly, N., and Hinton, G. E. A Simple Way to Initialize Recurrent Networks of Rectified Linear Units. ArXiv e-prints, April 2015. find
  30. Lei Ba, J., Kiros, J. R., and Hinton, G. E. Layer Normalization. ArXiv e-prints, July 2016.
  31. Linnainmaa, S. The representation of the cumulative rounding error of an algorithm as a taylor expansion of the local rounding errors. Master’s thesis, Univ. Helsinki, 1970. find
  32. Linnainmaa, Seppo. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2):146–160, 1976. ISSN 1572-9125. find
  33. Marcus, Mitchell P., Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):313–330, June 1993. ISSN 0891-2017.
  34. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. ArXiv e-prints, September 2016.
  35. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. SLT, 12:234–239, 2012. find
  36. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In Interspeech, volume 2, pp. 3, 2010. find
  37. Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. How to construct deep recurrent neural networks. ArXiv e-prints, December 2013.
  38. Press, O. and Wolf, L. Using the Output Embedding to Improve Language Models. ArXiv e-prints, August 2016.
  39. Robinson, A. J. and Fallside, F. The utility driven dynamic error propagation network. Technical Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department, 1987. find
  40. Schmidhuber, Jürgen. Reinforcement learning in markovian and non-markovian environments. In Advances in Neural Information Processing Systems 3. Morgan-Kaufmann, 1991. find
  41. Schmidhuber, Jürgen. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242, 1992. find
  42. Schmidhuber, Jürgen. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. find
  43. Srivastava, Rupesh K, Greff, Klaus, and Schmidhuber, Juergen. Training very deep networks. In Advances in Neural Information Processing Systems 28, pp. 2368–2376. Curran Associates, Inc., 2015a. find
  44. Srivastava, Rupesh Kumar, Steunebrink, Bas R, and Schmidhuber, Jürgen. First experiments with powerplay. Neural Networks, 41:130–136, 2013. find
  45. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmidhuber, Jürgen. Highway networks. arXiv preprint arXiv:1505.00387, 2015b.
  46. Werbos, Paul J. System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31 – September 4, 1981, chapter Applications of advances in nonlinear sensitivity analysis, pp. 762–770. find
  47. Springer Berlin Heidelberg, 1982. find
  48. Werbos, Paul J. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1(4): 339–356, 1988. find
  49. Williams, R. J. Complexity of exact gradient computation algorithms for recurrent neural networks. Technical Report NU-CCS-89-27, Boston: Northeastern University, College of Computer Science, 1989. find
  50. Wu, Y., Zhang, S., Zhang, Y., Bengio, Y., and Salakhutdinov, R. On Multiplicative Integration with Recurrent Neural Networks. ArXiv e-prints, June 2016. find
  51. Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent Neural Network Regularization. ArXiv e-prints, September 2014.
  52. Zhang, Yu, Chen, Guoguo, Yu, Dong, Yao, Kaisheng, Khudanpur, Sanjeev, and Glass, James. Highway long short-term memory RNNS for distant speech recognition. In 2016 IEEE, ICASSP, 2016. find
  53. Zoph, Barret and Le, Quoc V. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016. 8. Supplementary Material 8.1. Details of Experimental Setups The following paragraphs describe the precise experimental settings used to obtain results in this paper. The sour…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zilly-2016-recurrent-highway-networks/