Elamur Research⌕
☰
home / papers / 2016 / Hierarchical Multiscale Recurrent Neural Networks

Hierarchical Multiscale Recurrent Neural Networks

Paper · 2016 · Junyoung Chung, Sungjin Ahn and Yoshua Bengio

Summary

Recurrent networks have long struggled to learn representations that are both temporal and hierarchical. Earlier multiscale RNNs either fixed each layer's update rate in advance or needed explicit segment boundaries, such as word tokenization. Chung, Ahn and Bengio propose the hierarchical multiscale RNN (HM-RNN), which learns its own layer-wise timescales from raw sequences. Each layer has a binary boundary detector that fires when the layer finishes processing a segment at its level of abstraction. Based on these boundary signals, every layer chooses one of three operations at each step:

  • UPDATE: a sparse, LSTM-like state update.
  • COPY: carries the previous state over unchanged, so no information is lost.
  • FLUSH: sends a summary of the finished segment to the layer above and resets the state.

The discrete boundary variables are trained with the straight-through estimator, with a slope annealing trick added. On character-level language modelling, the model reaches state-of-the-art results on Text8 and near state-of-the-art results on Penn Treebank and Hutter Prize Wikipedia. It also beats a standard RNN on handwriting generation (IAM-OnDB). The boundaries it learns closely match the real structure of the data. This matters because it lets an RNN find a usable hierarchy without labelled segments.

Key points

  • The paper presents HM-RNN as the first RNN to learn latent hierarchical structure without any explicit boundary information. Lower layers tend to settle on fine timescales and upper layers on coarse ones.
  • The three operations are UPDATE, COPY and FLUSH. Unlike the leaky integration in LSTM or GRU, COPY keeps the full state intact, which helps with long-range dependencies and saves computation.
  • The authors contrast HM-RNN with CW-RNN, which uses fixed clock rates where module i updates every 2^(i−1) steps. They also contrast it with biscale RNNs, which use soft gating. HM-RNN instead learns hard, data-driven timescales.
  • Training uses the straight-through estimator for the binary detectors, and the paper introduces the slope annealing trick to improve it.
  • Evaluation covers Text8 (state-of-the-art), Penn Treebank and Hutter Prize Wikipedia (comparable to state-of-the-art), and IAM-OnDB handwriting (better than a standard RNN). The exact BPC scores are in the results section, which is missing from the provided extract.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

Cite

Cite as Junyoung Chung, Sungjin Ahn and Yoshua Bengio. 2016. Hierarchical Multiscale Recurrent Neural Networks. arXiv:1609.01704.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1609.01704 · arXiv 1609.01704

@misc{chung2016hierarchical,
  title = {{Hierarchical Multiscale Recurrent Neural Networks}},
  author = {Junyoung Chung, Sungjin Ahn and Yoshua Bengio},
  year = {2016},
  doi = {10.48550/arxiv.1609.01704},
  eprint = {1609.01704},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1609.01704}
}

References

59 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  2. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Yoshua Bengio, et al. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4945–4949. find
  3. Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R in Machine Learning, 2(1):1–127, 2009. find
  4. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. find
  5. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4960–4964. find
  6. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), October 2014.
  7. Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015. find
  8. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. A character-level decoder without explicit segmentation for neural machine translation. Association for Computational Linguistics (ACL), 2016.
  9. Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. Recurrent batch normalization. arXiv preprint arXiv:1603.09025, 2016. — [arXiv:1603.09025](https://arxiv.org/abs/1603.09025) find
  10. Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.
  11. Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies. In Advances in Neural Information Processing Systems, pp. 493–499. Citeseer, 1995. find
  12. Santiago Fernández, Alex Graves, and Jürgen Schmidhuber. Sequence labelling in structured domains with hierarchical recurrent neural networks. In Proceedings of the 20th international joint conference on Artifical intelligence, pp. 774–779. Morgan Kaufmann Publishers Inc., 2007. find
  13. Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  14. David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016. find
  15. G. Hinton. Neural networks for machine learning. Coursera, video lectures, 2012. find
  16. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  17. Judy Hoffman, Eric Tzeng, Jeff Donahue, Yangqing Jia, Kate Saenko, and Trevor Darrell. One-shot adaptation of supervised deep convolutional models. arXiv preprint arXiv:1312.6204, 2013. find
  18. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382, 2016.
  19. Marcus Hutter. The human knowledge compression contest. 2012. URL http://prize.hutter1.net/. ↗
  20. Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. arXiv:1507.01526, 2015. find
  21. Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014.
  22. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. arXiv preprint arXiv:1508.06615, 2015.
  23. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  24. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  25. Lingpeng Kong, Chris Dyer, and Noah A Smith. Segmental recurrent neural networks. arXiv preprint arXiv:1511.06018, 2015. — [arXiv:1511.06018](https://arxiv.org/abs/1511.06018) find
  26. Jan Koutník, Klaus Greff, Faustino Gomez, and Jürgen Schmidhuber. A clockwork rnn. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014), 2014. find
  27. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097–1105, 2012.
  28. David Krueger and Roland Memisevic. arXiv:1511.08400, 2015. — [arXiv:1511.08400](https://arxiv.org/abs/1511.08400) find
  29. Regularizing rnns by stabilizing activations. David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, Aaron Courville, et al. Zoneout: Regularizing rnns by randomly preserving hidden activations. arXiv preprint ar… find
  30. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015. find
  31. Tsungnan Lin, Bill G Horne, Peter Tino, and C Lee Giles. Learning long-term dependencies in narx recurrent neural networks. IEEE Transactions on Neural Networks, 7(6):1329–1338, 1996. find
  32. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. Character-based neural machine translation. arXiv preprint arXiv:1511.04586, 2015.
  33. Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten text on a whiteboard. In Eighth International Conference on Document Analysis and Recognition (ICDAR’05), pp. 956–961. IEEE, 2005. find
  34. Matthew V Mahoney. Adaptive weighing of context models for lossless data compression. 2005. find
  35. Matthew V Mahoney. Large text compression benchmark. URL: http://www.mattmahoney. net/text/text. html, 2009. ↗
  36. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993.
  37. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, volume 2, pp. 3, 2010. find
  38. Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and J Cernocky. Subword language modeling with neural networks. Preprint, 2012. URL http://www.fit.vutbr.cz/~imikolov/rnnlm/char.pdf. ↗
  39. Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 1791–1799, 2014. find
  40. Michael C Mozer. Induction of multiscale temporal structure. Advances in neural information processing systems, pp. 275–275, 1993. find
  41. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814, 2010. find
  42. Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? arXiv preprint arXiv:1301.5650, 2013. — [arXiv:1301.5650](https://arxiv.org/abs/1301.5650) find
  43. Grid long short-term memory. arXiv preprint arXiv preprint Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2012. find
  44. Kamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085, 2016a. — [arXiv:1607.03085](https://arxiv.org/abs/1607.03085) find
  45. Kamil M Rocki. Surprisal-driven feedback in recurrent networks. arXiv preprint arXiv:1608.06027, 2016b. — [arXiv:1608.06027](https://arxiv.org/abs/1608.06027) find
  46. Jürgen Schmidhuber. Neural sequence chunkers. 1991. find
  47. Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242, 1992. find
  48. Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. find
  49. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 5… find
  50. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1): 1929–1958, 2014.
  51. Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML’11), pp. 1017–1024, 2011. find
  52. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014.
  53. The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688,…
  54. Alexander Vezhnevets, Volodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray Kavukcuoglu, et al. Strategic attentive writer for learning macro-actions. arXiv preprint arXiv:1606.04695, 2016. find
  55. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156– 3164, 2015.
  56. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  57. Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan Salakhutdinov. On multiplicative integration with recurrent neural networks. arXiv preprint arXiv:1606.06630, 2016. find
  58. Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, and Yoshua Bengio. Architectural complexity measures of recurrent neural networks. arXiv preprint arXiv:1602.08210, 2016. find
  59. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. arXiv preprint arXiv:1607.03474, 2016.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chung-2016-hierarchical-multiscale-recurrent-neural-networks/