Elamur Research
☰
home / papers / 2015 / Visualizing and Understanding Recurrent Networks

Visualizing and Understanding Recurrent Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1506.02078

Summary

LSTM networks perform very well on sequence tasks, but it has been unclear what they learn and why they beat simpler models. Earlier ablation studies judged architectures only by overall test perplexity. Karpathy, Johnson and Fei-Fei use character-level language modeling as a transparent setting for probing this question. They train vanilla RNNs, LSTMs and GRUs (1–3 layers, 64–512 units) on two contrasting corpora. One is Tolstoy's War and Peace, which is nearly plain English prose. The other is the Linux Kernel source code, which is highly structured. Beyond comparing test cross-entropy, they inspect individual memory cells, compare LSTM predictions against finite-context n-gram models, and dissect the remaining mistakes with a sequence of oracles. Some cells reliably track long-range properties such as position within a line, open quotes and bracket depth. Most of the LSTM's advantage over n-grams comes from characters that depend on distant structure. The study gives inspectable evidence of how recurrent memory works on real data and identifies error types worth targeting.

Key points

  • Two corpora were chosen to represent both ends of the structure spectrum:
    • War and Peace (WP) contains 3,258,246 characters.
    • Linux Kernel (LK) is a 6,206,996-character file made from randomly shuffled, concatenated header and source files.
    • WP is split 80/10/10 into train/val/test.
  • Penn Treebank and the Hutter Prize Wikipedia data were set aside because they mix ordinary language with markup.
  • On test cross-entropy, LSTM and GRU clearly beat the vanilla RNN.
    • The best WP result is about 1.077 (3-layer GRU, 512 units); a 3-layer LSTM reaches 1.082.
    • The best LK result is 0.829 (3-layer GRU); a 2-layer LSTM reaches 0.840.
  • Training used the following settings:
    • RMSProp with learning rate 2×10⁻³ and batch size 100.
    • Unrolling over 100 time steps and up to 50 epochs.
    • Uniform initialization in [−0.08, 0.08], early stopping, and per-model tuned dropout.
  • Interpretable LSTM cells emerge without supervision. They count line length and respond to quotes and brackets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 106 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.02078 · arXiv 1506.02078

@misc{karpathy2015visualizing,
  title = {{Visualizing and Understanding Recurrent Networks}},
  author = {Karpathy, Andrej and Johnson, Justin and Fei-Fei, Li},
  year = {2015},
  doi = {10.48550/arxiv.1506.02078},
  eprint = {1506.02078},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.02078}
}

References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  2. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166, 1994. find
  3. Chen, Stanley F and Goodman, Joshua. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–393, 1999. find
  4. Cho, Kyunghyun, van Merriënboer, Bart, Bahdanau, Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014. find our card
  5. Chung, Junyoung, Gulcehre, Caglar, Cho, KyungHyun, and Bengio, Yoshua. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. find
  6. Dai, Andrew M and Le, Quoc V. arXiv:1511.01432, 2015. — [arXiv:1511.01432](https://arxiv.org/abs/1511.01432) find
  7. Dauphin, Yann N, de Vries, Harm, Chung, Junyoung, and Bengio, Yoshua. Rmsprop and equi-librated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390, 2015. — [arXiv:1502.04390](https://arxiv.org/abs/1502.04390) find
  8. Donahue, Jeff, Hendricks, Lisa Anne, Guadarrama, Sergio, Rohrbach, Marcus, Venugopalan, Subhashini, Saenko, Kate, and Darrell, Trevor. Long-term recurrent convolutional networks for visual recognition and description. CVPR, 2015. find our card
  9. Graves, Alex. Generating sequences with recurrent neural networks. arXiv:1308.0850, 2013. find
  10. Graves, Alex, Mohamed, A-R, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. find
  11. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. arXiv:1410.5401, 2014. find
  12. Greff, Klaus, Srivastava, Rupesh Kumar, Koutnı́k, Jan, Steunebrink, Bas R., and Schmidhuber, Jürgen. LSTM: A search space odyssey. CoRR, abs/1503.04069, 2015. URL http://arxiv. org/abs/1503.04069. find
  13. Heafield, Kenneth, Pouzyrevsky, Ivan, Clark, Jonathan H., and Koehn, Philipp. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pp. 690–696, Sofia, Bulgaria, August 2013. URL http://kheafield.com/profes… ↗ our card
  14. Hermans, Michiel and Schrauwen, Benjamin. Training and analysing deep recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 190–198, 2013. find
  15. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  16. Hoiem, Derek, Chodpathumwan, Yodsawalai, and Dai, Qieyun. Diagnosing error in object detectors. In Computer Vision–ECCV 2012, pp. 340–353. find
  17. Huang, Xuedong, Acero, Alex, Hon, Hsiao-Wuen, and Foreword By-Reddy, Raj. Spoken language processing: A guide to theory, algorithm, and system development. Prentice Hall PTR, 2001. find
  18. Hutter, Marcus. The human knowledge compression contest. 2012. find
  19. Jelinek, Frederick, Merialdo, Bernard, Roukos, Salim, and Strauss, Martin. A dynamic language model for speech recognition. In HLT, volume 91, pp. 293–295, 1991. find
  20. Semi-supervised sequence learning. arXiv preprint arXiv preprint Neural turing machines. arXiv preprint Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. CoRR, abs/1503.01007, 2015. URL http://arxiv.org/abs/1503.01007. ↗
  21. Jozefowicz, Rafal, Zaremba, Wojciech, and Sutskever, Ilya. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 2342–2350, 2015. find
  22. Karpathy, Andrej and Fei-Fei, Li. Deep visual-semantic alignments for generating image descriptions. CVPR, 2015. find our card
  23. Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993. find our card
  24. Mikolov, Tomáš. Statistical language models based on neural networks. Presentation at Google, Mountain View, 2nd April, 2012. find
  25. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pp. 1045–1048, 20… find
  26. Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2012. find
  27. Pascanu, Razvan, Gülçehre, Çaglar, Cho, Kyunghyun, and Bengio, Yoshua. How to construct deep recurrent neural networks. CoRR, abs/1312.6026, 2013. URL http://arxiv.org/abs/1312.6026. ↗
  28. Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J. Learning internal representations by error propagation. Technical report, DTIC Document, 1985. find
  29. Schmidhuber, J. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. doi: 10.1016/j.neunet.2014.09.003. Published online 2014; based on TR arXiv:1404.7828 [cs.NE]. — [doi:10.1016/j.neunet.2014.09.003](https://doi.org/10.1016/j.neunet.2014.09.003) find
  30. Sutskever, Ilya, Martens, James, and Hinton, Geoffrey E. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017–1024, 2011. find
  31. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014. find
  32. Van der Maaten, Laurens and Hinton, Geoffrey. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2579-2605):85, 2008. find
  33. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. CVPR, 2015. find our card
  34. Werbos, Paul J. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1(4):339–356, 1988. find
  35. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. CoRR, abs/1410.3916, 2014. URL http://arxiv.org/abs/1410.3916. ↗