Elamur Research⌕
☰
home / papers / 2015 / LSTM: A Search Space Odyssey

LSTM: A Search Space Odyssey

Paper · 2015 · Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnik et al.

Summary

Since its introduction in the mid-1990s, LSTM has accumulated many architectural modifications, but no one had systematically tested which of its parts actually matter. Greff et al. address this gap with a large controlled comparison. They take the widely used "vanilla LSTM" as a baseline. It has input, forget and output gates, peephole connections, an output activation function, and is trained with full BPTT. They then build eight variants, each changing exactly one component. All nine models are tested on three tasks: speech recognition (acoustic modeling), online handwriting recognition, and polyphonic music modeling. Hyperparameters for every model–dataset pair are tuned separately with random search. The fANOVA framework then estimates how much each hyperparameter contributes to performance. No variant reliably beat the vanilla LSTM. Removing the forget gate or the output activation function caused the largest drops. The work matters because it gives empirical backing to the standard design and practical advice for tuning recurrent models.

Key points

  • Scale: 5400 experimental runs, roughly 15 years of CPU time, which the authors describe as the largest study of its kind on LSTM.
  • Variants: eight single-change variants of vanilla LSTM, so the effect of each component can be isolated. The paper also places them in the history of LSTM, alongside GRU (Cho et al.) and Sak et al.'s projection layer.
  • Benchmarks: the references point to TIMIT for speech, IAM-OnDB for handwriting, and the chorales data of Allan & Williams for music. The extract omits the setup section, so these mappings should be checked against the full paper.
  • Most critical components: the forget gate, plus the activation function applied at the output.
  • Hyperparameters: fANOVA shows they interact very little, so each can be tuned more or less independently. The authors turn this into tuning guidelines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Università della Svizzera italianaLugano · Switzerland10 works · 8,004 citations🔬Dalle Molle Institute for Artificial Intelligence ResearchLugano · Switzerland10 works · 8,825 citations🎓University of Applied Sciences and Arts of Southern SwitzerlandManno · Switzerland8 works · 7,374 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnik, Bas R. Steunebrink and Jurgen Schmidhuber. 2015. LSTM: A Search Space Odyssey. arXiv:1503.04069 · IEEE Transactions on Neural Networks and Learning Systems ( Volume: 28, Issue: 10, Oct. 2017 ) Pages: 2222 - 2232.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/tnnls.2016.2582924 · arXiv 1503.04069

@misc{greff2015lstm,
  title = {{LSTM: A Search Space Odyssey}},
  author = {Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnik, Bas R. Steunebrink and Jurgen Schmidhuber},
  year = {2015},
  doi = {10.1109/tnnls.2016.2582924},
  eprint = {1503.04069},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/tnnls.2016.2582924}
}

References

44 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Masters Thesis, Technische Universität München, München, 1991. find
  2. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001. find
  3. A Graves, M Liwicki, S Fernandez, R Bertolami, H Bunke, and J Schmidhuber. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5), 2009. find
  4. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Dropout improves Recurrent Neural Networks for Handwriting Recognition. arXiv:1312.4569 [cs], November 2013. URL http://arxiv.org/abs/1312.4569. — [arXiv:1312.4569](https://arxiv.org/abs/1312.4569) ↗
  5. Patrick Doetsch, Michal Kozielski, and Hermann Ney. Fast and robust training of recurrent neural networks for offline handwriting recognition. In 14th International Conference on Frontiers in Handwriting Recognition, 2014. URL http://people.sabanciuniv.edu/berrin/cs581/Papers/icfhr2014/data/4334a279… ↗
  6. Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs], August 2013. URL http://arxiv.org/abs/1308.0850. ↗
  7. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent Neural Network Regularization. arXiv:1409.2329 [cs], September 2014. URL http://arxiv.org/abs/1409.2329. ↗
  8. Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the Rare Word Problem in Neural Machine Translation. arXiv preprint arXiv:1410.8206, 2014. URL http://arxiv.org/abs/1410.8206. ↗
  9. Hasim Sak, Andrew Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Proceedings of the Annual Conference of International Speech Communication Association (INTERSPEECH), 2014. URL http://193.6.4.39/ ∼ czap/letoltes/IS… ↗
  10. Yuchen Fan, Yao Qian, Fenglong Xie, and Frank K. Soong. TTS synthesis with bidirectional LSTM based recurrent neural networks. In Proc. Interspeech, 2014. find
  11. Søren Kaae Sønderby and Ole Winther. Protein Secondary Structure Prediction with Long Short Term Memory Networks. arXiv:1412.7828 [cs, q-bio], December 2014. URL http://arxiv.org/abs/1412.7828. arXiv: 1412.7828. — [arXiv:1412.7828](https://arxiv.org/abs/1412.7828) ↗
  12. E. Marchi, G. Ferroni, F. Eyben, L. Gabrielli, S. Squartini, and B. Schuller. Multi-resolution linear prediction based features for audio onset detection with bidirectional LSTM neural networks. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2164–216… — [doi:10.1109/icassp.2014.6853982](https://doi.org/10.1109/icassp.2014.6853982) find
  13. Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term Recurrent Convolutional Networks for Visual Recognition and Description. arXiv:1411.4389 [cs], November 2014. URL http://arxiv.org/abs/1411.4389. arXiv: 1411.4389… ↗
  14. Sepp Hochreiter and Jürgen Schmidhuber. Long Short Term Memory. Technical Report FKI-207-95, Technische Universität München, München, August 1995. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.51.3117. ↗
  15. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL http://www.bioinf.jku.at/publications/older/2604.pdf. ↗
  16. R. L. Anderson. Recent Advances in Finding Best Operating Conditions. Journal of the American Statistical Association, 48(264):789–798, December 1953. ISSN 0162-1459. doi: 10.2307/2281072. URL http://www.jstor.org/stable/2281072. — [doi:10.2307/2281072](https://doi.org/10.2307/2281072) ↗
  17. Francisco J. Solis and Roger J.-B. Wets. Minimization by Random Search Techniques. Mathematics of Operations Research, 6(1):19–30, February 1981. ISSN 0364-765X. doi: 10.1287/moor.6.1.19. URL http://pubsonline.informs.org/doi/abs/10.1287/moor.6.1.19. — [doi:10.1287/moor.6.1.19](https://doi.org/10.1287/moor.6.1.19) ↗
  18. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):281–305, 2012. URL http://dl.acm.org/citation.cfm?id=2188395. ↗
  19. Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. An Efficient Approach for Assessing Hyperparameter Importance. pages 754–762, 2014. URL http://jmlr.org/proceedings/papers/v32/hutter14.html. ↗
  20. Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5–6): 602–610, July 2005. ISSN 0893-6080. doi: 10.1016/j. neunet.2005.06.042. URL http://www.sciencedirect.com/science/article/pii/S08936080050012… ↗
  21. Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with LSTM. In Artificial Neural Networks, 1999. ICANN 99. Ninth International Conference on (Conf. Publ. No. 470), volume 2, pages 850–855, 1999. find
  22. Felix A. Gers and Jürgen Schmidhuber. Recurrent nets that time and count. In Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on, volume 3, pages 189–194. IEEE, 2000. ISBN 0769506194. find
  23. AJ Robinson and Frank Fallside. The utility driven dynamic error propagation network. University of Cambridge Department of Engineering, 1987. find
  24. R. J. Williams. Complexity of exact gradient computation algorithms for recurrent neural networks. Technical Report Technical Report NU-CCS-89-27, Boston: Northeastern University, College of Computer Science, 1989. find
  25. P. J. Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 1988. find
  26. JS Garofolo, LF Lamel, WM Fisher, JG Fiscus, DS Pallett, and NL Dahlgren. DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CD-ROM. National Institute of Standards and Technology, NTIS Order No PB91-505065, 1993. find
  27. Felix A. Gers, Juan Antonio Pérez-Ortiz, Douglas Eck, and Jürgen Schmidhuber. DEFK-LSTM. In ESANN 2002, Proceedings of the 10th Eurorean Symposium on Artificial Neural Networks, 2002. find
  28. J Schmidhuber, D Wierstra, M Gagliolo, and F J Gomez. Training Recurrent Networks by EVOLINO. Neural Computation, 19(3):757–779, 2007. find
  29. Justin Bayer, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. Evolving memory cell structures for sequence learning. In Artificial Neural Networks–ICANN 2009, pages 755–764. Springer, 2009. URL http://link.springer.com/chapter/10.1007/978-3-642-04277-5 76. — [doi:10.1007/978-3-642-04277-5](https://doi.org/10.1007/978-3-642-04277-5) ↗
  30. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2342– 2350, 2015. find
  31. Sebastian Otte, Marcus Liwicki, and Andreas Zell. Dynamic Cortex Memory: Enhancing Recurrent Neural Networks for Gradient-Based Sequence Learning. In Artificial Neural Networks and Machine Learning – ICANN 2014, number 8681 in Lecture Notes in Computer Science, pages 1–8. Springer International Publ… — [doi:10.1007/978-3-319-11179-7](https://doi.org/10.1007/978-3-319-11179-7) find
  32. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.1078, 2014. URL http://arxiv.org/abs/1406.1078. ↗
  33. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555 [cs], December 2014. URL http://arxiv.org/abs/1412.3555. ↗
  34. David Crystal. Dictionary of linguistics and phonetics, volume 30. John Wiley & Sons, 2011. find
  35. P. Mermelstein. Distance measures for speech recognition: Psychological and instrumental. In C. H. Chen, editor, Pattern Recognition and Artificial Intelligence, pages 374– 388. Academic Press, New York, 1976. find
  36. Alexander Graves. Supervised Sequence Labelling with Recurrent Neural Networks. Ph.d., The Technical University of Munich, 2008. find
  37. Andrew K. Halberstadt. Heterogeneous acoustic measurements and multiple classifiers for speech recognition. PhD thesis, Massachusetts Institute of Technology, 1998. find
  38. Marcus Liwicki and Horst Bunke. IAM-OnDB-an on-line English sentence database acquired from handwritten text on a whiteboard. In Document Analysis and Recognition, 2005. Proceedings. Eighth International Conference on, pages 956–961. IEEE, 2005. find
  39. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376, 2006. URL http://dl.acm.org/ci… ↗
  40. Moray Allan and Christopher KI Williams. Harmonising chorales by probabilistic inference. Advances in neural information processing systems, 17:25–32, 2005. find
  41. Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription. pages 1159–1166, 2012. URL http://icml.cc/discuss/2012/590.html. ↗
  42. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In JMLR, pages 1139–1147, 2013. URL http://jmlr.org/proceedings/papers/v28/sutskever13. html. ↗
  43. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 2951–2959. Curran Associates, Inc., 2012. find
  44. F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential Model-Based Optimization for General Algorithm Klaus Greff received his Diploma in Computer Science from the University of Kaiserslautern, Germany in 2011. Currently he is pursuing his PhD at IDSIA in Lugano, Switzerland, under the supervision o… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/greff-2015-lstm-search-space-odyssey/