Elamur Research
☰
home / papers / 2017 / Regularizing and Optimizing LSTM Language Models

Regularizing and Optimizing LSTM Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1708.02182

Summary

The paper tackles a practical gap in word-level language modeling. Standard regularizers such as dropout and batch normalization transfer poorly to recurrent networks, and many RNN-specific fixes require custom cells that cannot run on fast libraries like NVIDIA cuDNN. The authors build AWD-LSTM, which relies only on techniques that leave the LSTM implementation untouched. The core idea is the weight-dropped LSTM: a single DropConnect mask on the hidden-to-hidden matrices, reused across all timesteps. They pair it with NT-ASGD, an averaged SGD variant that starts averaging automatically once validation perplexity stops improving over several checks. Additional regularizers include:

  • randomized BPTT lengths
  • dropout applied to the word embedding matrix
  • AR and TAR, penalties on the size of activations and on how much hidden states change from one timestep to the next
  • weight decay
  • tied, smaller embeddings

Evaluated on Penn Treebank and WikiText-2, the model sets new state-of-the-art perplexities, and adding a neural cache lowers them further. The work matters because carefully combined simple methods beat elaborate architectures while staying compatible with optimized black-box LSTM code. AWD-LSTM became a widely used baseline.

Key points

  • AWD-LSTM (tied) reaches test perplexity 57.3 on Penn Treebank and 65.8 on WikiText-2. With a continuous cache pointer, these fall to 52.8 and 52.0.
  • NT-ASGD uses a constant learning rate and a non-monotone trigger with logging interval L = one epoch and n = 5, values that worked across models and datasets. Removing it raises PTB test perplexity from 57.3 to 63.7.
  • In ablations, dropping weight-dropping hurts most, adding up to about 11 perplexity points (PTB validation 60.0 → 71.1).
  • Making embeddings full-sized grows the model to 43M parameters (PTB) and 70M (WT2) and worsens perplexity by roughly 8 points, which the authors attribute to overfitting. Fixed BPTT length costs about 1 point.
  • On WikiText-2, the cache helps most on <unk> and rare names such as Meridian, which occurs only 161 times. It hurts frequent tokens like punctuation and stop words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

178 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 170 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1708.02182 · arXiv 1708.02182

@misc{merity2017regularizing,
  title = {{Regularizing and Optimizing LSTM Language Models}},
  author = {Merity, Stephen and Keskar, Nitish Shirish and Socher, Richard},
  year = {2017},
  doi = {10.48550/arxiv.1708.02182},
  eprint = {1708.02182},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1708.02182}
}

References

44 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120–1128, 2016. find
  2. Ba, J., Kiros, J., and Hinton, G. E. Layer normalization. CoRR, abs/1607.06450, 2016. find our card
  3. Balduzzi, D. and Ghifary, M. Strongly-typed recurrent neural networks. arXiv preprint arXiv:1602.02218, 2016. — [arXiv:1602.02218](https://arxiv.org/abs/1602.02218) find
  4. Bottou, L., Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. arXiv preprint arXiv:1606.04838, 2016. — [arXiv:1606.04838](https://arxiv.org/abs/1606.04838) find
  5. Bradbury, J., Merity, S., Xiong, C., and Socher, R. Quasi-Recurrent Neural Networks. arXiv preprint arXiv:1611.01576, 2016. — [arXiv:1611.01576](https://arxiv.org/abs/1611.01576) find
  6. Cooijmans, T., Ballas, N., Laurent, C., and Courville, A. C. Recurrent batch normalization. CoRR, abs/1603.09025, 2016. — [arXiv:1603.09025](https://arxiv.org/abs/1603.09025) find
  7. Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121– 2159, 2011. find
  8. Földiák, P. Learning invariance from transformation sequences. Neural Computation, 3(2):194–200, 1991. find
  9. Gal, Y. and Ghahramani, Z. A theoretically grounded application of dropout in recurrent neural networks. In NIPS, 2016. find our card
  10. Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. find
  11. Grave, E., Joulin, A., and Usunier, N. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426, 2016. find
  12. Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.01240, 2015. — [arXiv:1509.01240](https://arxiv.org/abs/1509.01240) find
  13. Hinton, G. E. Connectionist learning procedures. Artificial intelligence, 40(1-3):185–234, 1989. find
  14. Inan, H., Khosravi, K., and Socher, R. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. arXiv preprint arXiv:1611.01462, 2016. find
  15. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. find
  16. Jing, L., Shen, Y., Dubček, T., Peurifoy, J., Skirlo, S., Tegmark, M., and Soljačić, M. Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNN. arXiv preprint arXiv:1612.05231, 2016. — [arXiv:1612.05231](https://arxiv.org/abs/1612.05231) find
  17. Jonschkowski, R. and Brock, O. Learning state representations with robotic priors. Auton. Robots, 39:407–428, 2015. find
  18. Keskar, N. and Saon, G. A nonmonotone learning rate strategy for sgd training of deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 4974–4978. find
  19. Kim, Y., Jernite, Y., Sontag, D., and Rush, A. M. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence, 2016. find our card
  20. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  21. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. Moses: Open source toolkit for statistical machine translation. In ACL, 2007. find
  22. Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N., Goyal, A., Bengio, Y., Larochelle, H., Courville, A., et al. Zoneout: Regularizing RNNss by randomly preserving hidden activations. arXiv preprint arXiv:1606.01305, 2016. — [arXiv:1606.01305](https://arxiv.org/abs/1606.01305) find
  23. Luciw, M. and Schmidhuber, J. Low complexity proto-value function learning from sensory observations with incremental slow feature analysis. Artificial Neural Networks and Machine Learning–ICANN 2012, pp. 279– 287, 2012. find
  24. Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017. — [arXiv:1704.04289](https://arxiv.org/abs/1704.04289) find
  25. Melis, G., Dyer, C., and Blunsom, P. On the State of the Art of Evaluation in Neural Language Models. arXiv preprint arXiv:1707.05589, 2017. find
  26. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. arXiv preprint arXiv:1609.07843, 2016. find our card
  27. Merity, S., McCann, B., and Socher, R. Revisiting activation regularization for language rnns. arXiv preprint arXiv:1708.01009, 2017. — [arXiv:1708.01009](https://arxiv.org/abs/1708.01009) find
  28. Mikolov, T. and Zweig, G. Context dependent recurrent neural network language model. SLT, 12:234–239, 2012. find
  29. Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. Recurrent neural network based language model. In INTERSPEECH, 2010. find
  30. Panageas, I. and Piliouras, G. Gradient descent converges to minimizers: The case of non-isolated critical points. CoRR, abs/1605.00405, 2016. — [arXiv:1605.00405](https://arxiv.org/abs/1605.00405) find
  31. Polyak, B. and Juditsky, A. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992. find
  32. Press, O. and Wolf, L. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016. find our card
  33. Semeniuta, S., Severyn, A., and Barth, E. Recurrent dropout without memory loss. In COLING, 2016. find
  34. Seo, M., Min, S., Farhadi, A., and Hajishirzi, H. Query-Reduction Networks for Question Answering. arXiv preprint arXiv:1606.04582, 2016. — [arXiv:1606.04582](https://arxiv.org/abs/1606.04582) find
  35. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014. find
  36. Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139–1147, 2013. find
  37. Tallec, C. and Ollivier, Y. Unbiasing truncated backpropagation through time. arXiv preprint arXiv:1705.08209, 2017. — [arXiv:1705.08209](https://arxiv.org/abs/1705.08209) find
  38. Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31, 2012. find
  39. Wan, L., Zeiler, M., Zhang, S., LeCun, Y, and Fergus, R. Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning (ICML-13), pp. 1058–1066, 2013. find
  40. Wilson, A. C, Roelofs, R., Stern, M., Srebro, N., and Recht, B. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017. — [arXiv:1705.08292](https://arxiv.org/abs/1705.08292) find
  41. Wisdom, S., Powers, T., Hershey, J., Le Roux, J., and Atlas, L. Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 4880–4888, 2016. find
  42. Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. find our card
  43. Zilly, J. G., Srivastava, R. K., Koutník, J., and Schmidhuber, J. Recurrent highway networks. arXiv preprint arXiv:1607.03474, 2016. find
  44. Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016. find our card