Elamur Research⌕
☰
home / papers / 2018 / Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

Paper · 2018 · Urvashi Khandelwal, He He, Peng Qi et al.

Summary

Neural language models beat n-gram models, and this is often credited to their use of distant context. This paper tests that claim directly. It asks three things: how many preceding tokens an LSTM language model actually exploits, whether near and far context are represented differently, and how a copy mechanism changes context use. The authors take a standard pretrained AWD-LSTM (Merity et al., 2018) trained on Penn Treebank and WikiText-2. At test time only, they alter its history by truncating it, shuffling or reversing word order, or dropping and replacing target words. They then measure how much the loss rises. The model draws on roughly 200 tokens of history, and this holds across hyperparameter settings. Exact word order matters only in about the last sentence; farther back, the model seems to keep only a loose sense of topic. The LSTM can reproduce some recently seen words, but copying from distant context depends largely on the neural cache. The ablation method is model-agnostic and points to ways of improving dropout schemes, caches and memory-augmented models.

Key points

  • Effective context size: dropping tokens beyond about 150 (PTB) or 250 (WikiText-2) raises perplexity by only around 1%. Changing dropout, model size or BPTT length does not shift this pattern.
  • Context needs differ by word type:
    • Infrequent words (fewer than 800 training occurrences) need more than 200 tokens, while frequent words gain little beyond 50.
    • Nouns and verbs benefit from distant context, whereas determiners rely on only about the last 10 tokens.
  • Word order: order is important within roughly the 20 most recent tokens. Shuffling context beyond about 50 tokens has almost no effect.
  • Neural cache (Grave et al., 2017b): its chief contribution is enabling the model to copy words from long-range context. However, it hurts words absent from the history (about 36% on PTB and 20% on Wiki), where its distribution tends to be flat and uncertain.
  • Setup:
    • Dev perplexities without a cache are 59.07 (PTB) and 67.29 (Wiki).
    • Results are averaged over three random seeds.
    • The code is released as urvashik/lm-context-analysis on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Urvashi Khandelwal, He He, Peng Qi and Dan Jurafsky. 2018. Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context. arXiv:1805.04623.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1805.04623 · arXiv 1805.04623

@misc{khandelwal2018sharp,
  title = {{Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context}},
  author = {Urvashi Khandelwal, He He, Peng Qi and Dan Jurafsky},
  year = {2018},
  doi = {10.48550/arxiv.1805.04623},
  eprint = {1805.04623},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1805.04623}
}

References

28 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=BJh6Ztuxl. ↗
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations (ICLR) https://arxiv.org/pdf/1409.0473.pdf. ↗
  3. Jordan Boyd-Graber and David Blei. 2009. Syntactic topic models. In Advances in neural information processing systems. pages 185– 192. https://papers.nips.cc/paper/3398-syntactic-topic-models.pdf. ↗
  4. Ciprian Chelba, Mohammad Norouzi, and Samy Bengio. 2017. N-gram language modeling using recurrent neural network estimation. arXiv preprint arXiv:1703.10724 https://arxiv.org/pdf/1703.10724.pdf. — [arXiv:1703.10724](https://arxiv.org/abs/1703.10724) ↗
  5. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. International Conference on Machine Learning (ICML) https://arxiv.org/pdf/1612.08083.pdf. ↗
  6. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems (NIPS). pages 1019–1027. https://arxiv.org/pdf/1512.05287.pdf. ↗
  7. Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry Heck. 2016. Contextual lstm (clstm) models for large scale nlp tasks. Workshop on Large-scale Deep Learning for Data Mining, KDD https://arxiv.org/pdf/1602.06291.pdf. ↗
  8. Edouard Grave, Moustapha M Cisse, and Armand Joulin. 2017a. Unbounded cache model for online language modeling with open vocabulary. In Advances in Neural Information Processing Systems (NIPS). pages 6044–6054. https://papers.nips.cc/paper/7185-unbounded-cache-model-for-online-language-modeling-with… ↗
  9. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017b. Improving Neural Language Models with a Continuous Cache. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=B184E5qee. ↗
  10. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 https://arxiv.org/pdf/1308.0850.pdf. ↗
  11. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. International Conference on Learning Representations (ICLR) https://arxiv.org/pdf/1511.02301.pdf. ↗
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735. ↗
  13. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=r1aPbsFle. ↗
  14. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 https://arxiv.org/pdf/1602.02410.pdf. ↗
  15. Jey Han Lau, Timothy Baldwin, and Trevor Cohn. 2017. Topically Driven Neural Language Model. Association for Computational Linguistics (ACL) https://doi.org/10.18653/v1/P17-1033. ↗
  16. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. North American Association of Computational Linguistics (NAACL) http://www.aclweb.org/anthology/N16-1082. ↗
  17. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics (TACL) http://aclweb.org/anthology/Q16-1037. ↗
  18. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations. pages 55–60. https://doi.… ↗
  19. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics 19(2):313–330. http://aclweb.org/anthology/J93-2004. ↗
  20. Gabor Melis, Chris Dyer, and Phil Blunsom. 2018. On the State of the Art of Evaluation in Neural Language Models. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=ByJHuTgA-. ↗
  21. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and Optimizing LSTM Language Models. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=SyyGPP0TZ. ↗
  22. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=Byj72udxe. ↗
  23. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association. find
  24. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. European Chapter of the Association for Computational Linguistics http://aclweb.org/anthology/E17-2025. ↗
  25. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In International Conference on Machine Learning (ICML). pages 1058– 1066. find
  26. Tian Wang and Kyunghyun Cho. 2016. Larger-Context Language Modelling with Recurrent Neural Network. Association for Computational Linguistics (ACL) https://doi.org/10.18653/v1/P16-1125. ↗
  27. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2018. Breaking the softmax bottleneck: a high-rank rnn language model. International Conference on Learning Representations (ICLR) https://openreview.net/pdf?id=HkwZSG-CZ. ↗
  28. Hyperparameter PTB Wiki Word Emb. Size Hidden State Dim Layers Optimizer Learning Rate Gradient clip Epochs (train) Epochs (finetune) Batch Size Sequence Length LSTM Layer Dropout Recurrent Dropout Word Emb. Dropout Word Dropout FF Layers Dropout Weight Decay 400 400 1150 1150 3 3 ASGD ASGD 30 30 0.… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khandelwal-2018-sharp-nearby-fuzzy-far-away/