Elamur Research⌕
☰
home / papers / 2016 / Pointer Sentinel Mixture Models

Pointer Sentinel Mixture Models

Summary

Language models that predict words through a softmax over a fixed vocabulary handle rare or unseen words poorly. This holds even when the right word appeared a few sentences earlier, because an LSTM loses information when it carries a word across many hidden-state steps. Merity and colleagues propose the pointer sentinel mixture model. It combines a standard softmax-RNN with a pointer network that attends over a window of recent hidden states and can copy a word from that window. A learned sentinel vector is added to the pointer's attention scores. This lets the pointer decide how much probability to pass to the vocabulary softmax, based on both the current query and what is in the window. A medium-sized pointer sentinel-LSTM scores 70.9 test perplexity on the Penn Treebank, a new state of the art. It has about 21M parameters and beats LSTMs three times larger. Penn Treebank is small and limited to 10k words, so the authors also release WikiText, a larger corpus built from Wikipedia for studying long-range context. The work shows that a cheap copy mechanism helps substantially with rare words.

Key points

  • Penn Treebank: the pointer sentinel-LSTM (medium) scores 72.4 validation and 70.9 test perplexity with 21M parameters. Gal (2015)'s large variational LSTM uses 66M parameters and costly MC dropout at test time, and still scores 73.4.
  • WikiText-2: the model scores 80.8 test perplexity, compared with 96.3 for Gal's variational LSTM and 100.9 for Zoneout + Variational LSTM. All three use a two-layer LSTM with 650 units.
  • How the model works: the pointer uses pointer sum attention (Kadlec et al., 2016), adding up attention over every position where a word occurs. The query comes from a tanh projection of the last hidden state, which avoids a bias toward the most recent word. The gate g is the sentinel's share of the joint softmax.
  • Where it helps: gains are largest for rare words, such as names like Rosenthal (seen 3 times in training). The pointer also helps with frequent tokens like units and number scales. It attends as far back as position 97 of its 100-word window, well beyond the 35-step backpropagation limit.
  • Resource: the WikiText-2 and WikiText-103 corpora are freely available as benchmarks for long-range dependencies and realistic vocabularies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SalesforceSan Francisco · United States29 works · 14,417 citations

📚 Shelves 4 Computer Science · 7,349 works Artificial Intelligence · 7,183 works Arts and Humanities · 7,418 works Language and Linguistics · 7,357 works

Cite

Cite as Stephen Merity, Caiming Xiong, James Bradbury and Richard Socher. 2016. Pointer Sentinel Mixture Models. arXiv:1609.07843.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1609.07843 · arXiv 1609.07843

@misc{merity2016pointer,
  title = {{Pointer Sentinel Mixture Models}},
  author = {Stephen Merity, Caiming Xiong, James Bradbury and Richard Socher},
  year = {2016},
  doi = {10.48550/arxiv.1609.07843},
  eprint = {1609.07843},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1609.07843}
}

References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adi, Yossi, Kermany, Einat, Belinkov, Yonatan, Lavi, Ofer, and Goldberg, Yoav. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. arXiv preprint arXiv:1608.04207, 2016. our card
  2. Ahn, Sungjin, Choi, Heeyoul, Pärnamaa, Tanel, and Bengio, Yoshua. A Neural Knowledge Language Model. CoRR, abs/1608.00318, 2016. find
  3. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR, 2015. our card
  4. Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. arXiv preprint arXiv:1312.3005, 2013. our card
  5. Cheng, Jianpeng, Dong, Li, and Lapata, Mirella. Long Short-Term Memory-Networks for Machine Reading. CoRR, abs/1601.06733, 2016. our card
  6. Cheng, Wei-Chen, Kok, Stanley, Pham, Hoai Vu, Chieu, Hai Leong, and Chai, Kian Ming Adam. Language Modeling with Sum-Product Networks. In INTERSPEECH, 2014. find
  7. Gal, Yarin. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. arXiv preprint arXiv:1512.05287, 2015. our card
  8. Gu, Jiatao, Lu, Zhengdong, Li, Hang, and Li, Victor O. K. Incorporating Copying Mechanism in Sequence-to-Sequence Learning. CoRR, abs/1603.06393, 2016. our card
  9. Gülçehre, Çaglar, Ahn, Sungjin, Nallapati, Ramesh, Zhou, Bowen, and Bengio, Yoshua. Pointing the Unknown Words. arXiv preprint arXiv:1603.08148, 2016. our card
  10. Hochreiter, Sepp and Schmidhuber, Jürgen. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, Nov 1997. ISSN 0899-7667. find
  11. Kadlec, Rudolf, Schmid, Martin, Bajgar, Ondrej, and Kleindienst, Jan. Text Understanding with the Attention Sum Reader Network. arXiv preprint arXiv:1603.01547, 2016. our card
  12. Kim, Yoon, Jernite, Yacine, Sontag, David, and Rush, Alexander M. Character-aware neural language models. CoRR, abs/1508.06615, 2016. our card
  13. Koehn, Philipp, Hoang, Hieu, Birch, Alexandra, Callison-Burch, Chris, Federico, Marcello, Bertoldi, Nicola, Cowan, Brooke, Shen, Wade, Moran, Christine, Zens, Richard, Dyer, Chris, Bojar, Ondej, Constantin, Alexandra, and Herbst, Evan. Moses: Open Source Toolkit for Statistical Machine Translation.… find
  14. Krueger, David, Maharaj, Tegan, Kramár, János, Pezeshki, Mohammad, Ballas, Nicolas, Ke, Nan Rosemary, Goyal, Anirudh, Bengio, Yoshua, Larochelle, Hugo, Courville, Aaron, et al. Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations. arXiv preprint arXiv:1606.01305, 2016. — [arXiv:1606.01305](https://arxiv.org/abs/1606.01305) find
  15. Kumar, Ankit, Irsoy, Ozan, Ondruska, Peter, Iyyer, Mohit, Bradbury, James, Gulrajani, Ishaan, Zhong, Victor, Paulus, Romain, and Socher, Richard. Ask me anything: Dynamic memory networks for natural language processing. In ICML, 2016. our card
  16. Ling, Wang, Grefenstette, Edward, Hermann, Karl Moritz, Kociský, Tomás, Senior, Andrew, Wang, Fumin, and Blunsom, Phil. Latent Predictor Networks for Code Generation. CoRR, abs/1603.06744, 2016. find
  17. Marcus, Mitchell P., Santorini, Beatrice, and Marcinkiewicz, Mary Ann. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330, 1993. our card
  18. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In SLT, 2012. find
  19. Mikolov, Tomas, Karafiát, Martin, Burget, Lukás, Cernocký, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH, 2010. find
  20. Pascanu, Razvan, Çaglar Gülçehre, Cho, Kyunghyun, and Bengio, Yoshua. How to Construct Deep Recurrent Neural Networks. CoRR, abs/1312.6026, 2013a. find
  21. Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. In ICML, 2013b. find
  22. Rosenfeld, Roni. A Maximum Entropy Approach to Adaptive Statistical Language Modeling. 1996. find
  23. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-To-End Memory Networks. In NIPS, 2015. find
  24. Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep. Pointer networks. In Advances in Neural Information Processing Systems, pp. 2692–2700, 2015. find
  25. Xiong, Caiming, Merity, Stephen, and Socher, Richard. Dynamic Memory Networks for Visual and Textual Question Answering. In ICML, 2016. find
  26. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. our card
  27. Zilly, Julian Georg, Srivastava, Rupesh Kumar, Koutnı́k, Jan, and Schmidhuber, Jürgen. Recurrent Highway Networks. arXiv preprint arXiv:1607.03474, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/