Pointer Sentinel Mixture Models
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1609.07843
Summary
Language models that predict words through a softmax over a fixed vocabulary handle rare or unseen words poorly. This holds even when the right word appeared a few sentences earlier, because an LSTM loses information when it carries a word across many hidden-state steps. Merity and colleagues propose the pointer sentinel mixture model. It combines a standard softmax-RNN with a pointer network that attends over a window of recent hidden states and can copy a word from that window. A learned sentinel vector is added to the pointer's attention scores. This lets the pointer decide how much probability to pass to the vocabulary softmax, based on both the current query and what is in the window. A medium-sized pointer sentinel-LSTM scores 70.9 test perplexity on the Penn Treebank, a new state of the art. It has about 21M parameters and beats LSTMs three times larger. Penn Treebank is small and limited to 10k words, so the authors also release WikiText, a larger corpus built from Wikipedia for studying long-range context. The work shows that a cheap copy mechanism helps substantially with rare words.
Key points
- Penn Treebank: the pointer sentinel-LSTM (medium) scores 72.4 validation and 70.9 test perplexity with 21M parameters. Gal (2015)'s large variational LSTM uses 66M parameters and costly MC dropout at test time, and still scores 73.4.
- WikiText-2: the model scores 80.8 test perplexity, compared with 96.3 for Gal's variational LSTM and 100.9 for Zoneout + Variational LSTM. All three use a two-layer LSTM with 650 units.
- How the model works: the pointer uses pointer sum attention (Kadlec et al., 2016), adding up attention over every position where a word occurs. The query comes from a tanh projection of the last hidden state, which avoids a bias toward the most recent word. The gate g is the sentinel's share of the joint softmax.
- Where it helps: gains are largest for rare words, such as names like Rosenthal (seen 3 times in training). The pointer also helps with frequent tokens like units and number scales. It attends as far back as position 97 of its 100-word window, well beyond the 35-step backpropagation limit.
- Resource: the WikiText-2 and WikiText-103 corpora are freely available as benchmarks for long-range dependencies and realistic vocabularies.
Cited by
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
- Identifying and Reducing Gender Bias in Word-Level Language Models
- Evaluating prose style transfer with the Bible
- Language Modeling with Gated Convolutional Networks
- Reducing Transformer Depth on Demand with Structured Dropout
- Representation Degeneration Problem in Training Natural Language Generation Models
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- Editing Models with Task Arithmetic
- Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
- SqueezeLLM: Dense-and-Sparse Quantization
- Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations
- Large language models are zero-shot reasoners
- Internet-Augmented Dialogue Generation
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- Contrastive Decoding: Open-ended Text Generation as Optimization
- Towards Debiasing Sentence Representations
- Towards Understanding and Mitigating Social Biases in Language Models
- LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
- Gender Bias in Neural Natural Language Processing
- Quark: Controllable Text Generation with Reinforced Unlearning
- LLM-Pruner: On the Structural Pruning of Large Language Models
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Regularizing and Optimizing LSTM Language Models
- Augmented Language Models: a Survey
- The Third Multilingual Surface Realisation Shared Task (SR'20): Overview and Evaluation Results
- A Comprehensive Overview of Large Language Models
- A Deep Reinforced Model for Abstractive Summarization
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Language Models as Knowledge Bases?
- Compressive Transformers for Long-Range Sequence Modelling
- Efficient Content-Based Sparse Attention with Routing Transformers
- Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Personality Traits in Large Language Models
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
- Retrieval Augmentation Reduces Hallucination in Conversation
- PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- Efficient Transformers: A Survey
- Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
- BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
- BLiMP: The Benchmark of Linguistic Minimal Pairs for English
- Emergent Abilities of Large Language Models
- Neural Text Generation with Unlikelihood Training
- Challenges in Data-to-Document Generation
- Dynamic Coattention Networks For Question Answering
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
- AlignScore: Evaluating Factual Consistency with A Unified Alignment Function
- A Survey of Large Language Models
- A Survey on Model Compression for Large Language Models
- Neural Architecture Search with Reinforcement Learning
Cite
@misc{merity2016pointer,
title = {{Pointer Sentinel Mixture Models}},
author = {Merity, Stephen and Xiong, Caiming and Bradbury, James and Socher, Richard},
year = {2016},
doi = {10.48550/arxiv.1609.07843},
eprint = {1609.07843},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1609.07843}
}References
- Adi, Yossi, Kermany, Einat, Belinkov, Yonatan, Lavi, Ofer, and Goldberg, Yoav. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. arXiv preprint arXiv:1608.04207, 2016. find our card
- Ahn, Sungjin, Choi, Heeyoul, Pärnamaa, Tanel, and Bengio, Yoshua. A Neural Knowledge Language Model. CoRR, abs/1608.00318, 2016. find
- Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR, 2015. find our card
- Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. arXiv preprint arXiv:1312.3005, 2013. find our card
- Cheng, Jianpeng, Dong, Li, and Lapata, Mirella. Long Short-Term Memory-Networks for Machine Reading. CoRR, abs/1601.06733, 2016. find our card
- Cheng, Wei-Chen, Kok, Stanley, Pham, Hoai Vu, Chieu, Hai Leong, and Chai, Kian Ming Adam. Language Modeling with Sum-Product Networks. In INTERSPEECH, 2014. find
- Gal, Yarin. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. arXiv preprint arXiv:1512.05287, 2015. find our card
- Gu, Jiatao, Lu, Zhengdong, Li, Hang, and Li, Victor O. K. Incorporating Copying Mechanism in Sequence-to-Sequence Learning. CoRR, abs/1603.06393, 2016. find our card
- Gülçehre, Çaglar, Ahn, Sungjin, Nallapati, Ramesh, Zhou, Bowen, and Bengio, Yoshua. Pointing the Unknown Words. arXiv preprint arXiv:1603.08148, 2016. find
- Hochreiter, Sepp and Schmidhuber, Jürgen. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, Nov 1997. ISSN 0899-7667. find
- Kadlec, Rudolf, Schmid, Martin, Bajgar, Ondrej, and Kleindienst, Jan. Text Understanding with the Attention Sum Reader Network. arXiv preprint arXiv:1603.01547, 2016. find
- Kim, Yoon, Jernite, Yacine, Sontag, David, and Rush, Alexander M. Character-aware neural language models. CoRR, abs/1508.06615, 2016. find our card
- Koehn, Philipp, Hoang, Hieu, Birch, Alexandra, Callison-Burch, Chris, Federico, Marcello, Bertoldi, Nicola, Cowan, Brooke, Shen, Wade, Moran, Christine, Zens, Richard, Dyer, Chris, Bojar, Ondej, Constantin, Alexandra, and Herbst, Evan. Moses: Open Source Toolkit for Statistical Machine Translation.… find
- Krueger, David, Maharaj, Tegan, Kramár, János, Pezeshki, Mohammad, Ballas, Nicolas, Ke, Nan Rosemary, Goyal, Anirudh, Bengio, Yoshua, Larochelle, Hugo, Courville, Aaron, et al. Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations. arXiv preprint arXiv:1606.01305, 2016. — [arXiv:1606.01305](https://arxiv.org/abs/1606.01305) find
- Kumar, Ankit, Irsoy, Ozan, Ondruska, Peter, Iyyer, Mohit, Bradbury, James, Gulrajani, Ishaan, Zhong, Victor, Paulus, Romain, and Socher, Richard. Ask me anything: Dynamic memory networks for natural language processing. In ICML, 2016. find our card
- Ling, Wang, Grefenstette, Edward, Hermann, Karl Moritz, Kociský, Tomás, Senior, Andrew, Wang, Fumin, and Blunsom, Phil. Latent Predictor Networks for Code Generation. CoRR, abs/1603.06744, 2016. find
- Marcus, Mitchell P., Santorini, Beatrice, and Marcinkiewicz, Mary Ann. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330, 1993. find our card
- Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In SLT, 2012. find
- Mikolov, Tomas, Karafiát, Martin, Burget, Lukás, Cernocký, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH, 2010. find
- Pascanu, Razvan, Çaglar Gülçehre, Cho, Kyunghyun, and Bengio, Yoshua. How to Construct Deep Recurrent Neural Networks. CoRR, abs/1312.6026, 2013a. find
- Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. In ICML, 2013b. find
- Rosenfeld, Roni. A Maximum Entropy Approach to Adaptive Statistical Language Modeling. 1996. find
- Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-To-End Memory Networks. In NIPS, 2015. find
- Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep. Pointer networks. In Advances in Neural Information Processing Systems, pp. 2692–2700, 2015. find
- Xiong, Caiming, Merity, Stephen, and Socher, Richard. Dynamic Memory Networks for Visual and Textual Question Answering. In ICML, 2016. find
- Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. find our card
- Zilly, Julian Georg, Srivastava, Rupesh Kumar, Koutnı́k, Jan, and Schmidhuber, Jürgen. Recurrent Highway Networks. arXiv preprint arXiv:1607.03474, 2016. find