Elamur Research
☰
home / papers / 2016 / The LAMBADA dataset: Word prediction requiring a broad discourse context

The LAMBADA dataset: Word prediction requiring a broad discourse context

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-1144.pdf

Summary

LAMBADA is an English benchmark for testing whether language models use broad discourse context or rely only on nearby words. Each item is a short narrative passage taken from unpublished novels in the Book Corpus. It has a context of about 4.6 sentences and a target sentence whose final word must be predicted. To build it, the authors removed passages that four simple language models found easy. Then they kept only items where crowd workers could guess the missing word from the full passage but not from the target sentence alone. The authors show that solving the items involves many linguistic phenomena, such as coreference, tracking dialogue participants and inferring how a scene develops. Several standard architectures were tested: N-gram models, RNN, LSTM and a Memory Network. None of them got above 1% accuracy. On an unfiltered control set, the same models got about 15–22% (8.5% for the Memory Network). Even guessing a random capitalized word from the passage beat all of them. This makes LAMBADA a hard test of whether models really understand discourse rather than just exploiting local statistics.

Key points

  • The source corpus is Book Corpus after deduplication and offensive-content filtering: 5,325 novels and 465M words. It was split by novel, so the training half (2,662 novels, over 200M words of raw text) shares no books with the dev/test passages.
  • Each context is the smallest run of whole sentences that adds up to at least 50 tokens. The target is always the final word of the next sentence.
  • Before human checking, a passage was dropped if any of four filtering language models gave its target word a probability of at least 0.00175.
  • On LAMBADA, the N-Gram, N-Gram w/cache, RNN, LSTM and Memory Network models scored 0–0.1% accuracy. Guessing a random capitalized word from the passage reached 7.3%.
  • N-Gram w/cache did best relative to the others, with perplexity 768 and median rank 87. The authors argue that strong models will need long-term memory plus reasoning over what is stored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Discourse analysis

Cited by

421 papers of our library cite it (of the 140673 whose bibliographies we read); 45 of them have a page here, 376 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1144 · arXiv 1606.06031

@inproceedings{paperno2016lambada,
  title = {{The LAMBADA dataset: Word prediction requiring a broad discourse context}},
  author = {Paperno, Denis and Kruszewski, Germán and Lazaridou, Angeliki and Pham, Ngoc Quan and Bernardi, Raffaella and Pezzelle, Sandro and Baroni, Marco and Boleda, Gemma and Fernández, Raquel},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1144},
  eprint = {1606.06031},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1144}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  2. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP, pages 632–642, Lisbon, Portugal. find
  3. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  4. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of NIPS, Montreal, Canada. Published online: https://papers.nips.cc/book/ advances-in-neural-information-processin… ↗ our card
  5. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The Goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of ICLR Conference Track, San Juan, Puerto Rico. Published online: http://www.iclr.cc/doku.php?id= iclr2016:main. ↗ our card
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–178–. find
  7. Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. http://arxiv.org/abs/ 1511.03962. ↗
  8. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Honza Cernocky. 2011. Rnnlm - recurrent neural network language. In Proceedings of ASRU. IEEE Automatic Speech Recognition and Understanding Workshop. find
  9. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. http://arxiv.org/ abs/1301.3781/. ↗
  10. Tomas Mikolov, Armand Joulin, Sumit Chopra, Michael Mathieu, and Marc’Aurelio Ranzato. 2015. Learning longer memory in recurrent neural networks. In Proceedings of ICLR Workshop Track, San Diego, CA. Published online: http://www. iclr.cc/doku.php?id=iclr2015:main. find
  11. Tomas Mikolov. 2014. Using neural networks for modelling and representing natural languages. Slides presented at COL-ING, online at http://www.coling-2014.org/COLING\\2014\\Tutorial-fix\\-\\TomasMikolov.pdf. ↗
  12. Tian Wang and Kyunghyun Cho. 2015. Larger-context language modelling. http://arxiv. org/abs/1511.03729. — [arXiv:1511.03729](https://arxiv.org/abs/1511.03729) find
  13. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards AI-complete question answering: A set of prerequisite toy tasks. http://arxiv.org/abs/1502.05698. ↗ our card
  14. Wenpeng Yin and Hinrich Schütze. 2015. Multi-GranCNN: An architecture for general matching of text chunks on multiple levels of granularity. In Proceedings of ACL, pages 63–73, Beijing, China. find
  15. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV 2015, pages 19–27. find our card
  16. Geoffrey Zweig and Christopher Burges. 2011. The Microsoft Research sentence completion challenge. Technical Report MSR-TR-2011-129, Microsoft Research. find
  17. Matthew Richardson, Christopher Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of EMNLP, pages 193–203, Seattle, WA. find our card
  18. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedings of ICLR Conference Track, San Juan, Puerto Rico. Published online: http://www. iclr.cc/doku.php?id=iclr2016:main. find our card
  19. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of NAACL, pages 196–205, Denver, CO. find our card
  20. Andreas Stolcke. 2002. Srilm-an extensible language modeling toolkit. In INTERSPEECH, volume 2002, page 2002. find
  21. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. http://arxiv.org/abs/1503. 08895. ↗
  22. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proceedings of the ICML Deep Learning Workshop, Lille, France. Published online: https://sites.google.com/site/ deeplearning2015/accepted-papers. ↗ our card