Elamur Research
☰
home / papers / 2015 / The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.02301

Summary

Standard language-model evaluation averages perplexity over every token, so it mostly rewards guessing frequent function words. Hill and colleagues at Facebook AI Research built the Children's Book Test (CBT) to measure how well models use wider narrative context to predict meaning-bearing words. CBT is drawn from Project Gutenberg children's books. In each item, a model reads 20 sentences of context, then chooses the word removed from the 21st sentence out of 10 candidates. Items are split into four classes: named entities, common nouns, verbs and prepositions.

The authors compared n-gram, embedding, LSTM and Memory Network (MemNN) models. Inside MemNNs they varied what one memory slot holds: a single word, a short window around a candidate, or a full sentence. LSTMs did well on verbs and prepositions but poorly on entities and nouns. MemNNs with medium-sized window memories did best on those content words, especially when attention over memories was trained with self-supervision. The authors call this medium-size optimum the "Goldilocks Principle." The same setup set a new state of the art on CNN QA. The work matters because it shows that how a model stores context strongly affects how well it captures meaning.

Key points

  • CBT size: 98 training books with 669,343 questions; 5 validation books (8,000 questions) and 5 test books (10,000 questions); vocabulary of 53,628 words. The data is released through fb.ai/babi.
  • Comparison with MSRCC: CBT is larger (10,000 vs. 1,040 test items), offers 10 candidates instead of 5, and gives every item its surrounding context.
  • Test accuracy for MemNN with window memory and self-supervision: 0.666 on named entities and 0.630 on common nouns. Sentence-level memory scored only 0.318 and 0.305.
  • Where the gain shows: the memory model's advantage appears on content words only. On syntactic words like verbs and prepositions, it gives no clear gain over LSTM baselines.
  • CNN QA: the self-supervised window MemNN reached 0.684 test accuracy, the best reported result at the time. Anonymising entities in CBT hurt verbs and prepositions more than named entities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Named entity recognition

Cited by

226 papers of our library cite it (of the 140673 whose bibliographies we read); 36 of them have a page here, 190 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.02301 · arXiv 1511.02301

@misc{hill2015goldilocks,
  title = {{The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations}},
  author = {Hill, Felix and Bordes, Antoine and Chopra, Sumit and Weston, Jason},
  year = {2015},
  doi = {10.48550/arxiv.1511.02301},
  eprint = {1511.02301},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.02301}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Altmann, Gerry and Steedman, Mark. Interaction with context during human sentence processing. Cognition, 30(3):191–238, 1988. find
  2. Baayen, R Harald and Lieber, Rochelle. Word frequency distributions and lexical semantics. Computers and the Humanities, 30(4):281–291, 1996. find
  3. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166, 1994. find
  4. Binder, Jeffrey R and Desai, Rutvik H. The neurobiology of semantic memory. Trends in cognitive sciences, 15(11):527–536, 2011. find
  5. Bordes, Antoine, Usunier, Nicolas, Chopra, Sumit, and Weston, Jason. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015. find
  6. Dyer, Chris, Ballesteros, Miguel, Ling, Wang, Matthews, Austin, and Smith, Noah A. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2015. find our card
  7. Graves, Alex, Liwicki, Marcus, Bunke, Horst, Schmidhuber, Jürgen, and Fernández, Santiago. Unconstrained on-line handwriting recognition with recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 577–584, 2008. find
  8. Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil. Learning to transduce with unbounded memory. NIPS, 2015. find
  9. Hassall, John. Goldilocks and the three bears. In The Old Nursery Stories and Rhymes. Blackie & Son: London, 1904. find
  10. Heafield, Kenneth, Pouzyrevsky, Ivan, Clark, Jonathan H., and Koehn, Philipp. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pp. 690–696, Sofia, Bulgaria, August 2013. find our card
  11. Hermann, Karl Moritz, Kočiský, Tomáš, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa, and Blunsom, Phil. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS), 2015. URL http://arxiv.org/abs/1506.03340. ↗ our card
  12. Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012. find
  13. Huang, Eric H, Socher, Richard, Manning, Christopher D, and Ng, Andrew Y. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pp. 873–882. Association for Com… find
  14. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. NIPS, 2015. find
  15. Kuhn, Roland and De Mori, Renato. A cache-based natural language model for speech recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 12(6):570–583, 1990. find
  16. Kumar, Ankit, Irsoy, Ozan, Su, Jonathan, Bradbury, James, English, Robert, Pierce, Brian, Ondruska, Peter, Gulrajani, Ishaan, and Socher, Richard. Ask me anything: Dynamic memory networks for natural language processing. http://arxiv.org/abs/1506.07285, 2015. ↗ our card
  17. Luong, Minh-Thang, Pham, Hieu, and Manning, Christopher D. Effective approaches to attention-based neural machine translation. Proceedings of EMNLP, 2015. find
  18. Manning, Christopher D, Surdeanu, Mihai, Bauer, John, Finkel, Jenny, Bethard, Steven J, and Mc-Closky, David. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60, 2014. find our card
  19. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In SLT, pp. 234–239, 2012. find
  20. Richardson, Matthew, Burges, Christopher JC, and Renshaw, Erin. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, volume 1, pp. 2, 2013. find our card
  21. Rush, Alexander M, Chopra, Sumit, and Weston, Jason. A neural attention model for abstractive sentence summarization. Proceedings of EMNLP, 2015. find
  22. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-to-end memory networks. Proceedings of NIPS, 2015. find
  23. Wan, Li, Zeiler, Matthew, Zhang, Sixin, Cun, Yann L, and Fergus, Rob. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1058–1066, 2013. find
  24. Weston, Jason, Bengio, Samy, and Usunier, Nicolas. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learning, 81(1):21–35, 2010. find
  25. Weston, Jason, Bordes, Antoine, Chopra, Sumit, and Mikolov, Tomas. Towards ai-complete question answering: a set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015a. find our card
  26. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. Proceedings of ICLR, 2015b. find
  27. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  28. Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Courville, Aaron, Salakhutdinov, Ruslan, Zemel, Richard, and Bengio, Yoshua. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning (ICML’15), 2015. URL http://arxiv.org/abs/… ↗ our card
  29. Zweig, Geoffrey and Burges, Christopher JC. The microsoft research sentence completion challenge. Technical report, Technical Report MSR-TR-2011-129, Microsoft, 2011. find