Elamur Research
☰
home / papers / 2015 / Teaching Machines to Read and Comprehend

Teaching Machines to Read and Comprehend

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1506.03340

Summary

Research on machine reading had no large supervised datasets of documents paired with questions and answers. Earlier systems therefore depended on hand-built grammars, information extraction or small synthetic corpora. The authors generate this kind of data automatically from CNN and Daily Mail news articles. Each article comes with bullet-point highlights that summarise it in new wording rather than copying it. They turn each highlight into a Cloze-style query by blanking out one entity at a time. Models should have to use the document, not background knowledge or language-model guesses. To ensure this, a coreference system groups mentions of the same entity, each entity becomes an abstract marker, and the markers are shuffled again every time an example is loaded. On this resource of about one million examples they compare frequency baselines, a PropBank frame-semantic pipeline, a word-distance heuristic and several LSTM-based neural readers. The attention-based Attentive and Impatient Readers score highest. They beat the symbolic NLP methods and a Deep LSTM Reader that has no attention. The work created a widely adopted benchmark and showed that attention helps networks combine evidence spread across long texts.

Key points

  • Corpus scale: about 93k CNN and 220k Daily Mail articles, giving roughly 380k CNN and 879k Daily Mail training queries. Documents average about 760–810 tokens, and articles longer than 2,000 tokens were removed. The generation code is released at github.com/deepmind/rc-data.
  • Task difficulty: after anonymisation, choosing the most frequent entity in the document gives the correct answer only 30.5% of the time on CNN and 25.6% on Daily Mail.
  • Symbolic benchmarks: the frame-semantic approach suffers from poor parser coverage and cannot handle queries whose answer needs several sentences. The word-distance benchmark (maximum penalty m = 8) does surprisingly well, especially on Daily Mail, because the highlights often reuse words from the article.
  • Neural models: the Uniform Reader, which ignores attention weights, performs poorly, which supports the claim that attention drives the gains. The Deep LSTM Reader copes reasonably with sequences of up to 2,000 tokens. The Attentive and Impatient Readers do best even though they use only single-layer LSTMs.
  • Analysis: attention heat maps show the Attentive Reader handling synonyms ("killed" → "deceased") and coreference while also using how close query words sit to candidate answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

1285 papers of our library cite it (of the 140673 whose bibliographies we read); 116 of them have a page here, the first 100 below, 1169 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.03340 · arXiv 1506.03340

@misc{hermann2015teaching,
  title = {{Teaching Machines to Read and Comprehend}},
  author = {Hermann, Karl Moritz and Kočiský, Tomáš and Grefenstette, Edward and Espeholt, Lasse and Kay, Will and Suleyman, Mustafa and Blunsom, Phil},
  year = {2015},
  doi = {10.48550/arxiv.1506.03340},
  eprint = {1506.03340},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.03340}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ellen Riloff and Michael Thelen. A rule-based question answering system for reading comprehension tests. In Proceedings of the ANLP/NAACL Workshop on Reading Comprehension Tests As Evaluation for Computer-based Language Understanding Sytems. find
  2. Hoifung Poon, Janara Christensen, Pedro Domingos, Oren Etzioni, Raphael Hoffmann, Chloe Kiddon, Thomas Lin, Xiao Ling, Mausam, Alan Ritter, Stefan Schoenmackers, Stephen Soderland, Dan Weld, Fei Wu, and Congle Zhang. Machine reading at the University of Washington. In Proceedings of the NAACL HLT 20… find
  3. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. CoRR, abs/1410.3916, 2014. find
  4. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. CoRR, abs/1503.08895, 2015. find
  5. Terry Winograd. Understanding Natural Language. Academic Press, Inc., Orlando, FL, USA, 1972. find
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014. find our card
  7. Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Advances in Neural Information Processing Systems 27. find
  8. Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. DRAW: A recurrent neural network for image generation. CoRR, abs/1502.04623, 2015. find
  9. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of EMNLP. find our card
  10. Krysta Svore, Lucy Vanderwende, and Christopher Burges. Enhancing single-document summarization by combining RankNet and third-party sources. In Proceedings of EMNLP/CoNLL. find
  11. Kristian Woodsend and Mirella Lapata. Automatic generation of story highlights. In Proceedings of ACL, 2010. find
  12. Wilson L Taylor. “Cloze procedure”: a new tool for measuring readability. Journalism Quarterly, 30:415–433, 1953. find
  13. Dipanjan Das, Desai Chen, André F. T. Martins, Nathan Schneider, and Noah A. Smith. Frame-semantic parsing. Computational Linguistics, 40(1):9–56, 2013. find our card
  14. Karl Moritz Hermann, Dipanjan Das, Jason Weston, and Kuzman Ganchev. Semantic frame identification with distributed word representations. In Proceedings of ACL, June 2014. find
  15. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In Proceedings of ACL, 2014. find our card
  16. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537, November 2011. find our card
  17. Ilya Sutskever, Oriol Vinyals, and Quoc V. V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, November 1997. find
  19. Alex Graves. Supervised Sequence Labelling with Recurrent Neural Networks, volume 385 of Studies in Computational Intelligence. Springer, 2012. find
  20. T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012. find