Elamur Research
☰
home / papers / 2016 / A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1606.02858

Summary

This paper asks how much real language understanding the large cloze-style reading comprehension datasets from Hermann et al. (2015) actually require. Those datasets were built from CNN and Daily Mail news articles. Each question is a summary bullet point with one anonymized entity removed, and the system must recover that entity from the article. The authors study the task in two ways. First, they build two straightforward systems: a feature-based entity ranking classifier and a simplified attention-based neural model derived from the AttentiveReader. Second, they hand-label 100 CNN questions by the kind of reasoning each one needs. The neural model reaches 73.6% on CNN and 76.6% on Daily Mail, which is 7–10 points above earlier published results. The hand analysis shows that most answerable questions can be solved by finding one relevant sentence and matching it, often through paraphrase. About a quarter of the questions are broken by coreference errors or anonymization, which leaves little room for further gains. The authors conclude that the benchmark is easier and noisier than it first appeared, so it says little about deeper multi-sentence reasoning.

Key points

  • Data: CNN has about 380k training questions and Daily Mail about 879k. Passages average roughly 30 sentences and 760–810 tokens, with around 26 candidate entities per passage. Entities are replaced with @entityN markers after NER and coreference resolution.
  • Entity-centric classifier: it ranks candidate entities with hand-built features, including frequency, first position, n-gram match around the placeholder, word distance, sentence co-occurrence and dependency-parse matches. On its own it already beats the earlier baselines.
  • Hand analysis of 100 CNN items: 13 are exact match, 41 paraphrasing, 19 partial clue, 2 multi-sentence, 8 coreference errors and 17 ambiguous or hard. On this sample the classifier scores 66% and the neural net 74%.
  • Paraphrasing: the neural model's advantage comes mainly from paraphrasing and partial-clue questions, where learned distributed representations handle lexical variation better. Both systems get all exact-match questions right and fail on most noisy ones.
  • Code: released at github.com/danqi/rc-cnn-dailymail. The data comes from DeepMind's rc-data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Coreference resolution

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1606.02858 · arXiv 1606.02858

@misc{chen2016thorough,
  title = {{A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task}},
  author = {Chen, Danqi and Bolton, Jason and Manning, Christopher D.},
  year = {2016},
  doi = {10.48550/arxiv.1606.02858},
  eprint = {1606.02858},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.02858}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP), pages 1499–1510. find
  2. Christopher J.C. Burges. 2013. Towards the machine comprehension of text: An essay. Technical report, Microsoft Research Technical Report MSR-TR-2013-125. find
  3. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Empirical Methods in Natural Language Processing (EMNLP), pages 740–750. find our card
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734… find
  5. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS), pages 1684–1692. find our card
  6. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR). find our card
  7. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Association for Computational Linguistics (ACL). find
  8. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Dynamic entity representation with max-pooling improves machine reading. In North American Association for Computational Linguistics (NAACL). find
  9. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International Conference on Machine Learning (ICML). find our card
  10. Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. 2016. Reasoning in vector space: An exploratory study of question answering. In International Conference on Learning Representations (ICLR). find
  11. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421. find
  12. Peter Norvig. 1978. A Unified Theory of Inference for Text Understanding. Ph.D. thesis, University of California, Berkeley. find
  13. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  14. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193–203. find our card
  15. Mrinmaya Sachan, Kumar Dubey, Eric Xing, and Matthew Richardson. 2015. Learning answer-entailing structures for machine comprehension. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL/IJCNLP), pages 239–249. find
  16. Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems (NIPS), pages 2431–2439. find
  17. Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL/IJCNLP), pages 700–706. find
  18. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In International Conference on Learning Representations (ICLR). find
  19. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards AI-complete question answering: A set of prerequisite toy tasks. In International Conference on Learning Representations (ICLR). find our card
  20. Qiang Wu, Christopher J. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval, pages 254–270. find