Elamur Research⌕
☰
home / papers / 2019 / Understanding Dataset Design Choices for Multi-hop Reasoning

Understanding Dataset Design Choices for Multi-hop Reasoning

Paper · 2019 · Jifan Chen and Greg Durrett

Summary

Chen and Durrett ask whether WikiHop and HotpotQA, two datasets built to test multi-hop reading comprehension, actually force models to combine evidence across sentences. They design probes that cannot reason across sentences by construction. The first is a sentence-factored model, which scores each sentence against the question in isolation, in a bi-GRU bilinear variant and a Factored BiDAF variant. The second is a no-context model that compares only the question and the answer candidates. They also convert each dataset into the other's format (HotpotQA into 10-way multiple choice, WikiHop into span extraction) to separate the effect of supervision type from other dataset properties. The probes locate the answer sentence for more than half of the questions. The passage-free baseline nearly matches published WikiHop systems. Under multiple choice, MC-BiDAF++ and MC-MemNet beat that baseline by only about 2 points. Span supervision proves harder to exploit, yet strong results may still come from single-sentence matching. The authors recommend span-based supervision, routine no-context checks and adversarial sentence-factored tests when building multi-hop benchmarks.

Key points

  • On dev sets, Factored BiDAF found the answer-bearing sentence in 66.1% of WikiHop and 57.2% of HotpotQA cases (88.0% on SQuAD). The random baselines scored 6.5%, 5.4% and 22.1%.
  • The no-context baseline scored 59.70 on the WikiHop dev set. This beats Coref-GRU (56.00) and approaches MHQA-GRN (62.80) and Entity-GCN (64.80). It far exceeds the BiDAF result of 42.9% reported in the original WikiHop paper.
  • HotpotQA-MC was built from the answer plus 9 random entity distractors. On it, the scores were NoContext 68.01, MC-MemNet 68.75 and MC-BiDAF++ 70.01. On WikiHop-MC they were 59.70, 61.80 and 61.32.
  • Span2MC-BiDAF++, trained with span supervision and evaluated as multiple choice, reached 76.01 on HotpotQA, beating natively multiple-choice training. It scored 59.85 on WikiHop.
  • Adding more distractor options lowered all scores, but the no-context baseline still kept pace with the more complex models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States72 works · 55,722 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Reasoning · 510 works

Cite

Cite as Jifan Chen and Greg Durrett. 2019. Understanding Dataset Design Choices for Multi-hop Reasoning. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1405

@inproceedings{chen2019understanding,
  title = {{Understanding Dataset Design Choices for Multi-hop Reasoning}},
  author = {Jifan Chen and Greg Durrett},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1405},
  url = {https://doi.org/10.18653/v1/n19-1405}
}

References

27 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task. ACL. our card
  2. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International Conference on Machine Learning, pages 1378–1387. our card
  3. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. Deep Learning workshop at NeurIPS 2014. find
  4. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. ACL. find
  5. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. ACL. our card
  6. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. EMNLP. find
  7. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2018. Question answering by reasoning across documents with graph convolutional networks. EMNLP. our card
  8. Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2018. Neural Models for Reasoning over Multiple Mentions using Coreference. NAACL. our card
  9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. our card
  10. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations. arXiv preprint arXiv:1511.02301. our card
  11. Peter Jansen. 2018. Multi-hop Inference for Sentence-level TextGraphs: How Challenging is Meaningfully Combining Information for Science Question Answering? TextGraphs. find
  12. Peter A Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton T Morrison. 2018. Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. LREC. our card
  13. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. ACL. our card
  14. Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. EMNLP. our card
  15. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:… our card
  16. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. EMNLP. find
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. EMNLP. our card
  18. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. In EMNLP, volume 3, page 4. our card
  19. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. ICLR. our card
  20. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1047–1055. ACM. find
  21. Linfeng Song, Zhiguo Wang, Mo Yu, Yue Zhang, Radu Florian, and Daniel Gildea. 2018. Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks. arXiv preprint arXiv:1809.02040. — [arXiv:1809.02040](https://arxiv.org/abs/1809.02040) find
  22. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What Makes Reading Comprehension Questions Easier? EMNLP. our card
  23. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making Neural QA as Simple as Possible but not Simpler. CoNLL. find
  24. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics, 6:287–302. our card
  25. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards AI-Complete Question Answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. our card
  26. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP. our card
  27. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. ICLR. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2019-understanding-dataset-design-choices-multi/