Elamur Research⌕
☰
home / papers / 2019 / Compositional Questions Do Not Necessitate Multi-hop Reasoning

Compositional Questions Do Not Necessitate Multi-hop Reasoning

Summary

The paper asks whether questions that appear compositional actually force a reading comprehension system to combine evidence from several paragraphs. Focusing on HotpotQA, the authors argue that whether reasoning is truly multi-hop depends on both the question and the evidence supplied with it. They build single-paragraph BERT, a model that reads each candidate paragraph separately, so it cannot link facts across them. It then returns the answer from the paragraph it judges most likely to contain one. In the distractor setting, this model performs close to dedicated multi-hop systems. A human study and a manual categorization of bridge questions point to the main causes: distractors that are easy to rule out by entity type, and questions whose extra conditions are redundant. The authors also try harder distractors, chosen adversarially and filtered by type. These lower accuracy at first, but retraining recovers most of the loss. The study implies that future multi-hop datasets must select evidence carefully, either through stronger distractors or through better retrieval in open-domain settings.

Key points

  • Single-paragraph BERT scores 67.08 F1 on the HotpotQA distractor validation set, near multi-hop leaderboard models. It reaches only 38.06 F1 in the open-domain (Fullwiki) setting with bigram TF-IDF retrieval.
  • On 200 bridge questions, NLP graduate students scored 87.37 F1 with all ten paragraphs. With one gold paragraph withheld, they still scored 82.06 F1.
  • Of 100 manually analyzed bridge questions, only 27% truly required multi-hop reasoning. Another 35% were solvable because of weak distractors, and 8% were non-compositional single-hop questions. For comparison questions, the model performs near chance on the multi-hop types.
  • Adversarially selected distractors reduced F1 from 67.08 to 46.84, and type-filtered distractors reduced it to 40.73. After retraining, the model recovered to 60.10 and 58.42 F1, respectively.
  • With 500 TF-IDF-retrieved paragraphs, F1 is 39.12. It rises to 53.12 when the two gold paragraphs are added, which shows that retrieval failures dominate the open-domain errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 84 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1416

@inproceedings{min2019compositional,
  title = {{Compositional Questions Do Not Necessitate Multi-hop Reasoning}},
  author = {Min, Sewon and Wallace, Eric and Singh, Sameer and Gardner, Matt and Hajishirzi, Hannaneh and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1416},
  url = {https://doi.org/10.18653/v1/p19-1416}
}

References

18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP. find
  2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL. find our card
  3. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In NAACL. find
  4. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  6. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. find our card
  7. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. find our card
  8. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  9. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. TACL. find our card
  10. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop. find
  11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. find our card
  12. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. find our card
  13. Saku Sugawara, Kentaro fInui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In EMNLP. find
  14. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In NAACL. find
  15. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In TACL. find our card
  16. Caiming Xiong, Victor Zhong, and Richard Socher. 2018. DCN+: Mixed objective and deep residual coattention for question answering. In ICLR. find
  17. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP. find our card
  18. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/min-2019-compositional-questions-do-not-necessitate/