Elamur Research⌕
☰
home / papers / 2019 / Multi-hop Reading Comprehension through Question Decomposition and Rescoring

Multi-hop Reading Comprehension through Question Decomposition and Rescoring

Summary

Multi-hop reading comprehension asks a system to combine evidence from several paragraphs. The paper presents DecompRC, which splits a compositional question into simpler single-hop sub-questions. An ordinary off-the-shelf single-hop RC model then answers these sub-questions.

Writing sub-questions word by word would need a lot of annotated data. The authors avoid this by treating decomposition as span prediction. A BERT-based model, Pointer_c, marks a few boundary positions inside the question. Light rule-based editing then turns those spans into sub-questions for three reasoning types: bridging, intersection and comparison. Only 400 annotated decompositions are needed to train it.

All reasoning types are tried in parallel, so the system does not commit to one interpretation up front. A decomposition scorer then judges each candidate together with its answers and evidence, and picks the final answer.

On HotpotQA, DecompRC reached state-of-the-art results in both the distractor and full wiki settings. It also proved more robust than an end-to-end BERT baseline under adversarial distractors and questions. The sub-questions make each prediction explainable.

Key points

  • Reasoning types: The authors manually inspected 200 HotpotQA dev questions, of which 184 needed multi-hop reasoning. Of these, 47% were bridging, 23% intersection and 22% comparison; the remaining 8% fit none of the three types.
  • Sample efficiency: Sub-questions produced by the span-based decomposer, trained on just 400 examples, worked about as well as sub-questions written by humans.
  • Rescoring vs. alternatives: The answer-aware scorer beat a pipeline that chooses the reasoning type beforehand, and also beat selection by RC-model confidence. It still trails an oracle by 6.2 F1.
  • Human upper bound: On 50 bridging questions, humans answered the annotated sub-questions without seeing the original question and reached 72.67 F1. Of the 15 failures, 53% came from groundtruth or human issues and 47% from genuine decomposition problems.
  • Limitations and resources: The method fails on questions that need implicit reasoning or commonsense inference, and on operations such as counting. Code and a demo are available at github.com/shmsw25/DecompRC.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

112 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1613

@inproceedings{min2019multi,
  title = {{Multi-hop Reading Comprehension through Question Decomposition and Rescoring}},
  author = {Min, Sewon and Zhong, Victor and Zettlemoyer, Luke and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1613},
  url = {https://doi.org/10.18653/v1/p19-1613}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP. find
  2. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In NAACL. find
  3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In ACL. find our card
  4. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP. find our card
  5. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. find our card
  6. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In NAACL. find
  7. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL. find our card
  8. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  10. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In TACL. find our card
  11. Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2018. Neural models for reasoning over multiple mentions using coreference. In NAACL. find our card
  12. Caiming Xiong, Victor Zhong, and Richard Socher. 2018. DCN+: Mixed objective and deep residual coattention for question answering. In ICLR. find
  13. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Artificial Intelligence Research. find
  14. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP. find our card
  15. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. find our card
  16. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. find our card
  17. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  18. Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In ACL. find
  19. Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and robust question answering from minimal context over documents. In ACL. find
  20. Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In ICLR. find
  21. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In AAAI/IAAI. find
  22. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI. find
  23. Victor Zhong, Caiming Xiong, Nitish Shirish Keskar, and Richard Socher. 2019. Coarse-grain fine-grain coattention network for multi-evidence question answering. In ICLR. find
  24. Jekaterina Novikova, Ondej Duek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In EMNLP. find
  25. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. find
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/min-2019-multi-hop-reading-comprehension-through/