Elamur Research
☰
home / papers / 2020 / Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.coling-main.580.pdf

Summary

Multi-hop reading comprehension benchmarks are meant to make a model combine facts from several paragraphs. Existing ones have two problems: they rarely explain how the answer is reached, and many of their questions can be solved in a single hop. The authors address this with 2WikiMultiHopQA, built where Wikipedia article summaries overlap with Wikidata statements. Questions come from hand-crafted templates and from logical rules over the knowledge graph. For example, a "mother" link followed by a "father" link becomes "maternal grandfather". Templates that previous analysis showed lead to single-hop or context-dependent questions were removed. Besides sentence-level supporting facts, each example includes "evidence": a set of Wikidata triples that traces the reasoning path. This supports a new evidence-generation task, scored together with answer and supporting-fact prediction. A multi-hop baseline scores lower on this dataset than on HotpotQA, while human scores are about the same on both. A single-hop BERT model also loses 8.7 F1 compared with HotpotQA. Together these results suggest real multi-hop reasoning is needed. The dataset is a large, explainable benchmark for testing reasoning in QA.

Key points

  • 2WikiMultiHopQA contains 192,606 questions of four types: comparison, inference, compositional and bridge-comparison. It is far larger than R4C, the other dataset with explanations, which has 4,588 questions.
  • There are three tasks: answer span prediction, sentence-level supporting-fact prediction and evidence (triple) generation. Each is scored with EM and F1, plus joint EM/F1 metrics that combine all three.
  • Comparison templates were derived from 17,456 HotpotQA comparison questions using spaCy NER labels. Inference questions use rules of the form r1(a,b) ∧ r2(b,c) ⇒ r(a,c).
  • A manual check of 100 training samples found 8 where the Wikipedia sentence and the Wikidata triple don't match.
  • The data, baseline model and construction resources are released at github.com/Alab-NII/2wikimultihop.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

482 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 470 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.580 · arXiv 2011.01060

@inproceedings{ho2020constructing,
  title = {{Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps}},
  author = {Ho, Xanh and Nguyen, Anh-Khoa Duong and Sugawara, Saku and Aizawa, Akiko},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.580},
  eprint = {2011.01060},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.coling-main.580}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard Weikum. 2017. Automated template generation for question answering over knowledge graphs. In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, page 1191–1200, Republic and Canton of Geneva, CHE. International… find
  2. Rakesh Agrawal, Tomasz Imieliundefinedski, and Arun Swami. 1993. Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, SIGMOD ’93, page 207–216, New York, NY, USA. Association for Computing Machinery. find
  3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October. Association for Computational Ling… find
  4. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. volume abs/1506.02075. find
  5. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for Multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4026–4032, Minne… find
  6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada, July. Association for Computation… find our card
  7. Wenhu Chen, Hanwen Zha, Zhi yu Chen, Wenhan Xiong, Hong Wang, and Wei Wang. 2020. HybridQA: A dataset of multi-hop question answering over tabular and textual data. ArXiv, abs/2004.07347. find our card
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Dennis Diefenbach, Thomas Tanon, Kamal Singh, and Pierre Maret. 2017. Question answering benchmarks for Wikidata. 10. find
  10. Luis Antonio Galárraga, Christina Teflioudi, Katja Hose, and Fabian Suchanek. 2013. AMIE: Association rule mining under incomplete evidence in ontological knowledge bases. In Proceedings of the 22nd International Conference on World Wide Web, WWW ’13, page 413–422, New York, NY, USA. Association fo… find
  11. Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. 2020. R4C: A benchmark for evaluating RC systems to get the right answer for the right reason. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740–6750, Online, July. Association for Computational Lin… find
  12. Peter Jansen, Niranjan Balasubramanian, Mihai Surdeanu, and Peter Clark. 2016. What’s in an explanation? Characterizing knowledge and inference requirements for elementary science exams. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers,… find
  13. The COLING 2016 Organizing Committee. find
  14. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. volume abs/1907.11692. find
  16. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Prismatic Inc, Steven J. Bethard, and David Mcclosky. 2014. The Stanford CoreNLP natural language processing toolkit. In In ACL, System Demonstrations. find our card
  17. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249–4257, Florence, Italy, July. As… find
  18. Stephen Muggleton. 1995. Inverse entailment and Progol. find
  19. J. R. Quinlan. 1990. Learning logical definitions from relations. volume 5, page 239–266, USA, September. Kluwer Academic Publishers. find
  20. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! Leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932–4942, Florence, Italy, July. Association for Co… find our card
  21. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. Association for Computational Lin… find our card
  22. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia, July. Association for Computationa… find our card
  23. Stefan Schoenmackers, Jesse Davis, Oren Etzioni, and Daniel Weld. 2010. Learning first-order Horn clauses from web text. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1088–1098, Cambridge, MA, October. Association for Computational Linguistics. find
  24. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4208–4219, Brussels, Belgium, October-November. Association for Computational Lingu… find
  25. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, L… find
  26. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. volume 6, pages 287–302. Transactions of the Association for Computational Linguistics. find our card
  27. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369… find our card
  28. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XL-Net: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32, pages 5753–5763. Curran Associates, Inc. find
  29. Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur… find
  30. Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2018b. Variational reasoning for question answering with knowledge graph. In The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18). find