Elamur Research
☰
home / papers / 2018 / Constructing Datasets for Multi-hop Reading Comprehension Across Documents

Constructing Datasets for Multi-hop Reading Comprehension Across Documents

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q18-1021.pdf

Summary

Most reading comprehension benchmarks can be solved by finding one passage that contains the answer. This paper asks whether models can answer a question when the facts they need are spread over several texts. The authors define a multi-hop task: a model receives a query, an unordered set of support documents and a list of candidate answers, and must reason across documents to pick the right one. To build data at scale, they turn knowledge-base triples into queries and use breadth-first search over a graph that links documents to the entities they mention. The search also collects documents leading to plausible wrong answers of the same type. This method produced WikiHop, built from Wikipedia and Wikidata, and MedHop, built from MEDLINE abstracts and DrugBank for drug–drug interactions. The authors also describe dataset-construction pitfalls and how they worked around them. Strong baselines and two neural readers fall far short of humans, so the datasets are a demanding test of reasoning across documents.

Key points

  • The two datasets are WikiHop and MedHop, released as the QAngaroo resource at qangaroo.cs.ucl.ac.uk. Labels come from distant supervision using Wikidata and DrugBank as ground truth.
  • Annotators judged that the given documents support the answer in 74.1% of WikiHop samples and 68.0% of MedHop samples.
  • Each sample includes wrong candidates of the same type as the answer, so models cannot rely on type-matching shortcuts. The authors also flag pitfalls such as answers that co-occur spuriously with specific documents.
  • The two neural readers tested are BiDAF (Seo et al.) and FastQA (Weissenborn et al.). The best reached 54.5% accuracy on an annotated test set, compared with 85.0% for humans.
  • Both models get much better when given only documents known to be relevant. The authors therefore point to choosing the right documents as the most important direction for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

245 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 223 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00021

@article{welbl2018constructing,
  title = {{Constructing Datasets for Multi-hop Reading Comprehension Across Documents}},
  author = {Welbl, Johannes and Stenetorp, Pontus and Riedel, Sebastian},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00021},
  url = {https://doi.org/10.1162/tacl_a_00021}
}

References

79 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Ashburner, Catherine A. Ball, Judith A. Blake, David Botstein, Heather Butler, J. Michael Cherry, Allan P. Davis, Kara Dolinski, Selina S. Dwight, Janan T. Eppig, Midori A. Harris, David P. Hill, Laurie Issel-Tarver, Andrew Kasarskis, Suzanna Lewis, John C. Matese, Joel E. Richardson, Martin… find
  2. Amos Bairoch, Brigitte Boeckmann, Serenella Ferro, and Elisabeth Gasteiger. 2004. Swiss-Prot: Juggling between evolution and stability. Briefings in Bioinformatics, 5(1):39–55. find
  3. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, pages 2670–2676. find our card
  4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544. find
  5. Tamara Bobic, Roman Klinger, Philippe Thomas, and Martin Hofmann-Apitius. 2012. Improving distantly supervised extraction of drug-drug and protein-protein interactions. In Proceedings of the Joint Workshop on Unsupervised and Semi-Supervised Learning in NLP, pages 35–43. find
  6. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD 08 Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. find
  7. Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In Empirical Methods for Natural Language Processing (EMNLP), pages 615–620. find
  8. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. CoRR, abs/1506.02075. Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daumé, III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In… find
  9. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski, and Wei Wang. 2018. Ask the right questions: Active question reformulation with reinforcement learning. International Conference on Learning Representations (ICLR). find
  10. Claudio Carpineto and Giovanni Romano. 2012. A survey of automatic query expansion in information retrieval. ACM Comput. Surv., 44(1):1:1–1:50, January. find
  11. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358–2367. find our card
  12. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pages 2580–25… find
  13. Kevin Bretonnel Cohen and Lawrence Hunter. 2004. Natural language processing and systems biology. Artificial Intelligence Methods and Tools for Systems Biology, pages 147–173. find
  14. Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology, pages 77–86. find
  15. Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. 2017. Chains of reasoning over entities, relations, and text using recurrent neural networks. European Chapter of the Association for Computational Linguistics (EACL), pages 132–141. Matthew Dunn, Levent Sagun, Mike Higgins, V. U… find our card
  16. Antonio Fabregat, Konstantinos Sidiropoulos, Phani Garapati, Marc Gillespie, Kerstin Hausmann, Robin Haw, Bijay Jassal, Steven Jupe, Florian Korninger, Sheldon McKay, Lisa Matthews, Bruce May, Marija Milacic, Karen Rothfels, Veronica Shamovsky, Marissa Webber, Joel Weiser, Mark Williams, Guanming Wu… find
  17. Daniel Fried, Peter Jansen, Gustave Hahn-Powell, Mihai Surdeanu, and Peter Clark. 2015. Higher-order lexical semantic models for non-factoid answer reranking. Transactions of the Association of Computational Linguistics, 3:197–210. find
  18. Matt Gardner, Partha Pratim Talukdar, Bryan Kisiel, and Tom M. Mitchell. 2013. Improving learning and inference in a large knowledge-base using latent syntactic cues. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 833– 838. find
  19. Alex Graves. 2016. Adaptive computation time for recurrent neural networks. CoRR, abs/1603.08983. find
  20. Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedical Informatics, 45(5):885 – 892. T… find
  21. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701. find our card
  22. William Hersh, Aaron Cohen, Lynn Ruslen, and Phoebe Roberts. 2007. find
  23. TREC 2007 genomics track overview. In NIST Special Publication. find
  24. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. WIKIREADING: A novel large-scale language understanding task over Wikipedia. In Proceedings of the The 54th Annual Meeting of the Association for Computational Lin… find
  25. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. find our card
  26. ICLR. Lynette Hirschman, Alexander Yeh, Christian Blaschke, and Alfonso Valencia. 2005. Overview of BioCre-AtIvE: Critical assessment of information extraction for biology. BMC Bioinformatics, 6(1):S1, May. find
  27. Minghao Hu, Yuxing Peng, and Xipeng Qiu. 2017. Mnemonic reader for machine comprehension. CoRR, abs/1705.02798. find
  28. Sarthak Jain. 2016. Question answering over knowledge base using factual memory networks. In Proceedings of NAACL-HLT, pages 109–115. find
  29. Peter Jansen, Rebecca Sharp, Mihai Surdeanu, and Peter Clark. 2017. Framing QA as building and ranking intersentence answer justifications. Computational Linguistics, 43(2):407–449. find
  30. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  31. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, July. find our card
  32. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 908–918. find
  33. Jin-Dong Kim, Yue Wang, Toshihisa Takagi, and Akinori Yonezawa. 2011. Overview of Genia event task in BioNLP shared task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop, pages 7–15. find
  34. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, Ishaan Gulrajani James Bradbury, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. International Conference on Machine Learning, 48:1378–1387. find our card
  35. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, pages 159–174. find
  36. Ni Lao and William W Cohen. 2010. Relational retrieval using a combination of path-constrained random walks. Machine learning, 81(1):53–67. find
  37. Ni Lao, Tom Mitchell, and William W. Cohen. 2011. Random walk inference and learning in a large scale knowledge base. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 529–539. find
  38. Vivian Law, Craig Knox, Yannick Djoumbou, Tim Jewison, An Chi Guo, Yifeng Liu, Adam Maciejew-ski, David Arndt, Michael Wilson, Vanessa Neveu, Alexandra Tang, Geraldine Gabriel, Carol Ly, Sakina Adamjee, Zerihun T. Dame, Beomsoo Han, You Zhou, and David S. Wishart. 2014. DrugBank 4.0: Shedding new li… find
  39. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, August. find
  40. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules for question-answering. Nat. Lang. Eng., 7(4):343–360, December. find
  41. Fei Liu and Julien Perez. 2017. Gated end-to-end memory networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Volume 1: Long Papers, pages 1–10. find
  42. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages… find
  43. Alvaro Morales, Varot Premtoon, Cordelia Avery, Sue Felshin, and Boris Katz. 2016. Learning to answer questions from Wikipedia infoboxes. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1930–1935. find
  44. Karthik Narasimhan, Adam Yala, and Regina Barzilay. 2016. Improving information extraction by acquiring external evidence with reinforcement learning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, pages 2355–2365. find
  45. Arvind Neelakantan, Benjamin Roth, and Andrew Mc-Callum. 2015. Compositional vector space models for knowledge base completion. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 156… find
  46. Anastasios Nentidis, Konstantinos Bougiatiotis, Anastasia Krithara, Georgios Paliouras, and Ioannis Kakadiaris. 2017. Results of the fifth edition of the BioASQ challenge. In BioNLP 2017, pages 48–57. find
  47. Mark Neumann, Pontus Stenetorp, and Sebastian Riedel. 2016. Learning to reason with adaptive computation. In Interpretable Machine Learning for Complex Systems at the 2016 Conference on Neural Information Processing Systems (NIPS), Barcelona, Spain, December. find
  48. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268. find our card
  49. Rodrigo Nogueira and Kyunghyun Cho. 2016. WebNav: A new large-scale task for natural language based sequential decision making. CoRR, abs/1602.02261. — [arXiv:1602.02261](https://arxiv.org/abs/1602.02261) find
  50. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 574–583. find
  51. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David A. McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, pages 2230–2235. Denis Paperno, Germán Kruszewski, A… find
  52. Michael Pazzani, Clifford Brunk, and Glenn Silverstein. 1991. A knowledge-intensive approach to learning relational concepts. In Proceedings of the Eighth International Workshop on Machine Learning, pages 432– 436, Evanston, IL. find
  53. Baolin Peng, Zhengdong Lu, Hang Li, and Kam-Fai Wong. 2015. Towards neural network-based reasoning. CoRR, abs/1508.05508. — [arXiv:1508.05508](https://arxiv.org/abs/1508.05508) find
  54. Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence N-ary relation extraction with graph LSTMs. Transactions of the Association for Computational Linguistics, 5:101–115. find
  55. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  56. Bethany Percha, Yael Garten, and Russ B Altman. 2012. Discovery and explanation of drug-drug interactions via text mining. In Pacific symposium on biocomputing, page 410. find
  57. NIH Public Access. John Ross Quinlan. 1990. Learning logical definitions from relations. Machine Learning, 5:239–266. find
  58. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392. find our card
  59. Bradley L. Richards and Raymond J. Mooney. 1991. First-order theory revision. In Proceedings of the Eighth International Workshop on Machine Learning, pages 447–451, Evanston, IL. find
  60. Matthew Richardson and Pedro Domingos. 2006. Markov logic networks. Mach. Learn., 62(1-2):107– 136. find
  61. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Proceedings of the 2010 European Conference on Machine Learning and Knowledge Discovery in Databases: Part III, ECML PKDD’10, pages 148–163. find
  62. Stefan Schoenmackers, Oren Etzioni, and Daniel S. Weld. 2008. Scaling textual inference to the web. In EMNLP ’08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 79–88. find
  63. Stefan Schoenmackers, Oren Etzioni, Daniel S. Weld, and Jesse Davis. 2010. Learning first-order horn clauses from web text. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 1088–1098. find
  64. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 15–… find
  65. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017a. Bidirectional attention flow for machine comprehension. In The International Conference on Learning Representations (ICLR). find our card
  66. Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2017b. Query-reduction networks for question answering. find
  67. ICLR. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. ReasoNet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, pages 1047–1055. find
  68. Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. CoRR, abs/1606.02245. Pontus Stenetorp, Goran Topić, Sampo Pyysalo, Tomoko Ohta, Jin-Dong Kim, and Jun’ichi Tsujii. 2011. BioNLP shared task 2011: Supporting resources. In Proce… — [arXiv:1606.02245](https://arxiv.org/abs/1606.02245) find
  69. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems, pages 2440–2448. find
  70. Don R. Swanson. 1986. Undiscovered public knowledge. The Library Quarterly, 56(2):103–118. find
  71. The UniProt Consortium. 2017. UniProt: the universal protein knowledgebase. Nucleic Acids Research, 45(D1):D158–D169. Denny Vrandečić. 2012. Wikidata: A new platform for collaborative data collection. In Proceedings of the 21st International Conference on World Wide Web, WWW ’12 Companion, pages 1… find
  72. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 271–280. Association for Computational Linguistics. find
  73. Dirk Weissenborn. 2016. Separating answers from queries for neural reading comprehension. CoRR, abs/1607.03316. — [arXiv:1607.03316](https://arxiv.org/abs/1607.03316) find
  74. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the Third Workshop on Noisy User-generated Text, pages 94–106. find our card
  75. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. find
  76. ICLR. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards AI-complete question answering: A set of prerequisite toy tasks. find our card
  77. ICLR. Georg Wiese, Dirk Weissenborn, and Mariana Neves. 2017. Neural question answering at BioASQ 5B. In Proceedings of the BioNLP 2017, pages 76–79. find
  78. Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. find our card
  79. ICLR. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018. find our card