Elamur Research
☰
home / papers / 2019 / Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering

Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1911.10470

Summary

Multi-hop open-domain question answering requires gathering several evidence passages, but some of them share almost no vocabulary or meaning with the question. Standard pipelines that fetch a fixed set of documents with TF-IDF or BM25 therefore often miss them. Asai et al. address this by treating retrieval as walking through a graph. Every Wikipedia paragraph is a node, and edges come from internal hyperlinks plus links between paragraphs of the same article. A recurrent retriever scores candidate paragraphs using BERT [CLS] encodings of question–paragraph pairs and updates its state after each choice. It stops by emitting an end-of-evidence token, so path length is not fixed. Beam search keeps several candidate paths, which a multi-task reader then re-ranks while extracting the answer span. Data augmentation and negative mining support training. The system sets a new state of the art on HotpotQA, beating the prior best full wiki result by over 14 points. It also matches or exceeds the best systems on SQuAD Open and Natural Questions Open, and it makes the reasoning chain inspectable.

Key points

  • Evaluated on HotpotQA (full wiki and distractor settings), SQuAD Open and Natural Questions Open, with the same architecture for all datasets.
  • On HotpotQA full wiki, the system outperforms the strongest earlier model by an absolute margin exceeding 14 points.
  • The retriever conditions each step on previously selected paragraphs. Paths end when the [EOE] symbol is chosen. The next candidate set combines hyperlinked paragraphs with K-best alternatives (K = 1 by default).
  • On the HotpotQA distractor dev set, the recurrent retriever's choice differs from a simple top-2 selection in 588 cases (7.9%).
  • Allowing paths longer than one paragraph helps little on single-hop SQuAD Open (+0.1% F1/EM). It gives about +1% EM on Natural Questions Open, where augmented training data lets the model learn multi-hop paths. Code and data are released at github.com/AkariAsai/learning_to_retrieve_reasoning_paths.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

133 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.10470 · arXiv 1911.10470

@misc{asai2019learning,
  title = {{Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering}},
  author = {Asai, Akari and Hashimoto, Kazuma and Hajishirzi, Hannaneh and Socher, Richard and Xiong, Caiming},
  year = {2019},
  doi = {10.48550/arxiv.1911.10470},
  eprint = {1911.10470},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.10470}
}

References

44 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450, 2016. find our card
  2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In ACL, 2017. find our card
  3. Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehension. In ACL, 2018. find our card
  4. Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. Multi-step retriever-reader interaction for scalable open-domain question answering. In ICLR, 2019. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. find
  6. Ming Ding, Chang Zhou, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. In ACL, 2019. find
  7. Yair Feldman and Ran El-Yaniv. Multi-hop paragraph retrieval for open-domain question answering. In ACL, 2019. find
  8. Paolo Ferragina and Ugo Scaiella. Fast and accurate annotation of short texts with wikipedia pages. IEEE software, 29(1):70–75, 2011. find
  9. Ameya Godbole, Dilip Kavarthapu, Rajarshi Das, Zhiyu Gong, Abhishek Singhal, Xiaoxiao Yu, Mo Guo, Tian Gao, Hamed Zamani, Manzil Zaheer, and Andrew McCallum. Multi-step entity-centric information retrieval for multi-hop question answering. In Proceedings of the 2nd Workshop on Machine Reading for Qu… find
  10. Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. Retrieve, read, rerank: Towards end-to-end multi-document reading comprehension. In ACL, 2019. find
  11. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In ICLR, 2017. find
  12. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015. find
  13. Bernhard Kratzwald and Stefan Feuerriegel. Adaptive document retrieval for deep question answering. In EMNLP, 2018. find
  14. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, et al. Natural questions: a benchmark for question answering research. TACL, 2019. find our card
  15. Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. Ranking paragraphs for improving answer recall in open-domain question answering. In EMNLP, 2018. find
  16. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In ACL, 2019. find our card
  17. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. Denoising distantly supervised open-domain question answering. In ACL, 2018. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv:1907.11692, 2019. find
  19. Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. Efficient and robust question answering from minimal context over documents. In ACL, 2018. find
  20. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. A discrete hard em approach for weakly supervised question answering. In EMNLP, 2019a. find
  21. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. Compositional questions do not necessitate multi-hop reasoning. In ACL, 2019b. find
  22. Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading comprehension through question decomposition and rescoring. In ACL, 2019c. find
  23. Yixin Nie, Songhe Wang, and Mohit Bansal. Revealing the importance of semantic retrieval for machine reading at scale. In EMNLP, 2019. find
  24. Kosuke Nishida, Kyosuke Nishida, Nagata Masaaki, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. Answering while summarizing: Multi-task learning for multi-hop qa with evidence extraction. In ACL, 2019. find
  25. Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. arXiv:1901.04085, 2019. find our card
  26. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In WMT, 2018. find our card
  27. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In EACL, 2017. find our card
  28. Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. Answering complex open-domain questions through iterative query generation. In EMNLP, 2019. find
  29. Martin Raison, Pierre-Emmanuel Mazaré, Rajarshi Das, and Antoine Bordes. Weaver: Deep co-encoding of questions and documents for machine reading. arXiv:1804.10490, 2018. — [arXiv:1804.10490](https://arxiv.org/abs/1804.10490) find
  30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. find our card
  31. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In ACL, 2018. find our card
  32. Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NeurIPS, 2016. find
  33. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In ICLR, 2017. find our card
  34. Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannaneh Hajishirzi. Real-time open-domain question answering with dense-sparse phrase index. In ACL, 2019. find
  35. Haitian Sun, Tania Bedrax-Weiss, and William Cohen. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In EMNLP, 2019. find
  36. Haoyu Wang, Mo Yu, Xiaoxiao Guo, Rajarshi Das, Wenhan Xiong, and Tian Gao. Do multi-hop readers dream of reasoning chains? In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, 2019a. find
  37. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. R 3 : Reinforced ranker-reader for open-domain question answering. In AAAI, 2018a. find
  38. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR, 2018b. find
  39. Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang. Multi-passage BERT: A globally normalized bert model for open-domain question answering. In EMNLP, 2019b. find
  40. Yunxuan Xiao, Yanru Qu, Lin Qiu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. Dynamically fused graph network for multi-hop reasoning. In ACL, 2019. find
  41. Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering. In ICLR, 2017. find our card
  42. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with BERTserini. In NAACL (Demonstrations), 2019. find our card
  43. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018. find our card
  44. Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. Transformer-XH: Multi-hop question answering with extra hop attention. In ICLR, 2020. find