Elamur Research⌕
☰
home / papers / 2019 / Answering Complex Open-domain Questions Through Iterative Query Generation

Answering Complex Open-domain Questions Through Iterative Query Generation

Paper · 2019 · Peng Qi, Xiaowen Lin, Leo Mehr et al.

Summary

Standard open-domain QA systems run one retrieval step using the question and then read the results. This fails for multi-hop questions, where a key entity, such as the author of a book, never appears in the question and cannot be found directly. The authors propose GoldEn (Gold Entity) Retriever. It alternates between reading the retrieved context and writing a new natural-language search query, which is then sent to an ordinary off-the-shelf IR engine. This replaces costly, opaque neural retrievers. Each query generator selects a span from the question plus the context gathered so far. It is trained on oracle queries derived from the gold supporting documents, so little extra supervision is needed. A QA module built on BiDAF++ reads the documents collected over two hops. On the fullwiki (open-domain) setting of HotpotQA, the system beats the best previously published model without using pretrained language models such as BERT. It also improves recall of gold evidence, and its readable queries let humans inspect and correct the reasoning process.

Key points

  • The evaluation uses the HotpotQA fullwiki setting rather than the few-document setting with distractors. The authors argue the latter can be gamed and is unrealistic.
  • With 10 retrieved documents, the generated queries raise recall of the second gold document (d2) from 36.91% to 61.01% compared with question-only retrieval. Recall of the first document (d1) improves only slightly, from 87.85% to 89.91%.
  • On non-comparison questions, d2 recall rises from 27.88% to 53.23%. This roughly doubles the number of questions for which the complete gold context is available.
  • Oracle substitution experiments show that most errors come from the second-hop generator (G2), which must pick spans from a longer context. Typical failures include choosing entity spans that are too short and picking the wrong entity.
  • Restricting queries to span selection and using a fixed number of hops are acknowledged limitations. Code and models are released at github.com/qipeng/golden-retriever.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States310 works · 284,480 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Information Systems · 523 works

🧭 Topics 2 Question answering · 242 works Information retrieval & search · 145 works

Cite

Cite as Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang and Christopher D. Manning. 2019. Answering Complex Open-domain Questions Through Iterative Query Generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1261

@inproceedings{qi2019answering,
  title = {{Answering Complex Open-domain Questions Through Iterative Query Generation}},
  author = {Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang and Christopher D. Manning},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1261},
  url = {https://doi.org/10.18653/v1/d19-1261}
}

References

30 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. 2018. Ask the right questions: Active question reformulation with reinforcement learning. In Proceedings of the International Conference on Learning Representations. find
  2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL). our card
  3. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). our card
  4. Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 55th Annual Meeting of the Association of Computational Linguistics. our card
  5. Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. In International Conference on Learning Representations. find
  6. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). our card
  8. Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Association of Computational Linguistics. find
  9. Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the Association of Computational Linguistics. find
  10. Clinton Gormley and Zachary Tong. 2015. Elasticsearch: The definitive guide: A distributed real-time search and analytics engine. O’Reilly Media, Inc. find
  11. Ahmed Hassan Awadallah, Ryen W White, Patrick Pantel, Susan T Dumais, and Yi-Min Wang. 2014. Supporting complex search tasks. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 829–838. ACM. find
  12. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. our card
  13. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR). find
  14. Bernhard Kratzwald and Stefan Feuerriegel. 2018. Adaptive document retrieval for deep question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  15. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. our card
  16. Rishabh Mehrotra, Prasanta Bhattacharya, and Emine Yilmaz. 2016. Deconstructing complex search tasks: A bayesian nonparametric approach for extracting sub-tasks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
  17. Rishabh Mehrotra and Emine Yilmaz. 2017. Extracting hierarchies of search tasks & subtasks via a bayesian nonparametric approach. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 285–294. ACM. find
  18. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the Annual Conference of the Association of Computational Linguistics. our card
  19. Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Multi-hop reading comprehension through question decomposition and rescoring. In Proceedings of the Annual Conference of the Association of Computational Linguistics. our card
  20. Kyosuke Nishida, Itsumi Saito, Atsushi Otsuka, Hisako Asano, and Junji Tomita. 2018. Retrieve-and-read: Multi-task learning of information retrieval and reading comprehension. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 647–656. ACM. find
  21. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. find
  22. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. arXiv preprint arXiv:1904.08375. our card
  23. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. our card
  24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). our card
  25. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109. our card
  26. Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, pages 1057–1063. find
  27. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). our card
  28. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics, 6:287–302. our card
  29. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). our card
  30. Victor Zhong, Caiming Xiong, Nitish Shirish Keskar, and Richard Socher. 2019. Coarse-grain fine-grain coattention network for multi-evidence question answering. In Proceedings of the International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qi-2019-answering-complex-open-domain-questions/