Elamur Research⌕
☰
home / papers / 2020 / Open-Domain Question Answering

Open-Domain Question Answering

Paper · 2020 · Danqi Chen and Wen-tau Yih

Summary

This is an ACL 2020 tutorial abstract, not an experimental paper. It reports no new results. Its contribution is a structured overview of open-domain QA, the task of answering arbitrary questions using a very large document collection that covers many topics.

The authors trace how the field moved from heavily modular pipelines (early systems, TREC competitions, IBM's DeepQA) to neural systems. These either combine IR with reading comprehension or are trained fully end to end. They also review datasets, evaluation metrics and benchmarks, and explain how reading-comprehension datasets should be reinterpreted for the open setting.

The core of the tutorial groups recent models into three families:

  • two-stage retriever-reader pipelines;
  • learned dense retrievers with joint training;
  • retriever-free methods that rely only on knowledge stored inside pre-trained models.

It further covers combining text with knowledge bases. It closes with open questions about real progress, current limitations and balancing efficiency against accuracy.

Its value is a coherent map of a fast-moving area, with materials released publicly.

Key points

  • Models are split into three categories: two-stage retriever-reader (pioneered by Chen et al., 2017), dense retriever with end-to-end training, and retriever-free approaches.
  • In classic retriever-reader systems, retrieval uses sparse methods such as TF-IDF or BM25, and a neural reading comprehension model extracts the answer. Related techniques include multi-passage training, passage reranking and denoising distantly supervised data.
  • Dense retrieval must search spaces of millions to billions of documents. Three strategies for handling this are reviewed: hybrid dense-sparse phrase indexing (Seo et al., 2019), carefully designed learning algorithms (Karpukhin et al., 2020) and dedicated pre-training schemes (Guu et al., 2020; Lee et al., 2019).
  • Hybrid text-plus-KB QA is discussed using Freebase and Wikidata. KBs can either guide the retriever or reader, or be fused with text into one model.
  • The tutorial lasts 3.5 hours, including a 30-minute break. Materials are at github.com/danqi/acl2020-openqa-tutorial.

Interesting ideas

  • I1. Classify by retrieval role. Grouping systems by how they retrieve (non-learned, learned dense, or no retrieval at all) is a clean way to organize any survey or system design. The same lens could structure comparisons of search-based tools for Ukrainian NLP or trade-data lookup.
  • I2. Pre-trained model as a baseline. Treating a large pre-trained model as an implicit knowledge base, queried zero-shot or after fine-tuning on QA pairs, gives a cheap baseline with no document index. It shows how much a model already "knows" before retrieval infrastructure is built.
  • I3. Mix sparse and dense signals. Combining sparse lexical matching with dense representations, as in the hybrid phrase-indexing line of work, hedges against weaknesses of each. This is useful for morphologically rich languages, where exact-match and semantic matching often disagree.
  • I4. Clean distant supervision. Labels created by matching answers to retrieved text are noisy, so the tutorial highlights denoising them explicitly. This applies to any project that builds training data automatically rather than by hand.
  • I5. Ask about real progress and cost. The tutorial asks how far current systems have actually advanced over older ones, and frames compute time and memory against accuracy. This is a healthy evaluation habit for any project choosing between heavy and lightweight models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works Information Systems · 523 works

🧭 Topics 1 Question answering · 242 works

Cite

Cite as Danqi Chen and Wen-tau Yih. 2020. Open-Domain Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-tutorials.8

@inproceedings{chen2020open,
  title = {{Open-Domain Question Answering}},
  author = {Danqi Chen and Wen-tau Yih},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts},
  year = {2020},
  doi = {10.18653/v1/2020.acl-tutorials.8},
  url = {https://doi.org/10.18653/v1/2020.acl-tutorials.8}
}

References

31 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909. find
  2. Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations (ICLR).
  3. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL), pages 1601–1611.
  4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP), pages 1533– 1544.
  5. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
  6. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In ACM SIGMOD international conference on Management of data, pages 1247– 1250. find
  7. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Change, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  8. Eric Brill, Susan Dumais, and Michele Banko. 2002. An analysis of the AskMSR question-answering system. In Empirical Methods in Natural Language Processing (EMNLP), pages 257–264. find
  9. Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University. find
  10. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Association for Computational Linguistics (ACL), pages 6086–6096.
  11. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL), pages 1870–1879.
  12. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Association for Computational Linguistics (ACL), pages 1736–1745.
  13. Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Empirical Methods in Natural Language Processing (EMNLP), pages 4231–4242. find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  15. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. A discrete hard EM approach for weakly supervised question answering. In Empirical Methods in Natural Language Processing (EMNLP).
  16. Ellen M Voorhees et al. 1999. The TREC-8 question answering track report. In Trec, volume 99, pages 77–82. Citeseer.
  17. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledge base. find
  18. Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Knowledge guided text retrieval and reading for open domain question answering. arXiv preprint arXiv:1911.03868.
  19. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2018. Rˆ3: Reinforced reader-ranker for open-domain question answering. In Conference on Artificial Intelligence (AAAI). find
  20. Dan Moldovan, Sanda Harabagiu, Marius Pasca, Rada Mihalcea, Roxana Girju, Richard Goodrum, and Vasile Rus. 2000. The structure and performance of an open-domain question answering system. In Association for Computational Linguistics (ACL), pages 563–570. find
  21. Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage bert: A globally normalized bert model for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP), pages 5881– 5885.
  22. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085.
  23. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  24. Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Improving question answering over incomplete kbs with knowledge-aware reader. In Association for Computational Linguistics (ACL), pages 4258–4264. find
  25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. find
  26. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. In North American Association for Computational Linguistics (NAACL), pages 72–77.
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392.
  29. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910. find
  30. Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. In Association for Computational Linguistics (ACL), pages 4430–4441.
  31. Haitian Sun, Tania Bedrax-Weiss, and William W Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Empirical Methods in Natural Language Processing (EMNLP). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/