Latent Retrieval for Weakly Supervised Open Domain Question Answering
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1612.pdf
Summary
Open-domain question answering systems usually depend on a fixed IR engine such as TF-IDF or BM25 to find candidate passages. Many also need gold evidence annotations to train the reader. The authors argue that this setup caps recall and treats QA as if it were ordinary search. They present ORQA, a BERT-based model whose retriever and reader are trained together, end-to-end, from question–answer string pairs only. The choice of evidence from the whole English Wikipedia is treated as a latent variable. Training such a retriever from scratch would not work, so they first pre-train it with an unsupervised Inverse Cloze Task (ICT): the model sees a sentence and must pick out the passage it came from among other passages in the batch. ORQA was evaluated on open versions of five QA benchmarks. BM25 is still hard to beat when the question writers already knew the answer. When questions come from people who actually wanted the answer, learned retrieval helps a lot. The paper shows that retrieval can be learned for QA without any external search engine.
Key points
- The evidence corpus is English Wikipedia split into more than 13 million blocks. Retrieval scores are inner products of 128-dimensional question and block encodings, and the reader scores answer spans with an MLP.
- In ICT pre-training, the pseudo-question sentence is removed from its context 90% of the time. Always removing it costs almost 10 points, and never removing it costs about 6 points, which gives results close to BM25.
- On Natural Questions, WebQuestions and CuratedTrec, ORQA beats BM25 by 6 to 19 points of exact match.
- On SQuAD and TriviaQA, where question writers knew the answer, BM25 stays the stronger retriever.
- ORQA handles passages that share many words but differ in meaning better than BM25. However, its compact vectors struggle with very specific concepts, so the authors suggest hybrid sparse/dense retrieval as future work.
Cited by
- Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
- Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- Ragas: Automated Evaluation of Retrieval Augmented Generation
- Precise Zero-Shot Dense Retrieval without Relevance Labels
- Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
- Enabling Large Language Models to Generate Text with Citations
- RARR: Researching and Revising What Language Models Say, Using Language Models
- Injecting Numerical Reasoning Skills into Language Models
- AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Unsupervised Dense Information Retrieval with Contrastive Learning
- Evaluating Open-Domain Question Answering in the Era of Large Language Models
- Large Language Models Struggle to Learn Long-Tail Knowledge
- Dense Passage Retrieval for Open-Domain Question Answering
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
- Hurdles to Progress in Long-form Question Answering
- PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Lost in the Middle: How Language Models Use Long Contexts
- Sparse, Dense, and Attentional Representations for Text Retrieval
- Generation-Augmented Retrieval for Open-domain Question Answering
- Augmented Language Models: a Survey
- Text and Code Embeddings by Contrastive Pre-Training
- How Context Affects Language Models' Factual Predictions
- KILT: a Benchmark for Knowledge Intensive Language Tasks
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
- Learning To Retrieve Prompts for In-Context Learning
- Improving Passage Retrieval with Zero-Shot Question Generation
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
- ASQA: Factoid Questions Meet Long-Form Answers
- Long Range Arena: A Benchmark for Efficient Transformers
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Generate rather than Retrieve: Large Language Models are Strong Context Generators
- CREPE: Open-Domain Question Answering with False Presuppositions
- Big Bird: Transformers for Longer Sequences
- A Survey of Large Language Models
Cite
@inproceedings{lee2019latent,
title = {{Latent Retrieval for Weakly Supervised Open Domain Question Answering}},
author = {Lee, Kenton and Chang, Ming-Wei and Toutanova, Kristina},
booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
year = {2019},
doi = {10.18653/v1/p19-1612},
eprint = {1906.00300},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/p19-1612}
}References
- Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1(1):49–62. find our card
- Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1601–1611. find our card
- Petr Baudis and Jan Sedivý. 2015. Modeling of the question answering task in the yodaqa system. In CLEF. find
- Bernhard Kratzwald and Stefan Feuerriegel. 2018. Adaptive document retrieval for deep question answering. arXiv preprint arXiv:1808.06528. find
- Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155. find
- Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1545–1556. find
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544. find
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1870–1879. find our card
- James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world’s response. In Proceedings of the fourteenth conference on computational natural language learning, pages 18–27. Association for Computational Linguistics. find
- Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. In International Conference on Learning Representations. find
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
- Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904. — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) find
- Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. find our card
- Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1156– 1165. ACM. find
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Lingu… find our card
- Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. 2018. Ranking paragraphs for improving answer recall in open-domain question answering. arXiv preprint arXiv:1810.00494. find
- Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
- Percy Liang, Michael I Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446. find
- Jimmy Lin. 2019. The neural hype and comparisons against weak baselines. In ACM SIGIR Forum. find
- Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803.02893. find our card
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
- Karthik Narasimhan, Adam Yala, and Regina Barzilay. 2016. Improving information extraction by acquiring external evidence with reinforcement learning. arXiv preprint arXiv:1603.07954. find
- Christian S Perone, Roberto Silveira, and Thomas S Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. arXiv preprint arXiv:1806.06259. — [arXiv:1806.06259](https://arxiv.org/abs/1806.06259) find
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL. find
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find our card
- Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. find our card
- Amit Singh. 2012. Entity based q&a retrieval. In Proceedings of the 2012 Joint conference on empirical methods in natural language processing and computational natural language learning, pages 1266– 1277. Association for Computational Linguistics. find
- Wilson L Taylor. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. find
- Ellen M Voorhees. 2001. Overview of the trec 2001 question answering track. In In Proceedings of the Tenth Text REtrieval Conference (TREC. Citeseer. find
- Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R 3: Reinforced ranker-reader for open-domain question answering. In Thirty-Second AAAI Conference on Artificial Intelligence. find
- Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. ACM. find
- Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718. find our card