End-to-End Open-Domain Question Answering with BERTserini
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-4013.pdf
Summary
Most QA and reading-comprehension benchmarks give the model a short passage or a list of candidate sentences. They leave aside where that text comes from. This demo paper addresses open-domain QA over all of Wikipedia by combining a classic retrieval stage with a neural reader. In BERTserini, the Lucene-based Anserini toolkit uses BM25 to fetch the top k text segments for a question. A BERT-Base model fine-tuned on SQuAD v1.1 then extracts a candidate span from each segment. The final answer is chosen by linearly interpolating the retriever and reader scores. The authors compare three indexing units (whole articles, paragraphs and sentences) from the Dec. 2016 Wikipedia dump used by Dr.QA, and evaluate on SQuAD development questions. Paragraph indexing works best and clearly beats earlier systems in exact match. Error analysis shows that retrieval is not the main bottleneck. The larger problem is ranking the extracted spans. Overall, the work shows that a simple two-stage pipeline built on off-the-shelf BERT can substantially improve QA over a large corpus while staying fast enough for a live chatbot.
Key points
- Corpus: 5.08M Wikipedia articles, also split into 29.5M paragraphs and 79.5M sentences as alternative retrieval units. Retrieval uses Anserini with BM25 and the question treated as a bag of words.
- Reader: BERT-Base (uncased, 110M parameters) fine-tuned on SQuAD v1.1. The final softmax over spans is removed so that scores can be compared across segments. The combined score is (1 − µ)·S_Anserini + µ·S_BERT, with µ = 0.5.
- Results: At similar text budgets, paragraph retrieval (k = 29) reached EM 36.6 / F1 44.0, compared with 19.1 EM for articles (k = 5) and 34.0 for sentences (k = 78). With k = 100 paragraphs the system scored EM 38.6, F1 46.1 and recall 85.8. Prior results were 27.1 EM for Dr.QA and 34.7 for MINIMAL.
- Failure analysis: Top-1 EM levels off after about k = 10, while recall keeps rising. The largest gap comes from scoring and aggregating answer spans, not from retrieval.
- Demo: Deployed as a Slackbot and on RSVP.ai's chat platform with paragraph indexing and k = 10. Average latency is about 0.5 s for Anserini retrieval on CPU plus 0.18 s for BERT on a Tesla P40 GPU.
Cited by
- Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
- Dense Passage Retrieval for Open-Domain Question Answering
- Latent Retrieval for Weakly Supervised Open Domain Question Answering
- How Context Affects Language Models' Factual Predictions
- A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
- IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
- A Robustly Optimized BERT Pre-training Approach with Post-training
Cite
@inproceedings{yang2019end,
title = {{End-to-End Open-Domain Question Answering with BERTserini}},
author = {Yang, Wei and Xie, Yuqing and Lin, Aileen and Li, Xingyu and Tan, Luchen and Xiong, Kun and Li, Ming and Lin, Jimmy},
booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)},
year = {2019},
doi = {10.18653/v1/n19-4013},
url = {https://doi.org/10.18653/v1/n19-4013}
}References
- Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A human generated MAchine Reading COmprehension dataset. arXiv:1611.09268… find our card
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report. find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find our card
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879. find our card
- Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 845–855. find our card
- Stefanie Tellex, Boris Katz, Jimmy Lin, Gregory Marton, and Aaron Fernandes. 2003. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2003), p… find
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. find
- Ellen M. Voorhees and Dawn M. Tice. 1999. The TREC-8 Question Answering Track evaluation. In Proceedings of the Eighth Text REtrieval Conference (TREC-8), pages 83–106. find
- Bernhard Kratzwald and Stefan Feuerriegel. 2018. Adaptive document retrieval for deep question answering. In Proceedings of the 2018 Conference on Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2017. R 3 : Reinforced… find
- Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of Lucene for information retrieval research. In Proceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), pages 1253–1256. find
- Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible ranking baselines using Lucene. Journal of Data and Information Quality, 10(4):Article 16. find
- Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018. find our card
- Xuchen Yao, Benjamin Van Durme, Chris Callison-burch, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 858–867… find