Relevance-guided Supervision for OpenQA with ColBERT
Paper · 2021 · Omar Khattab, Christopher Potts and Matei Zaharia
Summary
Open-domain question answering (OpenQA) systems pair a retriever with a reader. Learned retrievers such as DPR, ORQA and REALM compress each passage and question into one vector. The authors argue this is too coarse for complex questions, and that current supervision is costly or brittle. Some methods need hand-labeled positives, some rely on BM25 sampling, and some re-index the corpus repeatedly or freeze the document encoder.
The paper presents ColBERT-QA, which adapts the ColBERT late-interaction retriever to OpenQA. ColBERT compares every question token embedding with every passage token embedding through MaxSim, yet still scales to millions of passages. Training uses relevance-guided supervision (RGS). A weak retriever, initially BM25, fetches top-k passages for each training question. A simple heuristic labels them as positives or negatives, and these labels train a stronger retriever. The cycle runs two to three times, needing only one or two corpus re-indexings and keeping the document encoder trainable.
On Natural Questions, TriviaQA and SQuAD, ColBERT-QA achieves the best retrieval and extractive OpenQA results reported at the time. This shows that richer interaction and cheap self-generated supervision both pay off.
Key points
-
RGS raises Success@20 by up to 2.3 points for ColBERT and up to 3.2 points for a single-vector retriever. Overall, ColBERT-QA beats prior retrievers by more than five Success@20 points.
-
The third round, ColBERT-QA 3, is significantly better than round 1 under a Wilcoxon signed-rank test with Bonferroni correction. Rounds 2 and 3 show no significant difference.
-
End-to-end exact match with a BERT-large (WWM) reader:
- NQ: 47.8
- TriviaQA: 70.1
- SQuAD: 54.7 on the Dec 2018 Wikipedia dump and 58.7 on the 2016 dump
These results surpass DPR, REALM and the generative RAG (BART-large).
-
With a BERT-base reader, ColBERT-QA scores 42.3 EM on NQ, 64.6 on TriviaQA and 47.7 on SQuAD. It beats all BERT-base systems and also outperforms RAG on TriviaQA.
-
Code is released at github.com/stanfordnlp/ColBERT-QA.
Institutions
United States📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works Information Systems · 523 works
🧭 Topics 1 Information retrieval & search · 145 works
Cite
Cite as Omar Khattab, Christopher Potts and Matei Zaharia. 2021. Relevance-guided Supervision for OpenQA with ColBERT. Transactions of the Association for Computational Linguistics.
@article{khattab2021relevance,
title = {{Relevance-guided Supervision for OpenQA with ColBERT}},
author = {Omar Khattab, Christopher Potts and Matei Zaharia},
journal = {Transactions of the Association for Computational Linguistics},
year = {2021},
doi = {10.1162/tacl\_a\_00405},
url = {https://doi.org/10.1162/tacl_a_00405}
}References
- Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr… find
- Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over wikipedia graph for question answering. our card
- ICLR 2020. \#\# Conclusion Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA. Association for Co… our card
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/khattab-2021-relevance-guided-supervision-openqa-colbert/