Okapi at TREC-3
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "author's copy (Microsoft Research) — read only" does not allow redistributing the text itself. Original — microsoft.com/en-us/research/wp-content/uploads/2016/02/okapi trec3.pdf
Summary
This paper reports City University's work with the Okapi retrieval system at TREC-3. Okapi was built on the classical Robertson–Sparck Jones probabilistic model. The team wanted to make it competitive again after topics lost their CONCEPTS fields, which had been the richest source of query terms. They introduced BM25, a weighting function that merges the earlier BM11 and BM15 schemes. BM25 accounts for term frequency in the document and in the query, and it normalises for document length through a tunable parameter b. The team also tested several other ideas: picking the best-matching passages at search time, expanding ad hoc queries with terms from the top documents of a trial search, and choosing routing terms one at a time by checking whether each term improves retrieval on training data. Experiments ran on TREC disks 1–3. Query expansion and stepwise term selection gave the clearest gains, while passages and the new weighting each helped a little. Together these changes produced one of the strongest automatic ad hoc runs. They also showed that a system grounded in a single theoretical model can compete with the best systems of the time.
Key points
- BM25(k1, k2, k3, b) blends BM11 (b = 1) and BM15 (b = 0); b ≈ 0.75 was the usual setting, often paired with a larger k1.
- Official automatic ad hoc run citya1 (BM25, passages and expansion): average precision 0.401 and P@5 0.740. The baseline BM25 run citya2 scored 0.337.
- Development used disks 1 & 2 with topics 101–150. The official tests were ad hoc topics 151–200 and routing on disk 3.
- Alternative term-ranking measures for feedback (RSV2, ARSV, the r criterion) did no better than the standard RSV. Stepwise selection of terms was costly to compute but very effective for routing.
- The official interactive run cityi1 reached only 0.250 average precision, well below the automatic runs. The authors warn that the evaluation method may partly explain this gap.
Cited by
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Unsupervised Dense Information Retrieval with Contrastive Learning
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- Document Expansion by Query Prediction
- LaMP: When Large Language Models Meet Personalization
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
- RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
- Toolformer: Language Models Can Teach Themselves to Use Tools
- A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
- Findings of the VarDial Evaluation Campaign 2017
- Large Language Models for Information Retrieval: A Survey
Cite
@inproceedings{robertson1994okapi,
title = {{Okapi at TREC-3}},
author = {Robertson, Stephen E. and Walker, Steve and Jones, Susan and Hancock-Beaulieu, Micheline and Gatford, Mike},
booktitle = {TREC-3 (NIST SP 500-225)},
year = {1994},
url = {https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/okapi_trec3.pdf}
}References
- The First Text REtrieval Conference (TREC{1) . Edited by D K Harman. Gaithersburg, MD: NIST, 1993. find
- Robertson S E et al . Okapi at TREC. In: [1]. p21{ 30. find
- The Second Text REtrieval Conference (TREC{2) . Edited by D K Harman. Gaithersburg, MD: NIST, 1994. find
- Robertson S E et al . Okapi at TREC{2. In: [3]. p21{34. find
- Robertson S E and Walker S. Some simple e ective approximations to the 2{Poisson model for probabilistic weighted retrieval. In: Croft W B and van Rijsbergen C J (eds). Proceedings of the 17th Annual International ACM{SIGIR Conference on Research and Development in Information Retrieval, Dublin 1994… find
- Robertson S E, Walker S and Hancock-Beaulieu, M. Large test collection experiments on an operational, interactive system: Okapi at TREC. Information Processing and Management (to appear). find
- Robertson S E and Sparck Jones K. Relevance weighting of search terms. Journal of the American Society for Information Science 27 May{June 1976 p129{146. find
- Robertson S E. On term selection for query expansion. Journal of Documentation 46 Dec 1990 p359{ 364. find
- Efthimiadis E N. A user-centred evaluation of ranking algorithms for interactive query expansion. SI-GIR Forum (USA), spec. issue. , 1993, p146{59, (Paper given at: Sixteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Pittsburgh, PA, USA, 27 June-… find
- Callan J P. Passage-level evidence in document retrieval. In: Croft W B and van Rijsbergen C J (eds). Proceedings of the 17th Annual International ACM{SIGIR Conference on Research and Development in Information Retrieval, Dublin 1994. Springer-Verlag 1994. (p302{310) find
- Evans D and Le erts R. Design and evaluation of the CLARIT{TREC{2 system. In: [3]. p137{150. find