Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "author's copy (University of Waterloo) — read only" does not allow redistributing the text itself. Original — cormack.uwaterloo.ca/cormacksigir09-rrf.pdf

Summary

The paper addresses how to merge ranked document lists from several retrieval systems without any training data. The authors propose Reciprocal Rank Fusion (RRF). Each document receives the sum of 1/(k + rank) over all input rankings, and documents are then sorted by that total. The constant k was set to 60 in a pilot study and never retuned. Under this formula, top positions count most, but lower positions still contribute something. RRF was compared with Condorcet Fuse, CombMNZ and the strongest single system on two kinds of input: rankings from Wumpus Search configurations and official TREC submissions. It was also used to fuse the baseline learning-to-rank outputs distributed with LETOR 3. RRF beat Condorcet in every TREC-based test and usually beat CombMNZ and the top individual run, with an average improvement of about 4–5%. On LETOR 3, the fused meta-learner outperformed every individual rank-learning method. The authors report it as the best published result on that dataset. Because the method uses only ranks, it needs no score calibration, no voting procedure and no training.

Key points

• On TREC topics 351–400, fusing 30 Wumpus Search configurations gave a MAP of 0.2145 at k = 60. The best single system scored 0.2016, Condorcet 0.2074 and CombMNZ 0.2039. Results changed little for k between roughly 30 and 100.
• RRF was used to combine runs submitted to the ad hoc tasks of TREC 3, 5 and 9, as well as to the 2004 Robust track of TREC. It exceeded Condorcet everywhere and lost to CombMNZ only on TREC 3. Its only loss to the best individual run was on TREC 9, where that run involved a human in the loop.
• Sign tests showed the gains were significant: RRF beat Condorcet in 7 of 7 cases (p ≈ 0.008).
• On LETOR 3 (583,850 document-query pairs), fusing ListNet, AdaRank-MAP, RankSVM, RankBoost and LGD reached a MAP of 0.6051. The best individual method, ListNet, scored 0.5846. CombMNZ scored 0.6107, slightly higher than RRF, but the difference was not significant (p ≈ 0.2).
• The authors suggest RRF exploits diversity among systems better than Condorcet: a few strong votes for a document can outweigh many weak preferences against it.