Elamur Research⌕
☰
home / papers / 2023 / RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

Paper · 2023 · Ronak Pradeep, Sahel Sharifymoghaddam and Jimmy Lin

Summary

LLM-based listwise reranking has so far relied on proprietary models such as GPT 3.5 and GPT 4 accessed through APIs. This makes results hard to reproduce and non-deterministic, and those models sometimes return malformed output. The authors present RankVicuna, which they describe as the first fully open-source LLM for zero-shot listwise document reranking. They fine-tuned a 7B Vicuna model to imitate the passage orderings produced by RankGPT 3.5 on 100K MS MARCO v1 training queries, each with 20 BM25 candidates. No human relevance labels were used, so the setting still counts as zero-shot. Malformed teacher outputs were removed, and training was augmented with shuffled candidate orders. When evaluated on the Deep Learning Track test sets from TREC 2019 and TREC 2020, RankVicuna performs on par with RankGPT 3.5 and falls just slightly short of GPT 4. It is also deterministic and always produces valid rankings. The model and the rank_llm code are publicly released, which gives researchers a reproducible baseline for LLM-based reranking.

Key points

  • The training data comes from RankGPT 3.5 orderings of BM25 top-20 candidates, retrieved with Pyserini for 100K sampled MS MARCO v1 queries. About 12% of the teacher outputs were malformed and were excluded.
  • When reranking the top 100 candidates, nDCG@10 reaches 0.6682 (DL19) and 0.6549 (DL20) with BM25 as the first stage. With SPLADE++ ED as the first stage, it rises to 0.7459 and 0.7473.
  • Data augmentation with shuffled inputs creates an effectiveness–robustness tradeoff:
    • Without it, the model scores somewhat higher on SPLADE++ ED candidates.
    • However, on shuffled candidates it loses up to 34% effectiveness and shows higher variance across runs.
  • With a strong first-stage retriever, a single pass reranking only the top 20 candidates gives gains close to reranking the top 100.
  • RankVicuna consistently beats PRPVicuna, a pairwise sliding-window baseline, and converges within a few passes. One pass takes about 30 s per query on an RTX A6000. Code is available at github.com/castorini/rank_llm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada80 works · 15,030 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksInformation retrieval & search · 552 works

Cite

Cite as Ronak Pradeep, Sahel Sharifymoghaddam and Jimmy Lin. 2023. RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models. arXiv:2309.15088.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.15088 · arXiv 2309.15088

@misc{pradeep2023rankvicuna,
  title = {{RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models}},
  author = {Ronak Pradeep, Sahel Sharifymoghaddam and Jimmy Lin},
  year = {2023},
  doi = {10.48550/arxiv.2309.15088},
  eprint = {2309.15088},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.15088}
}

References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nasreen Abdul-Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Donald Metzler, Mark D. Smucker, Trevor Strohman, Howard Turtle, and Courtney Wade. 2004. UMass at TREC 2004: Novelty and HARD. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), Gaithersb… find
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv:arXiv:1611…
  3. Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022), pages 2387–2392, Madrid, Spai… find
  4. Leonid Boytsov, Preksha Patel, Vivek Sourabh, Riddhi Nisar, Sayani Kundu, Ramya Ramanathan, and Eric Nyberg. 2023. InPars-Light: Cost-effective unsupervised training of efficient rankers. arXiv:2301.02998. — [arXiv:2301.02998](https://arxiv.org/abs/2301.02998) find
  5. B. Barla Cambazoglu, Hugo Zaragoza, Olivier Chapelle, Jiang Chen, Ciya Liao, Zhaohui Zheng, and Jon Degenhardt. 2010. Early exit optimizations for additive machine learned ranking systems. In Proceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM 2010), pages 411–… find
  6. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. arXiv:2102.07662. — [arXiv:2102.07662](https://arxiv.org/abs/2102.07662) find
  7. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 deep learning track. arXiv:2003.07820.
  8. Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples. arXiv:2209.11755.
  9. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv:2109.10086.
  10. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022. From distillation to hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2… find
  11. Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. Rethink training of BERT rerankers in multi-stage retrieval pipeline. In Proceedings of the 43rd European Conference on Information Retrieval (ECIR 2021). find
  12. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada.
  13. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval…
  14. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv:2112.09118.
  15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  16. Jimmy Lin. 2021. A proposed conceptual framework for a representational approach to information retrieval. arXiv:2110.01529. — [arXiv:2110.01529](https://arxiv.org/abs/2110.01529) find
  17. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021a. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and… find
  18. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021b. Pretrained Transformers for Text Ranking: BERT and Beyond. Morgan & Claypool Publishers. find
  19. Jimmy Lin, Ronak Pradeep, Tommaso Teofili, and Jasper Xian. 2023. Vector search with OpenAI embeddings: Lucene is all you need. arXiv:2308.14963. — [arXiv:2308.14963](https://arxiv.org/abs/2308.14963) find
  20. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. arXiv:2305.02156.
  21. Irina Matveeva, Chris Burges, Timo Burkard, Andy Lau-cius, and Leon Wong. 2006. High accuracy retrieval with multiple nested ranker. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), pages 437–444, Seattle, Washing… find
  22. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelei… — Text and Code Embeddings by Contrastive Pre-Training
  23. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv:1901.04085.
  24. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718. find
  25. Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. arXiv:1910.14424. find
  26. Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020. Beyond [CLS] through ranking by generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1722–1727, Online. find
  27. Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q. Tran. 2023. How does generative retrieval scale to millions of passages? arXiv:2305.11841. find
  28. Ronak Pradeep, Yilin Li, Yuetong Wang, and Jimmy Lin. 2022a. Neural query synthesis and domain-specific ranking templates for multi-stage clinical trial matching. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022), pages 23… find
  29. Ronak Pradeep, Yuqi Liu, Xinyu Zhang, Yilin Li, Andrew Yates, and Jimmy Lin. 2022b. Squeezing water from a stone: A bag of tricks for further improving cross-encoder effectiveness for reranking. In Proceedings of the 44th European Conference on Information Retrieval (ECIR 2022), Part I, pages 655–67… find
  30. Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. arXiv:2101.05667. — [arXiv:2101.05667](https://arxiv.org/abs/2101.05667) find
  31. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2023. Large language models are effective text rankers with pairwise ranking prompting. arXiv:2306.17563.
  32. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. find
  33. Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. 2023. Questions are all you need to train a dense passage retriever. Transactions of the Association for Computational Linguistics, 11:600–616.
  34. Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? Investigating large language models as re-ranking agent. arXiv:2304.09542.
  35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  36. Lidan Wang, Jimmy Lin, and Donald Metzler. 2011. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011), pages 105–114, Beijing, China. find
  37. Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2022. RankT5: Fine-tuning T5 for text ranking with ranking losses. arXiv:2210.10634. — [arXiv:2210.10634](https://arxiv.org/abs/2210.10634) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pradeep-2023-rankvicuna-zero-shot-listwise-document/