Elamur Research⌕
☰
home / papers / 2023 / Large Language Models are Strong Zero-Shot Retriever

Large Language Models are Strong Zero-Shot Retriever

Paper · 2023 · Tao Shen, Guodong Long, Xiubo Geng et al.

Summary

Zero-shot first-stage retrieval, where only a raw document collection is available and no labeled query–document pairs exist, usually trails supervised systems. Earlier LLM-based query expansion such as HyDE asks a model to write hypothetical answers to short, ambiguous queries. Those answers are often off-domain, and the self-supervised dense retriever paired with them limits the gains. The authors propose LameR, which uses only a frozen LLM and BM25. BM25 first retrieves top passages for the query. These passages, whether relevant or not, go into the prompt so the LLM can imitate, correct or summarize them while learning the collection's style. The query plus the generated answers is then sent back to BM25. Because lexical matching reads the LLM output literally, no learned retriever caps performance. On TREC Deep Learning 2019/2020, LameR beats other zero-shot methods and rivals or exceeds fully supervised retrievers. It also runs faster and needs a smaller index than HyDE. This suggests strong retrieval is possible without training data or neural retrievers.

Key points

  • Pilot experiments showed that HyDE improves Contriever about twice as much as it improves BM25. They also showed that stronger LLMs (text-davinci-003, gpt-3.5-turbo) barely help a Contriever-based pipeline.
  • With BM25, LameR raises nDCG@10 from 50.6 to 69.1 on DL19 and from 48.0 to 64.8 on DL20. Applied to Contriever, it yields similar gains of +19.8 and +16.1.
  • Using GPT-4 on DL20, LameR reaches 65.9 nDCG@10, ahead of HyDE (57.9) and supervised DPR (65.3).
  • Combined with the SoTA dense retriever SimLM, LameR reaches 76.5 on DL19 and 75.8 on DL20. These gains exceed few-shot Q2D by 3.6 and 4.2 points.
  • A second round of iterative augmentation lowered nDCG@10 by 1.0. The main limitations are LLM inference cost and sensitivity to prompt wording.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInformation retrieval & search · 473 works

Cite

Cite as Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Tianyi Zhou and Daxin Jiang. 2023. Large Language Models are Strong Zero-Shot Retriever. arXiv:2304.14233.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.14233 · arXiv 2304.14233

@misc{shen2023large,
  title = {{Large Language Models are Strong Zero-Shot Retriever}},
  author = {Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Tianyi Zhou and Daxin Jiang},
  year = {2023},
  doi = {10.48550/arxiv.2304.14233},
  eprint = {2304.14233},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.14233}
}

References

45 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Timo Schick, Patrick S. H. Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. CoRR, abs/2211.09260, 2022. doi: 10.48550/arXiv.2211.09260. URL https://doi.org/10.48550/ arXiv.2211.09260.
  2. Leonid Boytsov, Preksha Patel, Vivek Sourabh, Riddhi Nisar, Sayani Kundu, Ramya Ramanathan, and Eric Nyberg. Inpars-light: Cost-effective unsupervised training of efficient rankers. CoRR, abs/2301.02998, 2023. doi: 10.48550/arXiv.2301.02998. URL https://doi.org/10.48550/arXiv.2301.02998. — [doi:10.48550/arxiv.2301.02998](https://doi.org/10.48550/arxiv.2301.02998) ↗
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Yinqiong Cai, Yixing Fan, Jiafeng Guo, Fei Sun, Ruqing Zhang, and Xueqi Cheng. Semantic models for the first-stage retrieval: A comprehensive review. CoRR, abs/2103.04831, 2021. URL https://arxiv.org/abs/2103.04831. — [arXiv:2103.04831](https://arxiv.org/abs/2103.04831) ↗
  5. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1…
  6. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the TREC 2019 deep learning track. CoRR, abs/2003.07820, 2020. URL https://arxiv.org/abs/2003.07820. ↗
  7. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the TREC 2020 deep learning track. CoRR, abs/2102.07662, 2021. URL https://arxiv.org/abs/ 2102.07662. — [arXiv:2102.07662](https://arxiv.org/abs/2102.07662) ↗
  8. Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. CoRR, abs/2209.11755, 2022. doi: 10.48550/arXiv.2209.11755. URL https://doi.org/10.48550/arXiv.2209.11755. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Co…
  10. Dheeru Dua, Emma Strubell, Sameer Singh, and Pat Verga. To adapt or to annotate: Challenges and interventions for domain adaptation in open-domain question answering. CoRR, abs/2212.10381, 2022. doi: 10.48550/arXiv.2212.10381. URL https://doi.org/10.48550/ arXiv.2212.10381. find
  11. Luyu Gao and Jamie Callan. Long document re-ranking with modular re-ranker. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, editors, SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval,… — [doi:10.1145/3477495.3531860](https://doi.org/10.1145/3477495.3531860) find
  12. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. CoRR, abs/2212.10496, 2022. doi: 10.48550/arXiv.2212.10496. URL https://doi.org/10.48550/arXiv.2212.10496. ↗
  13. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: retrieval-augmented language model pre-training. CoRR, abs/2002.08909, 2020. URL https://arxiv.org/abs/2002.08909. ↗
  14. Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. CoRR, abs/2301.00303, 2023. doi: 10.48550/arXiv.2301.00303. URL https://doi.org/10.48550/arXiv.2301.00303. ↗
  15. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Towards unsupervised dense information retrieval with contrastive learning. CoRR, abs/2112.09118, 2021. URL https://arxiv.org/abs/2112.09118. ↗
  16. Vitor Jeronymo, Luiz Henrique Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto de Alencar Lotufo, Jakub Zavrel, and Rodrigo Frassetto Nogueira. Inpars-v2: Large language models as efficient dataset generators for information retrieval. CoRR, abs/2301.01820, 2023. doi: 10.48550/arXiv.2301.01820. URL… — [doi:10.48550/arxiv.2301.01820](https://doi.org/10.48550/arxiv.2301.01820) find
  17. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods…
  18. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, Jul…
  19. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Develo… find
  20. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Eneko Agirre, Marianna Apidianaki, and Ivan Vulic, editors, Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for De…
  21. Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. Z-ICL: zero-shot in-context learning with pseudo-demonstrations. CoRR, abs/2212.09865, 2022. doi: 10.48550/arXiv.2212.09865. URL https://doi.org/10.48550/arXiv.2212.09865. ↗
  22. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2…
  23. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods i…
  24. Niklas Muennighoff. SGPT: GPT sentence embeddings for semantic search. abs/2202.08904, 2022. URL https://arxiv.org/abs/2202.08904. CoRR, ↗
  25. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d’Avila Garcez, and Greg Wayne, editors, Proceedings of the Workshop on Cognitive Comp…
  26. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. CoRR, abs/2112.07899, 2021. URL https://arxiv.org/abs/2112. 07899. ↗
  27. Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emir…
  28. Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389, 2009. doi: 10.1561/1500000019. URL https://doi.org/10.1561/1500000019. ↗
  29. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguist…
  30. Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md. Arafat Sultan, and Christopher Potts. UDAPDR: unsupervised domain adaptation via LLM prompting and distillation of rerankers. CoRR, abs/2303.00807, 2023. doi: 10.48550/arXiv.2303.00807. URL htt… find
  31. Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. Lexmae: Lexicon-bottlenecked pretraining for large-scale retrieval. CoRR, abs/2208.14754, 2022. doi: 10.48550/arXiv.2208.14754. URL https://doi.org/10. 48550/arXiv.2208.14754. — [doi:10.48550/arxiv.2208.14754](https://doi.org/10.48550/arxiv.2208.14754) find
  32. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Ev…
  33. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  34. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663, 2021. URL https://arxiv.org/abs/2104.08663. ↗
  35. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302…
  36. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. CoRR, abs/2212.10509, 2022. doi: 10.48550/arXiv.2212.10509. URL https://doi.org/10.48550/ arXiv.2212.10509.
  37. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.03533, 2022. doi: 10.48550/arXiv.2212.03533. URL https://doi.org/10. 48550/arXiv.2212.03533.
  38. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval. CoRR, abs/2207.02578, 2022. doi: 10.48550/arXiv.2207.02578. URL https://doi.org/10.48550/arXiv.2207.02578. ↗
  39. Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. CoRR, abs/2303.07678, 2023. doi: 10.48550/arXiv.2303.07678. URL https://doi.org/10.48550/arXiv.2303.07678. ↗
  40. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/foru… ↗
  41. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-…
  42. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. CoRR, abs/2209.10063, 2022. doi: 10.48550/arXiv.2209.10063. URL https://doi.org/10.48550/arXiv.… ↗
  43. Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. Knowledge-grounded dialogue generation with pre-trained language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EM…
  44. Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Lan Luo, Ke Zhan, Enrui Hu, Xinyu Zhang, Hao Jiang, Zhao Cao, Fan Yu, Xin Jiang, Qun Liu, and Lei Chen. Hyperlink-induced pre-training for passage retrieval in open-domain question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editor… find
  45. Yucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Guodong Long, Binxing Jiao, and Daxin Jiang. Towards robust ranker for text retrieval. CoRR, abs/2206.08063, 2022. doi: 10.48550/arXiv.2206.08063. URL https://doi.org/10.48550/arXiv.2206.08063. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-strong-zero/