Elamur Research
☰
home / papers / 2023 / Query2doc: Query Expansion with Large Language Models

Query2doc: Query Expansion with Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.emnlp-main.585.pdf

Summary

Search queries are often short and vague. Classic query expansion, such as RM3 pseudo-relevance feedback, has helped little on modern benchmarks. The authors propose query2doc. An LLM (OpenAI's text-davinci-003) gets a brief instruction and four example query–passage pairs from MS-MARCO, then writes a passage for the new query. This generated pseudo-document is added to the query. For BM25, the original query is first repeated five times so its terms still carry weight next to the longer generated text. For dense retrievers, the query and pseudo-document are joined with a [SEP] token, and the models are trained on these expanded queries. The method was tested on MS-MARCO passage ranking, TREC DL 2019/2020 and five BEIR datasets. It improves BM25 by about 3–15% without fine-tuning any model. DPR, SimLM and E5 also improve, but by less. Larger LLMs give bigger gains. The approach matters because it is a cheap, plug-in way to bring LLM knowledge into existing retrieval pipelines. The trade-offs are higher latency and occasional made-up facts.

Key points

  • With BM25 + query2doc, nDCG@10 rises from 51.2 to 66.2 on TREC DL 19 and from 47.7 to 62.9 on TREC DL 20, and MS-MARCO dev MRR@10 goes from 18.4 to 21.4. RM3 only slightly improves R@1k.

  • For dense retrievers, MRR@10 changes as follows:

    • DPR bert-base: 33.7 → 35.1
    • SimLM: 41.1 → 41.5
    • E5 base + KD: 40.7 → 41.5

    Gains shrink when the model is distilled from a cross-encoder re-ranker. For DPR, the gain of about 1 point holds at every amount of labeled training data tested.

  • Model scale matters. On TREC DL 19, nDCG@10 is 52.0 with babbage (1.3B), 66.2 with davinci-003 and 69.2 with GPT-4. Using only the query (51.2) or only the pseudo-document (48.7) is clearly worse than combining them.

  • On zero-shot BEIR the results are mixed. Entity-heavy DBpedia gains up to +5.7 and Trec-Covid gains +6.6 with BM25, but on Scifact both SimLM and E5 lose 2.9 points.

  • There are practical costs. An LLM call takes over 2000 ms, and BM25 index search slows from 16 ms to 177 ms. Generated passages can also contain factual errors. All generations are released as the Hugging Face dataset intfloat/query2doc_msmarco.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

153 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 145 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.585 · arXiv 2303.07678

@inproceedings{wang2023query2doc,
  title = {{Query2doc: Query Expansion with Large Language Models}},
  author = {Wang, Liang and Yang, Nan and Wei, Furu},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.585},
  eprint = {2303.07678},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.585}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexander Bondarenko, Maik Fröbe, Johannes Kiesel, Shahbaz Syed, Timon Gurcke, Meriem Beloucif, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, et al. 2022. Overview of touché 2022: argument retrieval. In International Conference of the Cross-Language Evaluation Forum for Europea… find
  2. Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In European Conference on Information Retrieval, pages 716–722. Springer. find
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  4. Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashche… find
  5. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020a. Overview of the trec 2019 deep learning track. ArXiv preprint, abs/2003.07820. find our card
  6. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. 2020b. Overview of the trec 2020 deep learning track. ArXiv preprint, abs/2003.07820. find our card
  7. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. find our card
  8. Luyu Gao and Jamie Callan. 2021. Condenser: a pre-training architecture for dense retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981–993, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  9. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without relevance labels. ArXiv preprint, abs/2212.10496. find our card
  10. Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. Dbpedia-entity v2: A test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, S… find
  11. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Onl… find our card
  12. Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2825–2… find
  13. Victor Lavrenko and W. Bruce Croft. 2001. Relevance-based language models. ACM SIGIR Forum, 51:260 – 267. find
  14. J. J. Rocchio. 1971. Relevance feedback in information retrieval. find
  15. Jimmy J. Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. ArXiv preprint, abs/2106.14807. — [arXiv:2106.14807](https://arxiv.org/abs/2106.14807) find
  16. Jimmy J. Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, Rodrigo Nogueira, and David R. Cheriton. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. Proceedings of the 44th International ACM SIGIR Conference on R… find
  17. Linqing Liu, Minghan Li, Jimmy Lin, Sebastian Riedel, and Pontus Stenetorp. 2022. Query expansion using contextual clue sampling with language models. ArXiv preprint, abs/2210.07093. — [arXiv:2210.07093](https://arxiv.org/abs/2210.07093) find
  18. Yuanhua Lv and ChengXiang Zhai. 2009. A comparative study of methods for estimating query language models with pseudo feedback. Proceedings of the 18th ACM conference on Information and knowledge management. find
  19. George A. Miller. 1992. WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992. find our card
  20. Rodrigo Nogueira and Jimmy Lin. From doc2query to doctttttquery. find
  21. Rodrigo Nogueira, Wei Yang, Jimmy J. Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. ArXiv preprint, abs/1904.08375. find our card
  22. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774. find our card
  23. ArXiv, Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. find
  24. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Ass… find our card
  25. Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. ArXiv preprint, abs/2210.01296. — [arXiv:2210.01296](https://arxiv.org/abs/2210.01296) find
  26. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). find
  27. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv p… find
  28. Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pages 1–12. ACM New York, NY, USA. find
  29. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association… find our card
  30. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. ArXiv preprint, abs/2212.03533. find our card
  31. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2023. SimLM: Pre-training with representation bottleneck for dense passage retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  32. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria,… find our card
  33. W. Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. ArXiv preprint, abs/2209.10063. find our card
  34. Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2022. Adversarial retriever-ranker for dense text retrieval. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. find