Elamur Research⌕
☰
home / papers / 2024 / Bridging the Preference Gap between Retrievers and LLMs

Bridging the Preference Gap between Retrievers and LLMs

Summary

Retrieval-augmented generation (RAG) systems usually pair a retriever built around human reading habits, where rank order matters most, with an LLM that may want its context assembled differently. The authors call this mismatch a "preference gap." Their probes with a frozen PaLM2-S show that shuffling the order of the top-5 GTR passages hardly changes results, while changing which single passage is supplied changes them substantially. So choosing what goes into the prompt appears to matter more than ordering it.

To address this without touching either the retriever or the LLM, they propose BGM, a seq2seq bridge model (Flan-T5) that receives retrieved passages and emits a passage sequence for the LLM. This lets it select, reorder, or drop passages. Training happens in two stages:

  • supervised learning on silver sequences found by greedy search;
  • reinforcement learning that uses downstream task performance as the reward.

BGM was evaluated on QA (NQ, HotpotQA) and personalized generation (Avocado Email, Amazon Book reviews). It outperforms strong retrievers and reranking baselines, and it reduces the number of tokens sent to the LLM. This makes it practical for black-box LLMs.

Key points

  • Preference-gap probe: randomizing the order of top-5 passages shifted RAG performance by about 1%, whereas varying the top-1 selection shifted it by more than 5%.
  • Setup: GTR is the retriever, the frozen LLM is PaLM2-S (also PaLM2-XXS), and the default bridge is Flan-T5-XXL (11B). Bridges of large and XL size also beat plain GTR, and larger bridges perform better. Metrics are EM for QA and BLEU for generation.
  • In-domain results on PaLM2-S: BGM reaches 45.37 EM on NQ and 35.64 EM on HotpotQA.
  • Weak generalization:
    • A bridge trained on NQ scores 33.42 on HotpotQA.
    • On PaLM2-XXS, a bridge trained with PaLM2-S gets 30.63 / 24.55 EM (NQ / HotpotQA), compared with 39.88 / 28.69 when trained with PaLM2-XXS itself.
  • Case studies: BGM sometimes chooses no passages and answers from the LLM's own memory. On Email, it tends to pick messages that match in writing style or audience rather than topic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

31 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 30 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.562 · arXiv 2401.06954

@inproceedings{ke2024bridging,
  title = {{Bridging the Preference Gap between Retrievers and LLMs}},
  author = {Ke, Zixuan and Kong, Weize and Li, Cheng and Zhang, Mingyang and Mei, Qiaozhu and Bendersky, Michael},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.562},
  eprint = {2401.06954},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-long.562}
}

References

35 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimiliano Ciaramita, Lasse Espeholt, Thomas Hofmann, Yannic Kilcher, Sascha Rothe, Pier Giuseppe Sessa, et al. 2021. Boosting search engines with interactive agents. arXiv preprint arXiv:2109.00527. — [arXiv:2109.00527](https://arxiv.org/abs/2109.00527) find
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… find our card
  3. Andrea Bacciu, Florin Cocunasu, Federico Siciliano, Fabrizio Silvestri, Nicola Tonellotto, and Giovanni Trappolini. 2023. Rraml: Reinforced retrieval augmented machine learning. arXiv preprint arXiv:2307.12798. — [arXiv:2307.12798](https://arxiv.org/abs/2307.12798) find
  4. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learn… find
  5. Michiel De Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, and William W Cohen. 2023. Pre-computed memory or on-the-fly encoding? a hybrid approach to retrieval augmentation makes the most of your compute. In International Conference on Machine Learning, pages 73… find
  6. Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Sumit Sanghai, William W Cohen, and Joshua Ainslie. 2023. Glimmer: generalized late-interaction memory reranker. arXiv preprint arXiv:2306.10231. — [arXiv:2306.10231](https://arxiv.org/abs/2306.10231) find
  7. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929– 3938. PMLR. find
  8. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118. find our card
  9. Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282. find our card
  10. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299. find
  11. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906. find our card
  12. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. find
  13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  14. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474. find
  15. Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. 2023. Teach llms to personalize–an approach inspired by writing education. arXiv preprint arXiv:2308.07968. — [arXiv:2308.07968](https://arxiv.org/abs/2308.07968) find
  16. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. find our card
  17. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (… find our card
  18. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2021. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899. find our card
  19. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. arXiv preprint arXiv:1704.04572. find
  20. Douglas Oard, William Webber, David Kirsch, and Sergey Golitsynskiy. 2015. Avocado research email collection. Philadelphia: Linguistic Data Consortium. find
  21. OpenAI. 2023. Gpt-4 technical report. find our card
  22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020a. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. find
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020b. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. find
  24. Stephen E Robertson. 1977. The probability ranking principle in ir. Journal of documentation. find
  25. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023a. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR. find our card
  26. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023b. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652. find our card
  27. Zeng Wei, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2017. Reinforcement learning to rank with markov decision process. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. find
  28. Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. 2022. Memorizing transformers. arXiv preprint arXiv:2203.08913. find
  29. Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reit-ter, Hannaneh Hajishirzi, Mari Ostendorf, and Gaurav Singh Tomar. 2021. Conqrr: Conversational query rewriting for retrieval with reinforcement learning. arXiv preprint arXiv:2112.08558. find
  30. Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, Wei Zeng, and Xueqi Cheng. 2017. Adapting markov decision process for search result diversification. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. find
  31. Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408. find
  32. Jun Xu, Zeng Wei, Long Xia, Yanyan Lan, Dawei Yin, Xueqi Cheng, and Ji-Rong Wen. 2020. Reinforcement learning to rank with pairwise policy gradient. In Proceedings of the 43rd International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 509–518. find
  33. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP. find our card
  34. Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Retrieval-augmented multimodal language modeling. find
  35. Wei Zeng, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2018. Multi page search with reinforcement learning to rank. In Proceedings of the 2018 ACM SIGIR international conference on theory of information retrieval, pages 175–178. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ke-2024-bridging-preference-gap-between-retrievers/