Elamur Research⌕
☰
home / papers / 2026 / When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG

When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG

Paper · 2026 · Erfan Nourbakhsh, Rocky Slavin, Ke Yang et al.

Summary

Retrieval-augmented generation (RAG) is often recommended for medical question answering, and earlier benchmarks such as MIRAGE/MedRAG reported large gains. Those studies, however, mostly tested proprietary or 70B-scale models on multiple-choice exam questions. This paper asks whether the benefit holds for practical open-weight models and for everyday consumer-health queries. The authors ran a large grid of experiments. They paired five instruction-tuned models (7B–72B) with four retrievers and four corpora, two expert and two lay. They tested on ten biomedical QA datasets, both open-ended and multiple-choice, always against a no-retrieval baseline. Ablations covered the number of retrieved documents, few-shot prompting, and clean versus noisy retrieved context.

Retrieval turned out to help only slightly and inconsistently, usually by 1–2 points. The choice of backbone model mattered far more than the retriever or corpus. Expert and layman sources performed almost the same. The authors conclude that the main bottleneck is how well models use evidence, not retrieval quality. This points future work toward reranking, filtering and better evidence integration.

Key points

  • Setup:
    • Models: Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, LLaMA-3.1-70B-Instruct and Qwen2.5-72B-Instruct.
    • Retrievers: BM25, TF-IDF, MedCPT and Hybrid RRF.
    • Corpora: PubMed, medical textbooks, Yahoo Answers and HealthCareMagic.
  • Datasets:
    • Layman: MeQSum, MedRedQA, MedicationQA, MASH-QA and ChatDoctor-iCliniq.
    • Expert or mixed: BioASQ, MedQA, MedMCQA, MMLU Medical and MedQuAD.
    • Metrics: ROUGE, BLEU, METEOR, BERTScore and accuracy.
  • Small gains from retrieval: best RAG configuration vs. no retrieval in BERTScore was 62.88 vs. 61.72 for Llama-8B and 63.23 vs. 61.28 for Qwen-7B. Expert and lay corpora usually differed by under 2 points.
  • Ablations:
    • Retrieving more than k=5 documents adds little in open-ended settings.
    • Three-shot prompting gives small models a modest boost, but more shots hurt them sharply.
    • Adding distractor documents to relevant ones can make RAG worse than no retrieval.
  • Resources: code is released at github.com/erfan-nourbakhsh/BioMedicalRAG. The paper was published at BioNLP 2026 with NSF support.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Erfan Nourbakhsh, Rocky Slavin, Ke Yang and Anthony Rios. 2026. When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.72

@inproceedings{nourbakhsh2026when,
  title = {{When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG}},
  author = {Erfan Nourbakhsh, Rocky Slavin, Ke Yang and Anthony Rios},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.72},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.72}
}

References

34 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. B. Abacha, Y. Mrabet, M. Sharp, T. R. Goodwin, S. E. Shooshan, and D. Demner-Fushman. 2019. Bridging the gap between consumers’ medication questions and trusted answers. Stud Health Technol Inform, 264:25–29. 1879-8365 Abacha, Asma Ben Mrabet, Yassine Sharp, Mark Goodwin, Travis R Shooshan, Sonya… — [doi:10.3233/shti190176](https://doi.org/10.3233/shti190176) find
  2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  3. Asma Ben Abacha and Dina Demner-Fushman. 2019a. On the summarization of consumer health questions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2228– 2234, Florence, Italy. Association for Computational Linguistics. find
  4. Asma Ben Abacha and Dina Demner-Fushman. 2019b. A question-entailment approach to question answering. BMC Bioinformatics, 20(1):511. find
  5. Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 758–759, New York,…
  6. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1):32.
  7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 181 o… — The Llama 3 Herd of Models
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  9. Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6).
  10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  11. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315, Singapore. Association for Computational Linguistics.
  12. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  13. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14).
  14. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  15. Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. 2023. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(11):btad651. find
  16. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  17. A. Krithara, A. Nentidis, K. Bougiatiotis, and G. Paliouras. 2023. Bioasq-qa: A manually curated corpus for biomedical question answering. Sci Data, 10(1):170. 2052-4463 Krithara, Anastasia Orcid: 0000-0003-0491-4507 Nentidis, Anastasios Bougiatiotis, Konstantinos Paliouras, Georgios Dataset Journal… find
  18. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc…
  19. Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Martin Krallinger, Miguel Rodríguez-Ortega, Eduard Rodriguez-López, Natalia Loukachevitch, Andrey Sakhovskiy, Elena Tutubalina, Dimitris Dimitriadis, Grigorios Tsoumakas, George Giannakoulas, Alexandra Bekiaridou, Athanasios Samaras, Gior… find
  20. Vincent Nguyen, Sarvnaz Karimi, Maciej Rybinski, and Zhenchang Xing. 2023. MedRedQA for medical consumer question answering: Dataset, tasks, and neural baselines. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapt… find
  21. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.
  22. Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2024. Fine-tuning or retrieval? comparing knowledge injection in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 237–250, Miami, Florida, USA. Association for Computational Linguisti…
  23. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 24…
  24. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. find
  25. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9248–9274, Singapore. Association for…
  26. Ming Zhang, Shuai Dou, Ziyang Wang, and Yunfang Wu. 2022. Focus-driven contrastive learning for medical question summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6176–6186, Gyeongju, Republic of Korea. International Committee on Computational Lin… find
  27. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine…
  28. Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy. 2020. Question answering with long multiple-span answers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3840–3849, Online. Association for Computational Linguistics. find
  29. Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushm… — Large Language Models Encode Clinical Knowledge
  30. A KAREN SPARCK JONES. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–21. find
  31. Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. 2024. MedAgents: Large language models as collaborators for zero-shot medical reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 599–621, Bangkok, Thai…
  32. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  33. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, Bangkok, Thailand. Association for Computational Linguistics.
  34. Yahoo! Research. 2009. Yahoo! Webscope Datasets Catalog. Technical report, Yahoo! Inc. 19 Datasets Available. Accessed via Stanford InfoLab. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nourbakhsh-2026-when-retrieval-doesn-t-help/