Elamur Research⌕
☰
home / papers / 2026 / PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering

PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering

Paper · 2026 · Yiqing Zhang, Xiaozhong Liu and Fabricio Murai

Summary

Biomedical question answering needs answers that are both correct and supported by current, checkable literature. LLMs that rely on internal knowledge can hallucinate or become outdated. Standard RAG pipelines cannot repair a weak initial query, and self-reflection agents only revise after retrieval and generation are finished.

The authors propose PubMed Reasoner, a training-free agent that moves feedback to the start of the pipeline. It works in three stages:

  • An LLM proposes MeSH terms with rationales and combines them into a Boolean PubMed query. A self-critic then repeatedly judges each term for coverage, alignment and redundancy, using only titles and abstracts of preliminary hits.
  • Articles are read in small batches, and retrieval stops once the gathered evidence is sufficient, which limits token cost.
  • The system writes an answer with inline PMID citations.

With a GPT-4o backbone, the agent reaches 78.32% accuracy on PubMedQA, marginally above human experts. It also improves on MMLU Clinical Knowledge compared with plain LLM, RAG and self-reflection baselines. The work offers a template for transparent, citation-backed assistants in high-stakes domains without maintaining private knowledge stores.

Key points

  • Pipeline: three stages, namely self-critic query refinement over live PubMed metadata, batch-wise reflective retrieval with early stopping, and evidence-grounded generation with explicit citations.
  • Query feedback: three signals adapted from search benchmarks (Coverage, Alignment, Redundancy) score each MeSH term. Core concepts are joined with AND and synonyms with OR.
  • Main result: 78.32% accuracy on PubMedQA with GPT-4o, plus consistent gains on MMLU Clinical Knowledge.
  • Quality evaluation: LLM-as-judge favored the system's answers on four criteria: how sound the reasoning was, how well it was backed by evidence, how relevant it was clinically, and how trustworthy it was. Human annotators from medical schools checked how well these judgments agree with human ones.
  • Resources and limitations: code, prompts, judge setup and baselines are on GitHub (swiftzhang125/ACL_2026_PubMed_Reasoner), with an extended arXiv version (2603.27335). Stated limitations include possible LLM bias in the self-critic, missed lower-ranked articles, and errors propagating through the linear pipeline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Worcester Polytechnic InstituteWorcester · United States12 works · 380 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,685 worksQuestion answering · 487 worksInformation retrieval & search · 552 works

Cite

Cite as Yiqing Zhang, Xiaozhong Liu and Fabricio Murai. 2026. PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.763

@inproceedings{zhang2026pubmed,
  title = {{PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering}},
  author = {Yiqing Zhang, Xiaozhong Liu and Fabricio Murai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.763},
  url = {https://doi.org/10.18653/v1/2026.acl-long.763}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hasan Abu-Rasheed, Christian Weber, and Madjid Fathi. 2024. Knowledge graphs as context sources for llm-based explanations of learning recommendations. In 2024 IEEE Global Engineering Education Conference (EDUCON), pages 1–5. IEEE. find
  2. Arvind Arasu, Junghoo Cho, Hector Garcia-Molina, Andreas Paepcke, and Sriram Raghavan. 2001. Searching the web. ACM Transactions on Internet Technology (TOIT), 1(1):2–43. find
  3. Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. 2024. A survey on rag with llms. Procedia computer science, 246:3781–3790. find
  4. Leonid Boytsov. 2011. Indexing methods for approximate dictionary searching: Comparative analysis. Journal of Experimental Algorithmics (JEA), 16:1–1. find
  5. Sanjiban Choudhury. 2025. Process reward models for llm agents: Practical framework and directions. arXiv preprint arXiv:2502.10325. — [arXiv:2502.10325](https://arxiv.org/abs/2502.10325) find
  6. EJ Crane and Charles L Bernier. 1951. Indexing and index-searching. Punched Cards: Their Applications to Science and Industry, page 331. find
  7. Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. 2024. Bias and unfairness in information retrieval systems: New challenges in the llm era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6437–6447. find
  8. Wanling Gao, Yuqing Zhu, Zhen Jia, Chunjie Luo, Lei Wang, Zhiguo Li, Jianfeng Zhan, Yong Qi, Yongqiang He, Shiming Gong, and 1 others. 2013. Bigdatabench: a big data benchmark suite from web search engines. arXiv preprint arXiv:1307.0320. — [arXiv:1307.0320](https://arxiv.org/abs/1307.0320) find
  9. Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. 2023. Language models hallucinate, but may excel at fact verification. arXiv preprint arXiv:2310.14564. find
  10. Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Chaoyou Fu, Guanglu Song, and 1 others. 2024. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409.12959. — [arXiv:2409.12959](https://arxiv.org/abs/2409.12959) find
  11. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146.
  12. Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. 2025. Why language models hallucinate. arXiv preprint arXiv:2509.04664. — [arXiv:2509.04664](https://arxiv.org/abs/2509.04664) find
  13. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871.
  14. Zheng Liu, Yujia Zhou, Yutao Zhu, Jianxun Lian, Chaozhuo Li, Zhicheng Dou, Defu Lian, and Jian-Yun Nie. 2024. Information retrieval meets large language models. In Companion Proceedings of the ACM Web Conference 2024, pages 1586–1589. find
  15. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023a. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.
  16. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, and 1 others. 2023b. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452. find
  17. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  18. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  19. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  20. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  21. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, and 1 others. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, 31(3):943–950. find
  22. Karen Sparck Jones. 1974. Automatic indexing. Journal of documentation, 30(4):393–432. find
  23. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  24. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290…
  25. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817. 16763 11

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-pubmed-reasoner-dynamic-reasoning-based/