Elamur Research⌕
☰
home / papers / 2026 / Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems

Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems

Paper · 2026 · Yilun Zhao, Jinbiao Wei, Tingyu Song et al.

Summary

Retrievers inside agentic deep-research pipelines need to supply a complementary set of evidence that covers every part of a reasoning problem, not just one topically similar passage. Current resources miss this. BRIGHT gives each query only a few gold passages from one or two web pages and tests retrievers in isolation. Synthetic training data usually pairs a query with a single positive. The authors build BRIGHT-PRO on BRIGHT's StackExchange subset: domain experts re-audit gold passages, add new evidence, and label weighted reasoning aspects. Retrievers are then scored both as static rankers and inside an LLM search loop. The authors also create RTriever-Synth from MS MARCO seeds. Short queries become analytical questions, a reference answer is split into aspects, each aspect yields a positive passage, and hard negatives are written to share topic but omit the needed aspect. LoRA tuning Qwen3-Embedding-4B on this data produces RTriever-4B, which clearly outperforms its base. The study shows that standard metrics hide coverage and efficiency differences that matter for real agents.

Key points

  • BRIGHT-PRO covers the seven expert domains of BRIGHT's StackExchange subset. It is evaluated with aspect-aware static metrics (e.g., alpha-nDCG, aspect recall) and agentic metrics, including completeness, overall answer quality and AER, which penalises extra search rounds.
  • Static vs. agentic results:
    • BGE-Reasoner is strongest overall, the DIVER family forms a second tier, and Qwen-family embedders trail.
    • BM25 is weak in static retrieval but becomes competitive in the agentic loop, where follow-up queries reduce vocabulary mismatch.
  • Adaptive-round setting:
    • BGE-Reasoner-8B leads with AER 3.65 in 5.10 rounds.
    • RTriever-4B ranks second by mean AER across the GPT-5-mini and Qwen3.5-122B-A10B agents.
    • GTE-7B has the best overall quality (4.51), but needs 6.67 rounds, which lowers its AER to 3.44.
  • Agent dependence: switching agents drops DIVER-4B-1020 from second to seventh place (AER 3.56 → 3.11). Lower-tier rankings therefore reflect agent–retriever fit as much as retriever quality.
  • Qualitative analysis of 175 RTriever-4B + GPT-5-mini traces finds five failure and success patterns:
    • early-round efficiency;
    • speculative answers when no gold evidence is retrieved;
    • repetitive retrieval of the same passages;
    • "aspect tunnel vision";
    • continued hypothesis testing after early success.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations🎓New York University ShanghaiShanghai · China11 works · 644 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksInformation retrieval & search · 552 works

Cite

Cite as Yilun Zhao, Jinbiao Wei, Tingyu Song, Siyue Zhang, Chen Zhao and Arman Cohan. 2026. Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1705

@inproceedings{zhao2026rethinking,
  title = {{Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems}},
  author = {Yilun Zhao, Jinbiao Wei, Tingyu Song, Siyue Zhang, Chen Zhao and Arman Cohan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1705},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1705}
}

References

59 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abdallah, J. Holdcroft, M. Ali, and Adam Jatowt. 2026a. Are llm-based retrievers worth their cost? an empirical study of efficiency, robustness, and reasoning overhead. find
  2. Abdelrahman Abdallah, Mohamed Darwish Mounis, Mahmoud Abdalla, M. Kasem, Mostafa Farouk Senussi, Mohamed Mahmoud, Mohammed Ali, Adam Jatowt, and H. Kang. 2026b. Mm-bright: A multi-task multimodal benchmark for reasoning-intensive retrieval. ArXiv, abs/2601.09562. — [arXiv:2601.09562](https://arxiv.org/abs/2601.09562) find
  3. Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Litsearch: A retrieval benchmark for scientific literature search.
  4. Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, L. Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading comprehension dataset. ArXiv, abs/1611.09268.
  5. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763.
  6. Reza Esfandiarpoor, George Zerveas, Ruochen Zhang, Macton Mgonzo, Carsten Eickhoff, and Stephen H. Bach. 2025. Beyond contrastive learning: Synthetic data enables list-wise training with multiple levels of relevance. pages 22860–22882. find
  7. Xincan Feng, Noriki Nishida, Yusuke Sakai, and Yuji Matsumoto. 2026. The wisdom of many queries: Complexity-diversity principle for dense retriever training. ArXiv, abs/2602.09448. — [arXiv:2602.09448](https://arxiv.org/abs/2602.09448) find
  8. Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, Hidekazu Oiwa, E. Gribovskaya, Jan Ackermann, John Blitzer, S. Goldshtein, and D. Das. 2026. Deepsearchqa: Bridging the comprehensiveness gap for deep research agents. ArXiv, abs/2601.20975. — [arXiv:2601.20975](https://arxiv.org/abs/2601.20975) find
  9. Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. ArXiv, abs/2406.20094.
  10. Tiansheng Hu, Yilun Zhao, Canyu Zhang, Arman Cohan, and Chen Zhao. 2026. Sage: Benchmarking and improving retrieval for deep research agents. ArXiv, abs/2602.05975. — [arXiv:2602.05975](https://arxiv.org/abs/2602.05975) find
  11. Jianlyu Chen, Junwei Lan, Chaofan Li, Defu Lian, and Zheng Liu. 2025a. Reasonembed: Enhanced text embeddings for reasoning-intensive document retrieval. arXiv preprint arXiv:2510.08252. find
  12. Jiajie Jin, Yanzhao Zhang, Mingxin Li, Dingkun Long, Pengjun Xie, Yutao Zhu, and Zhicheng Dou. 2026. Laser: Internalizing explicit reasoning into latent space for dense retrieval. ArXiv, abs/2603.01425. — [arXiv:2603.01425](https://arxiv.org/abs/2603.01425) find
  13. Zijian Chen, Xueguang Ma, Shengyao Zhuang, Jimmy Lin, Akari Asai, and Victor Zhong. 2026. Agentir: Reasoning-aware retrieval for deep research agents. find
  14. Ehsan Kamalloo, Nandan Thakur, Carlos Lassance, Xueguang Ma, Jheng-Hong Yang, and Jimmy Lin. 2024. Resources for brewing beir: Reproducible reference models and statistical analyses. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, p… find
  15. Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. 2025b. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600. — [arXiv:2508.06600](https://arxiv.org/abs/2508.06600) find
  16. Charles L.A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, and Ian MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st Annual International ACM SI-GIR Conference on Research and Development in Information… find
  17. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428.
  18. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: A comprehensive survey on llm-based evaluation methods.
  19. Debrup Das, Sam O’ Nuallain, and Razieh Rahimi. 2025. Rader: Reasoning-aware dense retrieval models.
  20. Ruizhe Li, Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2026. Deepresearch bench ii: Diagnosing deep research agents via rubrics from expert report. ArXiv, abs/2601.08536. — [arXiv:2601.08536](https://arxiv.org/abs/2601.08536) find
  21. Hang Ding, Yilun Zhao, Tiansheng Hu, Manasi Patwardhan, and Arman Cohan. 2026. Scirag: Adaptive, citation-aware, and outline-guided retrieval and synthesis for scientific literature. pages 6440–6460.
  22. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023a. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  23. Zheng Dou, Deqing Wang, Fuzhen Zhuang, Jian Ren, and Yanling Hu. 2025. Flew: Facet-level and adaptive weighted representation learning of scientific documents. ArXiv, abs/2509.07531. — [arXiv:2509.07531](https://arxiv.org/abs/2509.07531) find
  24. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023b. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  25. Meixiu Long, Duolin Sun, Dan Yang, Junjie Wang, Yue Shen, Jian Wang, Peng Wei, Jinjie Gu, and Jiahai Wang. 2025. Diver: A multi-stage approach for reasoning-intensive information retrieval. find
  26. Jaron Lochner, Rob Evans, Rodney Kinney, Daniel S. Weld, Doug Downey, and Sergey Feldman. 2025. Ai2 scholar qa: Organized literature synthesis with attribution. find
  27. G. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2025. Improving text embedding models with positive-aware hard-negative mining. Proceedings of the 34th ACM International Conference on Information and Knowledge Management. find
  28. Tingyu Song, Guo Gan, Mingsheng Shang, and Yilun Zhao. 2025a. Ifir: A comprehensive benchmark for evaluating instruction-following in expert-domain information retrieval. pages 10186–10204. find
  29. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906.
  30. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelei… — Text and Code Embeddings by Contrastive Pre-Training
  31. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  32. Tingyu Song, Yilun Zhao, Siyue Zhang, Chen Zhao, and Arman Cohan. 2025b. Limrank: Less is more for reasoning-intensive information reranking. ArXiv, abs/2510.23544. find
  33. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2022. One embedder, any task: Instruction-finetuned text embeddings. arXiv preprint arXiv:2212.09741.
  34. Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. 2025. BRIGHT: A realistic and challenging benchmark for reasoning-intensive retrieval. In The Thirte…
  35. Yifei Shen, Yilun Zhao, Justice Ou, Tinglin Huang, and Arman Cohan. 2026. Patient-similarity cohort reasoning in clinical text-to-SQL. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1367–1412, Rabat, Moroc…
  36. Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel M. Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, Weiyi Wang, Zhe Li, Gus… — [arXiv:2509.20354](https://arxiv.org/abs/2509.20354) find
  37. Amanpreet Singh, Joseph Chee Chang, Chloe Anastasiades, Dany Haddad, Aakanksha Naik, Amber Tanaka, Angele Zamarron, Cecile Nguyen, Jena D. Hwang, Jason Dunkleberger, Matt Latzke, Smita Rao, Benben Wang, Minghao Tang, Hengran Zhang, Jiafeng Guo, and Keping Bi. 2026. Training dense retrievers with mul… — [arXiv:2602.12727](https://arxiv.org/abs/2602.12727) find
  38. Saron Samuel, Andrew Yates, Dawn J. Lawrie, Ian Soboroff, Trevor Adriaanse, Benjamin Van Durme, and Eugene Yang. 2026. Coveragebench: Evaluating information coverage across tasks and domains. find
  39. Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi,… — DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
  40. Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen tau Yih, Pang Wei Koh, and Luke Zettlemoyer. 2025b. ReasonIR: Training retrievers for reasoning tasks. In Second Conference on Language Modeling. find
  41. Jiayu Wang, Yifei Ming, Riya Dulepet, Qinglin Chen, Austin Xu, Zixuan Ke, Frederic Sala, Aws Albarghouthi, Caiming Xiong, and Shafiq Joty. 2025. Liveresearchbench: A live benchmark for user-centric deep research in the wild. find
  42. Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. 2025. Rank1: Test-time compute for reranking in information retrieval. find
  43. Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. 2024. Promptriever: Instruction-trained retrievers can be prompted like language models. arXiv preprint arXiv:2409.11136. — [arXiv:2409.11136](https://arxiv.org/abs/2409.11136) find
  44. Bin Wu, Arastun Mammadli, Xiaoyu Zhang, and Emine Yilmaz. 2026. Agentsearchbench: A benchmark for ai agent search in the wild. find
  45. Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, et al. 2025. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572.
  46. Chenghao Xiao, G Thomas Hudson, and Noura Al Moubayed. 2024. Rar-b: Reasoning as retrieval benchmark. arXiv preprint arXiv:2404.06347. — [arXiv:2404.06347](https://arxiv.org/abs/2404.06347) find
  47. Lei Xiong, Kun Luo, Ziyi Xia, Wenbo Zhang, Jin-Ge Yao, Zheng Liu, Jing Shao, Jianlyu Chen, Hongjin Qian, Xi Yang, Qian Yu, Hao Li, C. Yue, Xia’an Du, Yuyang Wang, Yesheng Liu, Haiyu Xu, and Zhicheng Dou. 2026. Autoresearchbench: Benchmarking ai agents on complex scientific literature discovery. find
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  49. John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025b. Swe-smith: Scaling data for software engineering agents. find
  50. Li S. Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar. 2025. Researchqa: Evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics. find
  51. Siyue Zhang, Yuan Gao, Xiao Zhou, Yilun Zhao, Tingyu Song, Arman Cohan, A. Luu, and Chen Zhao. 2025a. Mrmr: A realistic and expert-level multidisciplinary benchmark for reasoning-intensive multimodal retrieval. ArXiv, abs/2510.09510. — [arXiv:2510.09510](https://arxiv.org/abs/2510.09510) find
  52. Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Cohan, Anh Tuan Luu, and Chen Zhao. 2025b. Diffusion vs. autoregressive language models: A text embedding perspective. arXiv preprint arXiv:2505.15045.
  53. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025c. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  54. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. 2025d. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  55. Yilun Zhao, Yitao Long, Tintin Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Xiangru Tang, Yiming Zhang, Chen Zhao, and Arman Cohan. 2024a. FinDVer: Explainable claim verification over long and hybrid-content financial documents. In Proceedings of the 2024 Conference on Empirical Methods in Natura… find
  56. Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Yixin Liu, Xiangru Tang, Joseph Chee Chang, Jesse Dodge, Jonathan Bragg, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. 2026. Sciarena: An open evaluation platform for non-verifiable scientific literature-grounded tasks…
  57. Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. 2024b. Swift:a scalable lightweight infrastructure for fine-tuning. ArXiv, abs/2408.05517. — [arXiv:2408.05517](https://arxiv.org/abs/2408.05517) find
  58. Jiawei Zhou, Hang Ding, and Haiyun Jiang. 2025. Ark: Answer-centric retriever tuning via kg-augmented curriculum learning. ArXiv, abs/2511.16326.
  59. Junhan Yang, Jiahe Wan, Yichen Yao, Wei Chu, Yinghui Xu, Emma Wang, and Yuan Qi. 2025c. inf-retriever-v1 (revision 5f469d7). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-rethinking-reasoning-intensive-retrieval-evaluating/