Elamur Research⌕
☰
home / papers / 2026 / ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability

ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability

Paper · 2026 · Wenhan Liu, Xinyu Ma, Weiwei Sun et al.

Summary

LLM-based listwise rerankers struggle with complex search needs, such as expert StackExchange questions, coding, or math, because they are mostly trained on web search data like MSMARCO. That data rewards surface-level matching rather than reasoning. The authors address this lack of training data with an automatic, annotation-free pipeline. It collects queries from four scenarios (complex QA, coding, math, web search) and uses DeepSeek-R1 to pick positive and hard negative passages and to write reasoning chains and gold rankings. A self-consistency filter removes unreliable labels. The resulting 13K examples train ReasonRank in two stages: a cold-start SFT stage teaches the model to reason over passage lists, then an RL stage refines ranking quality. The RL stage uses a multi-view reward. Plain NDCG on a single sliding window does not guarantee good global rankings, so the reward also encourages relevant passages to move forward into later windows. ReasonRank sets a new state of the art on BRIGHT and R2MED and runs faster than the pointwise reasoning reranker Rank1. This shows that reasoning rerankers can be both more accurate and more efficient.

Key points

  • The training set has 13K synthesized examples. The complex QA part draws on six StackExchange sub-domains (Biology, Earth Science, Economics, Robotics, StackOverflow, Sustainability). Positives are mined from documents linked in accepted answers.
  • Reranking follows the standard sliding-window setup: top 100 passages, window size 20, step size 10. The 7B and 32B ReasonRank models beat the previous best results on BRIGHT by about 3 and 5 NDCG@10 points, respectively. The first-stage retriever was ReasonIR (8B).
  • Ablations on BRIGHT with ReasonRank (7B), full model averaging 35.74, show the following drops:
    • removing SFT: −7.05
    • training only on MSMARCO data: −5.66
    • removing RL: −2.59
    • replacing the multi-view reward with NDCG@10 alone: −1.54
    • non-reasoning variants also scored lower, at 32.96 and 33.60.
  • On 7 BEIR datasets with BM25 as the retriever, ReasonRank (32B) has the best average NDCG@10 (55.44). Its margin over baselines is smaller than on the reasoning-heavy benchmarks.
  • ReasonRank (7B) is 2–2.7× faster than Rank1 (7B). This is because it produces one reasoning chain per window of passages rather than one per passage.
  • Stated limitations: the model cannot switch to a non-reasoning mode, and it still relies on sliding windows instead of full-list ranking.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🏢BaiduBeijing · China74 works · 15,263 citations🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin and Zhicheng Dou. 2026. ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1005

@inproceedings{liu2026reasonrank,
  title = {{ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability}},
  author = {Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin and Zhicheng Dou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1005},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1005}
}

References

50 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yiqun Chen, Qi Liu, Yi Zhang, Weiwei Sun, Dait-ing Shi, Jiaxin Mao, and Dawei Yin. 2024. Tour-rank: Utilizing large language models for documents ranking with a tournament-inspired strategy. CoRR, abs/2406.11678. — [arXiv:2406.11678](https://arxiv.org/abs/2406.11678) find
  2. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR.
  3. Debrup Das, Seán Ó Nualláin, and Razieh Rahimi. 2025. Rader: Reasoning-aware dense retrieval models. CoRR, abs/2505.18405.
  4. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  5. Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. 2025a. Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning. CoRR, abs/2505.16410. — [arXiv:2505.16410](https://arxiv.org/abs/2505.16410) find
  6. Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. 2025b. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849. — [arXiv:2507.19849](https://arxiv.org/abs/2507.19849) find
  7. Yongqi Fan, Xiaoyang Chen, Dezhi Ye, Jie Liu, Hai-jin Liang, Jin Ma, Ben He, Yingfei Sun, and Tong Ruan. 2025a. Tfrank: Think-free reasoning enables practical pointwise llm ranking. arXiv preprint arXiv:2508.09539. — [arXiv:2508.09539](https://arxiv.org/abs/2508.09539) find
  8. Yongqi Fan, Kui Xue, Zelin Li, Xiaofan Zhang, and Tong Ruan. 2025b. An llm-based framework for biomedical terminology normalization in social media via multi-agent collaboration. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10712–10726. find
  9. Yongqi Fan, Yansha Zhu, Kui Xue, Jingping Liu, and Tong Ruan. 2024. Rrnorm: A novel framework for chinese disease diagnoses normalization via llm-driven terminology component recognition and reconstruction. In Findings of the Association for Computational Linguistics ACL 2024, pages 9162–9175. find
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks.
  11. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685.
  12. Zhaoheng Huang, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. 2025. CAGS: context-aware document ranking with contrastive graph sampling. IEEE Trans. Knowl. Data Eng., 37(1):89–101. find
  13. Lei Li, Xiao Zhou, and Zheng Liu. 2025. R2MED: A benchmark for reasoning-driven medical retrieval. CoRR, abs/2505.14558. — [arXiv:2505.14558](https://arxiv.org/abs/2505.14558) find
  14. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  15. Qi Liu, Bo Wang, Nan Wang, and Jiaxin Mao. 2024a. Leveraging passage embeddings for efficient listwise reranking with large language models. CoRR, abs/2406.14848. — [arXiv:2406.14848](https://arxiv.org/abs/2406.14848) find
  16. Wenhan Liu, Xinyu Ma, Yutao Zhu, Lixin Su, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2025. Coranking: Collaborative ranking with small and large ranking agents. CoRR, abs/2503.23427. find
  17. Wenhan Liu, Xinyu Ma, Yutao Zhu, Ziliang Zhao, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2024b. Sliding windows are not the end: Exploring full ranking with long-context large language models. CoRR, abs/2412.14574. find
  18. Wenhan Liu, Yujia Zhou, Yutao Zhu, and Zhicheng Dou. 2024c. How to personalize and whether to personalize? candidate documents decide. Knowledge and Information Systems, (9):5581–5604. find
  19. Wenhan Liu, Yutao Zhu, and Zhicheng Dou. 2024d. Demorank: Selecting effective demonstrations for large language models in ranking task. CoRR, abs/2406.16332. — [arXiv:2406.16332](https://arxiv.org/abs/2406.16332) find
  20. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th…
  21. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023a. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. CoRR, abs/2309.15088. find
  22. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023b. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! CoRR, abs/2312.02724. — [arXiv:2312.02724](https://arxiv.org/abs/2312.02724) find
  23. Xubo Qin, Jun Bai, Jiaqi Li, Zixia Jia, and Zilong Zheng. 2025. Tongsearch-qr: Reinforced query reasoning for retrieval. CoRR, abs/2506.11603. — [arXiv:2506.11603](https://arxiv.org/abs/2506.11603) find
  24. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2023. Large language models are effective text rankers with pairwise ranking prompting. CoRR, abs/2306.17563.
  25. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. KDD ’20. find
  26. Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi,…
  27. Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, and Luke Zettlemoyer. 2025. Reasonir: Training retrievers for reasoning tasks. CoRR, abs/2504.20595. find
  28. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300.
  29. Sahel Sharifymoghaddam, Ronak Pradeep, Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. 2025. Rankllm: A python package for reranking with llms. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Inform… find
  30. Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning. CoRR, abs/2505.17005. — [arXiv:2505.17005](https://arxiv.org/abs/2505.17005) find
  31. Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfe… — Qwen3 Technical Report
  32. Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan Ö. Arik, Danqi Chen, and Tao Yu. 2025. BRIGHT: A realistic and challenging benchmark for reasoning-intensive retrieval. In ICLR. Op…
  33. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023…
  34. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS Datasets and Benchmarks. find
  35. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, ab…
  36. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. In ACL (1), pages 11897–11916. Association for Computational Linguistics.
  37. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR. OpenReview.net.
  38. William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Trans. Inf. Syst., 28(4):20:1–20:38. find
  39. Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn J. Lawrie, and Benjamin Van Durme. 2025. Rank1: Test-time compute for reranking in information retrieval. CoRR, abs/2502.18418. — [arXiv:2502.18418](https://arxiv.org/abs/2502.18418) find
  40. Chenghao Xiao, G. Thomas Hudson, and Noura Al Moubayed. 2024. Rar-b: Reasoning as retrieval benchmark. CoRR, abs/2404.06347. — [arXiv:2404.06347](https://arxiv.org/abs/2404.06347) find
  41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Eugene Yang, Andrew Yates, Kathryn Ricci, Orion Weller, Vivek Chari, Benjamin Van Durme, and Dawn J. Lawrie. 2025b. Rank-k: Test-time reasoning for listwise reranking. CoRR, abs/2505.14432. — [arXiv:2505.14432](https://arxiv.org/abs/2505.14432) find
  42. Soyoung Yoon, Eunbi Choi, Jiyeon Kim, Hyeongu Yun, Yireun Kim, and Seung-won Hwang. 2024. Listt5: Listwise reranking with fusion-in-decoder improves zero-shot retrieval. In ACL (1), pages 2287–2308. Association for Computational Linguistics. find
  43. Soyoung Yoon, Gyuwan Kim, Gyu-Hwung Cho, and Seung-won Hwang. 2025. Acurank: Uncertainty-aware adaptive computation for listwise reranking. CoRR, abs/2505.18512. — [arXiv:2505.18512](https://arxiv.org/abs/2505.18512) find
  44. Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, and Aishwarya Agrawal. 2025. REARANK: reasoning re-ranking agent via reinforcement learning. CoRR, abs/2505.20046.
  45. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. CoRR, abs/2403.13372.
  46. Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. CoRR, abs/2308.07107.
  47. Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, Ji-Rong Wen, and Zhicheng Dou. 2024. INTERS: unlocking the power of large language models in search with instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume… find
  48. Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. Rankt5: Fine-tuning T5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIG… find
  49. Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. 2025. Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning. CoRR, abs/2503.06034. — [arXiv:2503.06034](https://arxiv.org/abs/2503.06034) find
  50. Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2024. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In SIGIR, pages 38–47. ACM. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-reasonrank-empowering-passage-ranking-strong/