Elamur Research⌕
☰
home / papers / 2026 / Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents

Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents

Paper · 2026 · Sahel Sharifymoghaddam and Jimmy Lin

Summary

Deep search agents answer hard multi-hop questions by alternating between issuing queries and reasoning. Scaling this test-time computation is costly, and the role of reranking in such pipelines has received little study. The authors use the BrowseComp-Plus benchmark, which runs agents against a fixed, human-verified corpus, to measure how reranking interacts with model size and reasoning effort. They pair gpt-oss-20b and gpt-oss-120b agents with the qwen3-embedding-8b retriever and insert zero-shot listwise reranking at depths of 10, 20 or 50 candidates. They also propose Effective Token Cost (ETC), which weights non-cached input, cached input and output tokens differently to reflect hardware or API pricing. Reranking improved both retrieval and final answer accuracy in every configuration tested. Moderate reranking combined with medium reasoning often matched or exceeded high-reasoning agents without reranking, while costing fewer effective tokens and less time. The results suggest that reranking is a cheaper way to improve deep research systems than adding more search-time reasoning.

Key points

  • Setup: The experiments cover all 830 BrowseComp-Plus queries. The agents run at low, medium and high reasoning effort, with output caps of 2k, 8k and 16k tokens. Reranking uses RankLLM with a window of 20, plus a sliding window with stride 10 at d = 50. In every case the agent receives only the top-5 documents, each truncated to 512 tokens.
  • ETC metric: ETC = Input_nc + α·Input_c + β·Output. The cache discount α was varied over {0.1, 0.3, 0.5} and the output premium β over {3, 5, 7}. A further term, γ, extends the metric to setups where the reranker is a different model from the agent.
  • One-shot reranking: Reranking raised NDCG@5 from 19.72 for the retriever alone to 46.05 with oss-120b at medium effort and d = 50. Increasing the depth d helped more than increasing reasoning effort. Once normalized by ETC, low-effort reranking was the most efficient choice and was adopted for the full pipeline.
  • End-to-end results:
    • oss-20b at high effort improved from 41.38% to 55.97% accuracy at d = 50.
    • oss-120b at high effort improved from 51.20% to 63.35%.
    • oss-120b at medium effort with d = 50 reached 57.97%, which beats high reasoning without reranking.
  • Heterogeneous setup: The cross-encoder qwen3-reranker-0.6b (γ = 0.32 relative to gpt-oss-20b) kept most of the accuracy gains at lower ETC. For example, oss-20b at high effort and d = 50 reached 52.91% accuracy, compared with 55.97% using listwise reranking.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada67 works · 13,592 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksInformation retrieval & search · 473 works

Cite

Cite as Sahel Sharifymoghaddam and Jimmy Lin. 2026. Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1289

@inproceedings{sharifymoghaddam2026rerank,
  title = {{Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents}},
  author = {Sahel Sharifymoghaddam and Jimmy Lin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1289},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1289}
}

References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024), Vienna, Austria.
  2. Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. 2025. BrowseComp-Plus: A more fair and… — [arXiv:2508.06600](https://arxiv.org/abs/2508.06600) find
  3. In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of the 6th Annual Conference on Machine Learning and Systems (MLSys 2024), pages 325–338, Santa Clara, CA, USA. find
  4. Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. 2025. Deep research agents: A systematic examination and roadmap. arXiv:2506.18096. — [arXiv:2506.18096](https://arxiv.org/abs/2506.18096) find
  5. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th Annual Conf…
  6. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv:1901.04085. pretrained sequence-to-sequence model. In Findings of the association for computational linguistics: EMNLP 2020, pages 708–718.
  7. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, and 1093 others. 2025. Humanity’s last e… — Humanity's Last Exam
  8. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.
  9. Sahel Sharifymoghaddam, Ronak Pradeep, Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. 2025. RankLLM: A Python package for reranking with LLMs. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Inform… find
  10. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023…
  11. Fali Wang, Hui Liu, Zongyu DAI, Jingying Zeng, Zhiwei Zhang, Zongyu Wu, Chen Luo, Zhen Li, Xianfeng Tang, Qi He, and Suhang Wang. 2025. AgentTTS: Large language model agent for test-time compute-optimal scaling strategy in complex tasks. In Proceedings of the 39th Annual Conference on Neural Informa… find
  12. Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. 2025. A survey of LLM-based deep search agents: Paradigm, optimization, and challenges. arXiv:2508.05668. — [arXiv:2508.05668](https://arxiv.org/abs/2508.05668) find
  13. Yuan-An Xiao, Pengfei Gao, Chao Peng, and Yingfei Xiong. 2025. Improving the efficiency of LLM agent systems through trajectory reduction. arXiv:2509.23586. — [arXiv:2509.23586](https://arxiv.org/abs/2509.23586) find
  14. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, and 1 others. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv:2506.05176.
  15. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a *Table 5: Hardware configuration for (a) reranking-only and (b) end-to-end deep research experiments.* Reranking-only Experiments Model Reasoning GPU Type oss-20b oss-120b Low, Medium Low, Medium H200 H200 \\… find
  16. (a) End-to-end Deep Research Experiments Model Reasoning Reranking GPU Type oss-20b oss-20b oss-120b oss-120b low, med, high low, med, high low, med, high low, med, high N/A (d = 0) listwise (d \> 0) N/A (d = 0) listwise (d \> 0) H200 H200 H200 H200 \\\#GPUs 1 2 1 2 oss-20b low, med, high cross-encoder… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharifymoghaddam-2026-rerank-before-you-reason-analyzing/