Elamur Research⌕
☰
home / papers / 2026 / AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning

AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning

Paper · 2026 · Jingbo Sun, Wenyue Chong, Songjun Tu et al.

Summary

Agentic RAG systems let LLMs call a retriever over several turns, but agents often keep searching after they already have enough evidence. This wastes compute and adds latency. Earlier fixes penalise the number of search steps, which can leave hard questions under-explored. The authors first run probing experiments. They show that each question has a minimal sufficient search depth, and that this depth depends on both how complex the question is and how capable the model is. They then propose AutoSearch, an RL framework trained mainly with PPO. After every search step, the agent writes an intermediate answer. Checking these answers locates the point where the agent first becomes correct, so the framework can reward reaching that depth and penalise steps beyond it. Two more rewards support this: a step-wise base reward for format, non-redundancy and correctness, and a quality reward based on F1 gains between intermediate answers. Across single-hop and multi-hop QA benchmarks, AutoSearch reaches competitive or higher EM than the baselines with fewer searches. The result matters because it cuts retrieval cost without the usual accuracy loss.

Key points

  • Pilot study (Qwen2.5-14B-Instruct): NQ is nearly solved after one search, while Bamboogle peaks at three steps and gets worse afterwards. On Bamboogle, the optimal depth also varies with model size: Qwen-3B keeps improving up to the largest depth tested, 7B peaks at two steps, and 14B at three.
  • Three-part reward design:
    • a base reward combining format and outcome, which treats repeated retrievals as invalid;
    • a search efficiency reward tied to the minimal sufficient depth;
    • a search quality reward based on F1 improvement of intermediate answers.
  • Evaluation setup: six datasets (NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, Bamboogle), compared against Search-R1, StepSearch and HiPRAG. Search-R1 and StepSearch search deeply and over-search often. HiPRAG searches less but loses accuracy.
  • Ablation (Qwen2.5-3B-Base): dropping the efficiency reward leaves NQ EM about the same (44.2 vs 43.7) but triples search depth (3.24 vs 1.03 steps). Removing either of the other two rewards lowers accuracy.
  • Generality and code: training with GRPO instead of PPO gives comparable results. Code is released at github.com/bofusun/AutoSearch. A stated limitation is that only small maximum search depths were explored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Jingbo Sun, Wenyue Chong, Songjun Tu, Qichao Zhang, Yaocheng Zhang, Jiajun Chai, Xiaohan Wang, Wei Lin, Guojun Yin and Dongbin Zhao. 2026. AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1399

@inproceedings{sun2026autosearch,
  title = {{AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning}},
  author = {Jingbo Sun, Wenyue Chong, Songjun Tu, Qichao Zhang, Yaocheng Zhang, Jiajun Chai, Xiaohan Wang, Wei Lin, Guojun Yin and Dongbin Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1399},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1399}
}

References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yong Deng, Guoqing Wang, Zhenzhe Ying, Xiaofeng Wu, Jinzhen Lin, Wenwen Xiong, Yuqin Dai, Shuo Yang, Zhanwei Zhang, Qiwen Wang, and 1 others. 2025. Atom-searcher: Enhancing agentic deep research via fine-grained atomic thought reward. arXiv preprint arXiv:2508.12800. — [arXiv:2508.12800](https://arxiv.org/abs/2508.12800) find
  2. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491– 6501.
  3. Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. 2026. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. In The Fourteenth International Conference on Learning Representations. find
  4. Yuqian Fu, Yuanheng Zhu, Jiajun Chai, Guojun Yin, Wei Lin, Qichao Zhang, and Dongbin Zhao. 2025. Rlae: Reinforcement learning-assisted ensemble for llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 13463–13477. find
  5. Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. 2026. Unlocking long-horizon agentic search with large-scale end-to-end rl. In The Fourteenth International Conference on Learning Representations. find
  6. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, Haofen Wang, and 1 others. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1):32.
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609– 6625.
  9. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  10. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. In Proceedings of the Second Conference on Language Modeling (COLM).
  11. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  12. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781.
  13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, and 1 others. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Lin…
  14. Xin Liu, Haoran Li, and Dongbin Zhao. 2025. Videos are sample-efficient supervisions: Behavior cloning from videos via latent representations. In Advances in Neural Information Processing Systems. find
  15. Runyu Lu, Peng Zhang, Ruochuan Shi, Yuanheng Zhu, Dongbin Zhao, Yang Liu, Dong Wang, and Cesare Alippi. 2025. Equilibrium policy generalization: A reinforcement learning framework for cross-graph zero-shot generalization in pursuit-evasion games. In Advances in Neural Information Processing Systems. find
  16. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computationa… find
  17. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711. find
  18. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. 2025. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958. find
  19. Yuanhao Shen, Xiaodan Zhu, and Lei Chen. 2024. Smartcal: An approach to self-aware tool-use evaluation and calibration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 774–789. find
  20. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. Replug: Retrieval-augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
  21. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025a. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592.
  22. Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025b. R1-searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning. arXiv preprint arXiv:2505.17005. — [arXiv:2505.17005](https://arxiv.org/abs/2505.17005) find
  23. Jingbo Sun, Songjun Tu, Haoran Li, Xin Liu, Yaran Chen, Ke Chen, Dongbin Zhao, and 1 others. 2025a. Unsupervised zero-shot reinforcement learning via dual-value forward-backward representation. In The Thirteenth International Conference on Learning Representations. find
  24. Jingbo Sun, Qichao Zhang, Songjun Tu, Xing Fang, Yupeng Zheng, Haoran Li, Ke Chen, and Dongbin Zhao. 2026. Saliency-guided representation with consistency policy learning for visual unsupervised reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit… find
  25. Songjun Tu, Qichao Zhang, Jingbo Sun, Yuqian Fu, Linjing Li, Xiangyuan Lan, Dongmei Jiang, Yaowei Wang, and Dongbin Zhao. 2025b. Perception-consistency multimodal large language models reasoning via caption-regularized policy optimization. arXiv preprint arXiv:2509.21854. — [arXiv:2509.21854](https://arxiv.org/abs/2509.21854) find
  26. Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. 2025. Acting less is reasoning more! teaching model to act efficiently. In Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning (LAW). find
  27. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
  28. Peilin Wu, Mian Zhang, Kun Wan, Wentian Zhao, Kaiyu He, Xinya Du, and Zhiyu Chen. 2026. Hiprag: Hierarchical process rewards for efficient agentic retrieval augmented generation. In The Fourteenth International Conference on Learning Representations. find
  29. Peilin Wu, Mian Zhang, Xinlu Zhang, Xinya Du, and Zhiyu Chen. 2025. Search wisely: Mitigating suboptimal agentic searches by reducing uncertainty. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19734–19745. find
  30. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  31. Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, Jiajun Chai, Guojun Yin, Wei Lin, Shuai Ma, Fuzhen Zhuang, Deqing Wang, and 1 others. 2026. Your group-relative advantage is biased. arXiv preprint arXiv:2601.08521. — [arXiv:2601.08521](https://arxiv.org/abs/2601.08521) find
  32. Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Yang Song, and Han Li. 2025b. Rearter: Retrieval-augmented reasoning with trustworthy process rewarding. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retri…
  33. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–…
  34. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (ACL), pages 10014–10037.
  35. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh International Conference on Learning Representations.
  36. Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. 2025a. Learning when to think: Shaping adaptive Chuhuai Yue, Chengqi Dong, Yinan Gao, Hang He, Jiajun Chai, Wei Lin, and Guojun Yin. 2026. Promoting efficient reasoning with verifiable stepwise reward. find
  37. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34530–34538. find
  38. Xuhui Zheng, Kang An, Ziliang Wang, Yuhang Wang, and Yichao Wu. 2025. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21816–21841.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-autosearch-adaptive-search-depth-efficient/