Elamur Research⌕
☰
home / papers / 2026 / Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training

Paper · 2026 · Tianle Xia, Ming Xu, Lingxiang Hu et al.

Summary

Agentic RAG systems trained with reinforcement learning, such as Search-R1, usually receive a single binary reward for the final answer. This signal is sparse, gives nothing back from failed rollouts, and slows convergence. Search-P1, from Tencent, replaces that reward with one that scores the whole reasoning trajectory.

The policy first writes an explicit plan. Its trajectory is then judged on two tracks, and the higher of the two scores is kept:

  • Self-consistency: whether the model carried out its own stated plan.
  • Reference alignment: how many essential steps it covered from a reference planner, matched without regard to order. The reference planners are built offline using rejection sampling and LLM voting.

A soft outcome score also gives partial credit to wrong answers. Efficiency ratios penalize redundant steps.

Training used Qwen2.5-7B/3B-Instruct on merged NQ and HotpotQA data. Evaluation covered seven public QA benchmarks plus the proprietary AD-QA set. Search-P1 beat Search-R1 by 7.7 accuracy points on average. The gains held for open-source evaluator LLMs and in a production A/B test. The method requires no step-level annotation and adds no cost at inference time.

Key points

  • Setup: retrieval used E5 over the 2018 Wikipedia dump, returning the top 3 passages per search.
  • Benchmarks: NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle, and AD-QA, an anonymized advertising set of 1,000 multi-hop test questions.
  • Reward: a weighted sum of path, outcome, and format terms. The soft outcome score uses α = 0.8, and the method works with both GRPO and PPO.
  • Evaluator sensitivity (on 7B):
    • The frontier evaluators (DeepSeek-V3.2, HY 2.0, GLM-5, GPT-4o) differ by only 1.1 average ACC points.
    • Qwen3-32B reaches 46.2, compared with 39.6 for Search-R1.
    • Step-coverage judgments remain the most robust signal.
  • Online A/B test against Search-R1: Search-P1 won 84.3%, tied 6.7%, and lost 9.0% of comparisons. Thumbs-up rose 325%, conversation turns rose 42%, and thumbs-down fell 40%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Tianle Xia, Ming Xu, Lingxiang Hu, Yiding Sun, Wenwei Li, Linfang Shang, Liqun Liu, Peng Shu, Huan Yu and Jie Jiang. 2026. Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.36

@inproceedings{xia2026search,
  title = {{Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training}},
  author = {Tianle Xia, Ming Xu, Lingxiang Hu, Yiding Sun, Wenwei Li, Linfang Shang, Liqun Liu, Peng Shu, Huan Yu and Jie Jiang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.36},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.36}
}

References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. 2025. Research: Learning to reason with search for llms via reinforcement learning. Preprint, arXiv:2503.19470. find
  2. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  3. Xinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Jie Zhou. 2025. Deeprag: Thinking to retrieval step by step for large language models. Preprint, arXiv:2502.01142. find
  4. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
  5. Lingxiang Hu, Chenyang Hei, Fuliang Li, Chengxi Gao, Jiaxing Shen, and Xingwei Wang. 2025. Smarttc: A real-time ml-based traffic classification with smartnic. In 2025 IEEE/ACM 33rd International Symposium on Quality of Service (IWQoS), pages 1–10. IEEE. find
  6. Jerry Huang, Siddarth Madala, Risham Sidhu, Cheng Niu, Hao Peng, Julia Hockenmaier, and Tong Zhang. 2025. Rag-rl: Advancing retrieval-augmented generation via rl and curriculum learning. Preprint, arXiv:2503.12759. — [arXiv:2503.12759](https://arxiv.org/abs/2503.12759) find
  7. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. Preprint, arXiv:2503.09516.
  8. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
  9. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  10. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th Internation…
  11. Jian Li, Xiaoxi Li, Yan Zheng, Yizhang Jin, Shuo Wang, Jiafu Wu, Yabiao Wang, Chengjie Wang, and Xiaotong Yuan. 2025a. A survey on ai search with large language models. Preprints. find
  12. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025b. Search-o1: Agentic search-enhanced large reasoning models. Preprint, arXiv:2501.05366.
  13. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  14. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics. find
  15. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  16. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  17. Zeyang Sha, Shiwen Cui, and Weiqiang Wang. 2025. Sem: Reinforcement learning for search-efficient large language models. Preprint, arXiv:2505.07903. hierarchical process rewards for efficient agentic retrieval augmented generation. arXiv preprint arXiv:2510.07794. — [arXiv:2505.07903](https://arxiv.org/abs/2505.07903) find
  18. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  19. Peilin Wu, Mian Zhang, Xinlu Zhang, Xinya Du, and Zhiyu Zoey Chen. 2025b. Search wisely: Mitigating sub-optimal agentic searches by reducing uncertainty. Preprint, arXiv:2505.17281. find
  20. Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. Agentic retrieval-augmented generation: A survey on agentic rag. Preprint, arXiv:2501.09136. — [arXiv:2501.09136](https://arxiv.org/abs/2501.09136) find
  21. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025a. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. Preprint, arXiv:2503.05592.
  22. Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025b. R1-searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning. Preprint, arXiv:2505.17005. — [arXiv:2505.17005](https://arxiv.org/abs/2505.17005) find
  23. Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Yang Song, and Han Li. 2025. Rearter: Retrieval-augmented reasoning with trustworthy process rewarding. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrie…
  24. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  25. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  26. Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, and Furu Wei. 2025. Chain-of-retrieval augmented generation. Preprint, arXiv:2501.14342. — [arXiv:2501.14342](https://arxiv.org/abs/2501.14342) find
  27. Tianle Xia, Liang Ding, Guojia Wan, Yibing Zhan, Bo Du, and Dacheng Tao. 2025. Improving complex reasoning over knowledge graph with logic-aware curriculum tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12881–12889. find
  28. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  29. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  30. Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, and 6 others. 2025a. The landscape of agentic reinforcement lea… — [arXiv:2509.02547](https://arxiv.org/abs/2509.02547) find
  31. Qi Zhang, Shouqing Yang, Lirong Gao, and Hao Chen. 2025b. LeTS: Learning to think-and-search via process-and-outcome reward hybridization. Preprint, arXiv:2505.17447. find
  32. Tianhua Zhang, Kun Li, Junan Li, and Yunxiang Li. 2026. TreePS-RAG: Tree-based process supervision for reinforcement learning in agentic rag. Preprint, arXiv:2601.06922. — [arXiv:2601.06922](https://arxiv.org/abs/2601.06922) find
  33. Wenlin Zhang, Xiangyang Li, Kuicai Dong, Yichao Wang, Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Derong Xu, Zhaocheng Du, Huifeng Guo, and 1 others. 2025c. Process vs. outcome reward: Which is better for agentic rag reinforcement learning. arXiv preprint arXiv:2505.14069. — [arXiv:2505.14069](https://arxiv.org/abs/2505.14069) find
  34. Zhao Wang, Ziliang Zhao, and Zhicheng Dou. 2026. ProRAG: Process-supervised reinforcement learning for retrieval-augmented generation. Preprint, arXiv:2601.21912. — [arXiv:2601.21912](https://arxiv.org/abs/2601.21912) find
  35. Peilin Wu, Mian Zhang, Kun Wan, Wentian Zhao, Kaiyu He, Xinya Du, and Zhiyu Chen. 2025a. Hiprag: find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2026-search-p1-path-centric-reward/