Elamur Research⌕
☰
home / papers / 2025 / ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning

ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning

Paper · 2025 · Changtai Zhu, Siyin Wang, Ruijun Feng et al.

Summary

In conversational search, user queries often depend on earlier turns. They may leave things out, be vague, or refer back to prior context. Conversational Query Reformulation (CQR) rewrites such queries into stand-alone versions that a standard retriever can handle. Earlier CQR systems learned from rewrites written by humans or produced by large LLMs, which is expensive. Their outputs were also not tuned to what the downstream retriever actually rewards.

ConvSearch-R1 needs no external reference rewrites at all. Its training has two stages:

  1. Self-Driven Policy Warm-Up: a 3B model is prompted with few-shot examples to produce candidate rewrites. Only the rewrites that rank the gold passage first are kept and used for fine-tuning.
  2. Reinforcement learning: GRPO is applied with a rank-incentive reward. This reward gives denser, smoother feedback than sparse metrics such as Recall@K. The model also reasons before writing each rewrite.

On TopiOCQA and QReCC, the method beats earlier state-of-the-art systems, including ones built on 7B models. This shows that retrieval feedback alone can train a strong rewriter cheaply.

Key points

  • Backbones are Qwen2.5-3B and Llama3.2-3B. Retrievers are BM25 (sparse) and ANCE (dense).
  • With dense retrieval on TopiOCQA, average gains over the previous best are 10.3% (Llama3.2-3B) and 10.7% (Qwen2.5-3B).
  • Ablations confirm the value of both components. Models trained without the warm-up stage, and models rewarded with raw MRR@3, NDCG@3, R@10 or R@100, all perform worse.
  • In scaling tests with Qwen2.5 models, even the 0.5B model beats the baseline with ANCE, and the gap widens as models get larger.
  • Code, data and models are released at github.com/BeastyZ/ConvSearch-R1. Stated limitations are that no model above 7B was trained and that reasoning makes outputs longer, which raises inference cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🏢ByteDanceBeijing · China41 works · 4,180 citations🎓UNSW SydneySydney · Australia34 works · 1,509 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksDialogue systems · 624 worksInformation retrieval & search · 473 works

Cite

Cite as Changtai Zhu, Siyin Wang, Ruijun Feng, Kai Song and Xipeng Qiu. 2025. ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1349

@inproceedings{zhu2025convsearch,
  title = {{ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning}},
  author = {Changtai Zhu, Siyin Wang, Ruijun Feng, Kai Song and Xipeng Qiu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1349},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1349}
}

References

51 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. 2021. Topiocqa: Open-domain conversational question answering with topic switching. Transactions of the Association for Computational Linguistics, 10:468–483.
  2. R. Anantha, Svitlana Vakulenko, Zhucheng Tu, S. Longpre, Stephen G. Pulman, and Srinivas Chappidi. 2020. Open-domain question answering goes conversational via question rewriting. In North American Chapter of the Association for Computational Linguistics.
  3. Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguist…
  4. Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading comprehension dataset. ArXiv, abs/1611.09268.
  5. Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Hua zeng Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. 2025. Research: Learning to reason with search for llms via reinforcement learning. ArXiv, abs/2503.19470. find
  6. Zhiyu Chen, Jie Zhao, Anjie Fang, Besnik Fetahu, Oleg Rokhlenko, and Shervin Malmasi. 2022. Reinforced question rewriting for conversational question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 357–370, Abu Dhabi, UAE. A… find
  7. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. 2025. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. ArXiv, abs/2501.17161.
  8. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020a. find
  9. Cast 2020: The conversational assistance track overview. In Text Retrieval Conference. find
  10. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020b. Trec cast 2019: The conversational assistance track overview. ArXiv, abs/2003.13624. — [arXiv:2003.13624](https://arxiv.org/abs/2003.13624) find
  11. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. TREC cast 2021: The conversational assistance track overview. In Proceedings of the Thirtieth Text REtrieval Conference, TREC 2021, online, November 15-19, 2021, volume 500-335 of NIST Special Publication. National Institute of Standards and Tec… find
  12. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 179 others. 2025. Deepseek-r1: Incentivizing reasoning capability i…
  13. Marco Del Tredici, Gianni Barlacchi, Xiaoyu Shen, Weiwei Cheng, and Adriá de Gispert. 2021. Question rewriting for open-domain conversational qa: Best practices and limitations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM ’21, page 2974–2978, N… find
  14. Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla. 2021. Explicable reward design for reinforcement learning agents. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA. Curran Associates Inc. find
  15. Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP…
  16. Jianfeng Gao, Chenyan Xiong, Paul N. Bennett, and Nick Craswell. 2022. Neural approaches to conversational information retrieval. Neural Approaches to Conversational Information Retrieval. find
  17. Christophe Van Gysel and M. de Rijke. 2018. Pytrec_eval: An extremely fast python interface to trec_eval. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. find
  18. Yunah Jang, Kang-il Lee, Hyunkyung Bae, Hwanhee Lee, and Kyomin Jung. 2024. IterCQR: Iterative conversational query reformulation with retrieval guidance. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  19. Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. 2025. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. ArXiv, abs/2503.00223. — [arXiv:2503.00223](https://arxiv.org/abs/2503.00223) find
  20. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. ArXiv, abs/2503.09516.
  21. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7:535–547.
  22. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. ArXiv, abs/1705.03551.
  23. Sungdong Kim and Gangwoo Kim. 2022. Saving dense retriever from shortcut dependency in conversational search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10278–10287, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  24. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  25. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  26. Yilong Lai, Jialong Wu, Congzhi Zhang, Haowen Sun, and Deyu Zhou. 2025. AdaCQR: Enhancing query reformulation for conversational search via sparse and dense retrieval alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7698–7720, Abu Dhabi, UAE. Associa…
  27. Jimmy J. Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations. ArXiv, abs/2102.10073. — [arXiv:2102.10073](https://arxiv.org/abs/2102.10073) find
  28. Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy J. Lin. 2020. Conversational question reformulation via sequence-to-sequence architectures and pretrained language models. ArXiv, abs/2004.01909. — [arXiv:2004.01909](https://arxiv.org/abs/2004.01909) find
  29. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-tuning llama for multi-stage text retrieval. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.
  30. Kelong Mao, Chenlong Deng, Haonan Chen, Fengran Mo, Zheng Liu, Tetsuya Sakai, and Zhicheng Dou. 2024. ChatRetriever: Adapting large language models for generalized and robust conversational dense retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pa… find
  31. Kelong Mao, Zhicheng Dou, Bang Liu, Hongjin Qian, Fengran Mo, Xiangli Wu, Xiaohua Cheng, and Zhao Cao. 2023a. Search-oriented conversational query editing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4160–4172, Toronto, Canada. Association for Computational Linguist… find
  32. Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian. 2023b. Large language models know your contextual search intent: A prompting framework for conversational search. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1211–1225, Singapore. Ass… find
  33. Fengran Mo, Abbas Ghaddar, Kelong Mao, Mehdi Rezagholizadeh, Boxing Chen, Qun Liu, and Jian-Yun Nie. 2024a. CHIQ: Contextual history enhancement for improving query rewriting in conversational search. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22… find
  34. Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, and Jian-Yun Nie. 2023. ConvGQR: Generative query reformulation for conversational search. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4998–5012, Toronto, Canada.… find
  35. Fengran Mo, Chen Qu, Kelong Mao, Tianyu Zhu, Zhan Su, Kaiyu Huang, and Jian-Yun Nie. 2024b. History-aware conversational dense retrieval. In Annual Meeting of the Association for Computational Linguistics. find
  36. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  37. Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit Iyyer. 2020. Open-retrieval conversational question answering. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  38. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290.
  39. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300.
  40. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. Proceedings of the Twentieth European Conference on Computer Systems.
  41. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. ArXiv, abs/2503.05592.
  42. Tianhua Zhang, Kun Li, Hongyin Luo, Xixin Wu, James R. Glass, and Helen M. Meng. 2024. Adaptive query rewriting: Aligning rewriters through marginal probability of conversational answers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13444–13461, Mi… find
  43. Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. 2025. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588.
  44. Svitlana Vakulenko, S. Longpre, Zhucheng Tu, and R. Anantha. 2020. Question rewriting for conversational question answering. Proceedings of the 14th ACM International Conference on Web Search and Data Mining. find
  45. Svitlana Vakulenko, Nikos Voskarides, Zhucheng Tu, and Shayne Longpre. 2021. A comparison of question rewriting methods for conversational passage retrieval. CoRR, abs/2101.07382. — [arXiv:2101.07382](https://arxiv.org/abs/2101.07382) find
  46. Nikos Voskarides, Dan Li, Pengjie Ren, E. Kanoulas, and M. de Rijke. 2020. Query resolution for conversational search with limited supervision. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  47. Zeqiu Wu, Yi Luan, Hannah Rashkin, D. Reitter, and Gaurav Singh Tomar. 2021. Conqrr: Conversational query rewriting for retrieval with reinforcement learning. In Conference on Empirical Methods in Natural Language Processing.
  48. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. ArXiv, abs/2007.00808.
  49. Fanghua Ye, Meng Fang, Shenghui Li, and Emine Yilmaz. 2023. Enhancing conversational search: Large language model-aided informative query rewriting. ArXiv, abs/2310.09716. find
  50. Chanwoong Yoon, Gangwoo Kim, Byeongguk Jeon, Sungdong Kim, Yohan Jo, and Jaewoo Kang. 2025. Ask optimal questions: Aligning large language models with retriever‘s preference in conversation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5899–5921, Albuquerque, New M… find
  51. Shih Yuan Yu, Jiahua Liu, Jingqin Yang, Chenyan Xiong, Paul N. Bennett, Jianfeng Gao, and Zhiyuan Liu. 2020. Few-shot generative conversational query rewriting. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-convsearch-r1-enhancing-query-reformulation/