Elamur Research⌕
☰
home / papers / 2025 / DEBATE, TRAIN, EVOLVE: Self-Evolution of Language Model Reasoning

DEBATE, TRAIN, EVOLVE: Self-Evolution of Language Model Reasoning

Paper · 2025 · Gaurav Srivastava, Zhenyu Bi, Meng Lu et al.

Summary

Large language models increasingly hit a data ceiling, which raises the question of whether they can improve reasoning without new labeled data. Multi-agent debate (MAD) helps agents catch each other's mistakes, but it requires running several models for every query at inference time. The authors propose DEBATE, TRAIN, EVOLVE (DTE), which turns debate into a training signal rather than an inference procedure. Several agents first debate a problem until they agree or reach a majority vote. The resulting reasoning traces and consensus answers then serve as pseudo-labels for GRPO fine-tuning of a single model, so no ground truth is needed. To make the debates more reliable, the authors introduce REFLECT-CRITIQUE-REFINE (RCR) prompting. Under RCR, each agent reviews its own reasoning, critiques exactly two peers, and must give a new reasoning step whenever it changes its answer, which curbs sycophancy and verbosity. Across six open-weight models and seven reasoning benchmarks, the evolved single models clearly outperform their base versions. This shows that the benefits of debate can be transferred into cheap single-pass inference.

Key points

  • On GSM-PLUS, accuracy rose by 8.92% on average. On the other benchmarks, which were not used in training, it rose by 5.8% on average, suggesting the gains carry over across domains.
  • Evaluation data included GSM8K, GSM-Plus, MATH, ARC, GPQA and CommonsenseQA. Models were from the Qwen-2.5 family (1.5B–14B) and Llama (3B, 8B).
  • Under standard MAD prompting, smaller models abandoned correct answers for confident wrong ones more than 28% of the time on average; RCR was designed to counter this.
  • Models under 3B parameters suffered forgetting in the second evolution round. Reducing the sampling temperature from 0.7 to 0.3 cut KL drift by roughly one third and recovered up to 76% of the lost accuracy.
  • Training on all debate traces beat using only debated cases (by 4.43 points) and a random 2K subset (by 9.17 points). GRPO gains saturated around 8K steps. Code and trained models are released at github.com/ctrl-gaurav/Debate-Train-Evolve.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Virginia TechBlacksburg · United States36 works · 43,036 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Gaurav Srivastava, Zhenyu Bi, Meng Lu and Xuan Wang. 2025. DEBATE, TRAIN, EVOLVE: Self-Evolution of Language Model Reasoning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1666

@inproceedings{srivastava2025debate,
  title = {{DEBATE, TRAIN, EVOLVE: Self-Evolution of Language Model Reasoning}},
  author = {Gaurav Srivastava, Zhenyu Bi, Meng Lu and Xuan Wang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1666},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1666}
}

References

32 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, and 1 others. 2025. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318. find
  2. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.
  3. Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085.
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  6. Caia Costello, Simon Guo, Anna Goldie, and Azalia Mirhoseini. 2025. Think, prune, train, improve: Scaling reasoning without scaling models. arXiv preprint arXiv:2504.18116. — [arXiv:2504.18116](https://arxiv.org/abs/2504.18116) find
  7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314.
  8. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning.
  9. Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. 2024. Acc-debate: An actor-critic approach to multi-agent debate. arXiv preprint arXiv:2411.00053. — [arXiv:2411.00053](https://arxiv.org/abs/2411.00053) find
  10. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, and 1 others. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations.
  11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  12. Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. Self-evolve: A code evolution framework via large language models. ArXiv, abs/2306.02907. — [arXiv:2306.02907](https://arxiv.org/abs/2306.02907) find
  13. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180.
  14. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. GSM-plus: A comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  15. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17…
  16. Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Weichao Wang, Xingshan Zeng, Lifeng Shang, and 1 others. 2023. Self: Self-evolution with language feedback. arXiv preprint arXiv:2310.00533. — [arXiv:2310.00533](https://arxiv.org/abs/2310.00533) find
  17. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  18. Someen Park, Jaehoon Kim, Seungwan Jin, Sohyun Park, and Kyungsik Han. 2024. Predict: Multi-agent-based debate simulation for generalized hate speech detection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20963–20987. find
  19. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Lidén, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. ArXiv, abs/2302.12813.
  20. Leonardo Ranaldi and Andrè Freitas. 2024. Self-refine instruction-tuning for aligning reasoning in language models. arXiv preprint arXiv:2405.00402.
  21. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  22. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076. — [arXiv:2310.10076](https://arxiv.org/abs/2310.10076) find
  23. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  24. Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. 2023. Should we be going mad? a look at multi-agent debate strategies for llms. In Forty-first International Conference on Machine Learning. find
  25. Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. 2025a. Towards reasoning ability of small language models. arXiv preprint arXiv:2502.11569.
  26. Gaurav Srivastava, Aafiya Hussain, Sriram Srinivasan, and Xuan Wang. 2025b. Llmthinkbench: Towards basic math reasoning and overthinking in large language models. Preprint, arXiv:2507.04023. — [arXiv:2507.04023](https://arxiv.org/abs/2507.04023) find
  27. Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. 2025. Multiagent finetuning: Self improvement with diverse reasoning chains. In The Thirteenth International Conference on Learning Representations. find
  28. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  29. Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. 2024. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? In Annual Meeting of the Association for Computational Linguistics. find
  30. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find
  31. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. 2024. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, volume 1126. find
  32. Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. 2024. Automatic instruction evolving for large language models. Preprint, arXiv:2406.00770.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srivastava-2025-debate-train-evolve-self-evolution/