Elamur Research⌕
☰
home / papers / 2026 / Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs

Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs

Paper · 2026 · Qibin Wang, Pu Zhao, Shaohan Huang et al.

Summary

Common test-time scaling strategies, such as majority voting and Best-of-N, can only pick from the candidates a model produces. If every candidate is wrong, they cannot reach a correct answer. Parallel self-refinement, where a model reads several candidates and writes a new answer, avoids this limit, but it has been unclear why it works.

The authors define the Refinement Gap, the accuracy of self-refinement minus majority voting over the same candidates. Measured across Qwen2.5-Instruct and R1/QwQ models from 1.5B to 72B, the gap grows with model size and is only loosely tied to base accuracy.

Building on this, they propose Generative Self-Refinement (GSR). A large teacher with a high gap refines a small student's own candidates, and the student is fine-tuned on these refinements together with direct solutions. One model therefore learns both to propose and to refine answers.

GSR reports state-of-the-art results among parallel aggregation methods on five math benchmarks. It also generalizes to logic puzzles, which suggests refinement is a trainable skill rather than one only large models have.

Key points

  • Refinement Gap: defined as Δ = selfRef@k − maj@k.
    • Larger gains appear from about 14B parameters upward.
    • Example: DeepSeek-R1-Distill-Qwen-7B scores 54.4% pass@1 on AIME24 but has Δ = −0.8.
    • In contrast, Qwen2.5-32B-Instruct scores only 18.6% pass@1 yet reaches Δ = 3.1.
  • Training data: built from OpenMathReasoning, with 184k direct-solving examples and 184k refinement examples.
    • Hard cases where all candidates are wrong were deliberately kept.
  • Training objective: an NLL loss mixing refinement and direct solving with λ = 0.5.
    • Training uses k = 4 candidates; at inference the model works without any external verifier.
  • Candidate scaling (AIME24): GSR-7B reaches 63.2% at k = 2, slightly above oracle pass@2 (62.5%).
    • It keeps outperforming majority voting as k grows to 10.
  • Out-of-distribution test (Knights and Knaves 4ppl): pass@1 rises from 12.9% (base model) to 37.3%.
    • Self-refinement changes from −7.6% relative to maj@4 for the base model to +7.2% for GSR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Qibin Wang, Pu Zhao, Shaohan Huang, Fangkai Yang, Lu Wang, Furu Wei, Qingwei Lin, Saravan Rajmohan and Dongmei Zhang. 2026. Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1291

@inproceedings{wang2026learning,
  title = {{Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs}},
  author = {Qibin Wang, Pu Zhao, Shaohan Huang, Fangkai Yang, Lu Wang, Furu Wei, Qingwei Lin, Saravan Rajmohan and Dongmei Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1291},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1291}
}

References

45 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI-MO. 2024a. AIMO Validation AIME Dataset. https://huggingface.co/datasets/AI-MO/ aimo-validation-aime. Accessed: 2025-03-29. ↗
  2. AI-MO. 2024b. AIMO Validation AMC Dataset. https://huggingface.co/datasets/AI-MO/ aimo-validation-amc. Accessed: 2025-03-29. ↗
  3. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. Preprint, arXiv:2407.21787.
  4. Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. 2025. Rm-r1: Reward modeling as reasoning. Preprint, arXiv:2505.02387.
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  6. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  7. Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, and Nanyun Peng. 2024. LLM self-correction with decrim: Decompose, critique, and refine for enhanced following of instructions with multiple constraints.… find
  8. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC: large language models can self-correct with tool-interactive critiquing. In Proceedings of ICLR 2024.
  9. Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. 2025. Reward reasoning model. Preprint, arXiv:2505.14674. — [arXiv:2505.14674](https://arxiv.org/abs/2505.14674) find
  10. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In…
  11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceedings of NeurIPS 2021.
  12. Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. 2025. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. Preprint, arXiv:2504.07086. find
  13. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In Proceedings of ICLR 2024.
  14. Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan, and William Beauchamp. 2023. Rewarding chatbots for real-world engagement with millions of users.…
  15. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of ACL 2023, pages 14165–14178. Association for Computational Linguistics.
  16. Ammar Khairi, Daniel D’souza, Marzieh Fadaee, and Julia Kreutzer. 2025. Making, not taking, the best of n. Preprint, arXiv:2510.00931. — [arXiv:2510.00931](https://arxiv.org/abs/2510.00931) find
  17. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. In Proceedings of NeurIPS 2023.
  18. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of EMNLP 2024, pages 4334– 4353. Associat…
  19. Ludmila I Kuncheva. 2014. Combining pattern classifiers: methods and algorithms. John Wiley & Sons. find
  20. Yafu Li, Zhilin Wang, Tingchen Fu, Ganqu Cui, Sen Yang, and Yu Cheng. 2025a. From drafts to answers: Unlocking llm potential via aggregation fine-tuning. Preprint, arXiv:2501.11877. — [arXiv:2501.11877](https://arxiv.org/abs/2501.11877) find
  21. Zichong Li, Xinyu Feng, Yuheng Cai, Zixuan Zhang, Tianyi Liu, Chen Liang, Weizhu Chen, Haoyu Wang, and Tuo Zhao. 2025b. Llms can generate a better answer by aggregating their own responses. Preprint, arXiv:2503.04104. — [arXiv:2503.04104](https://arxiv.org/abs/2503.04104) find
  22. Yen-Ting Lin. 2025. Aime 2025 dataset. https://huggingface.co/datasets/yentinglin/ aime_2025. Accessed: 2025-03-29. ↗
  23. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in llms. Preprint, arXiv:2410.18451.
  24. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  25. Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. 2024. Generative reward models. Preprint, arXiv:2410.12832. find
  26. Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. 2025. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset. Preprint, arXiv:2504.16891. find
  27. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser-assis…
  28. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2024. REFINER: reasoning feedback on intermediate representations. In Proceedings of EACL 2024, pages 1100–1126. Association for Computational Linguistics.
  29. QwenTeam. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  30. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, W. Bradley Knox, Chelsea Finn, and Scott Niekum. 2024. Scaling laws for reward model overoptimization in direct alignment algorithms. In Proceedings of NeurIPS 2024. find
  31. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. Preprint, arXiv:2408.03314.
  32. Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism. In Proceedings of NAACL 2025, pages 4195–4206. Association for Computational Linguistics.
  33. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. In Proceedings of NeurIPS 2020. find
  34. Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. 2025. Multiagent finetuning: Self improvement with diverse reasoning chains. In Proceedings of ICLR 2025. find
  35. Giorgos Vernikos, Arthur Brazinskas, Jakub Adámek, Jonathan Mallinson, Aliaksei Severyn, and Eric Malmi. 2024. Small language models improve giants by rewriting their outputs. In Proceedings of EACL 2024, pages 2703–2718. Association for Computational Linguistics. find
  36. Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2025. Mixture-of-agents enhances large language model capabilities. In Proceedings of ICLR 2025. find
  37. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In Proceedings of ICLR 2023.
  38. Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. 2025. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. Preprint, arXiv:2505.10320. — [arXiv:2505.10320](https://arxiv.org/abs/2505.10320) find
  39. Genta Indra Winata, David Anugraha, Lucky Susanto, Garry Kuwanto, and Derry Tanti Wijaya. 2025. Metametrics: Calibrating metrics for generation tasks using human preferences. In Proceedings of ICLR 2025.
  40. Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. 2025. On memorization of large language models in logical reasoning. Preprint, arXiv:2410.23123.
  41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  42. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025b. Qwen2.5 technical report. Preprint, arXiv:24… find
  43. Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. 2025c. Reasonflux: Hierarchical llm reasoning via scaling thought templates. Preprint, arXiv:2502.06772. find
  44. Bohan Zhang, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. 2025. Cot-based synthesizer: Enhancing llm performance through answer synthesis. Preprint, arXiv:2501.01668. find
  45. Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, and Ilia Kulikov. 2025. The majority is not always right: Rl training for solution aggregation. Preprint, arXiv:2509.06870. — [arXiv:2509.06870](https://arxiv.org/abs/2509.06870) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-learning-refine-self-refinement-parallel/