Elamur Research⌕
☰
home / papers / 2026 / CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

Paper · 2026 · Teng Pan, Yuchen Yan, Zixuan Wang et al.

Summary

Label-free reinforcement learning for LLM reasoning usually rewards a model for matching its own majority-voted answer. The authors show that this setup drifts into what they call the consensus trap. Optimizing for self-agreement shrinks output diversity, so the model becomes confident in recurring wrong answers, and reward accuracy erodes. Under TTRL it falls from about 95% to roughly 80%.

CoVerRL counters this by having one policy play two roles inside a multi-turn RL loop:

  • As generator, the model samples solutions and drafts a pseudo-label by majority vote.
  • As verifier, it inspects solutions that reach the majority answer and drops queries whose consensus it rejects.
  • In return, the vote supplies balanced positive and negative examples for training the verifier.
  • Self-correction: a negative judgment triggers a revision step. Everything is optimized with an Answer-Anchored GRPO variant.

Across Qwen and Llama models on mathematical reasoning benchmarks, CoVerRL beats label-free baselines by 4.7–5.9% on average. It also turns a near-chance self-verifier into a reliable one. This suggests that label-free self-improvement can escape stagnation without external supervision.

Key points

  • Training dynamics on Qwen3-1.7B-Base:
    • Under TTRL, label accuracy stalls near 70% while reward accuracy decays.
    • CoVerRL keeps reward accuracy above 90% and lifts label accuracy from 75% to almost 88%.
  • Gains over TTRL: average improvements of 5.7%, 5.9% and 4.7% across three models (Qwen3-1.7B-Base, Qwen2.5-7B, Llama-3.2-3B-Instruct), evaluated on four reasoning benchmarks including GPQA.
  • Self-verification accuracy: rises from 55.8% to 81.2% on Qwen3-1.7B and from 54.0% to 86.5% on Qwen2.5-7B.
  • Need for balanced sampling:
    • A TTRL + PAG baseline (PAG's verify-then-revise workflow on pseudo-labels, trained with PPO) collapses on Llama-3.2-3B-Instruct. Its verifier rejects nearly everything, and Correct Recall drops close to zero.
    • CoVerRL avoids this by enforcing |V+| = |V−| and using GRPO.
  • Ablations on Qwen3-1.7B-Base:
    • Removing verification filtering, verifier updates, self-correction, or AA-GRPO each lowers results.
    • Without self-correction training, GPQA Acc.@final (25.4%) falls below Acc.@first (31.0%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China74 works · 15,263 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Teng Pan, Yuchen Yan, Zixuan Wang, Ruiqing Zhang, Guiyang Hou, Wenqi Zhang, Weiming Lu, Jun Xiao and Yongliang Shen. 2026. CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1376

@inproceedings{pan2026coverrl,
  title = {{CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution}},
  author = {Teng Pan, Yuchen Yan, Zixuan Wang, Ruiqing Zhang, Guiyang Hou, Wenqi Zhang, Weiming Lu, Jun Xiao and Yongliang Shen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1376},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1376}
}

References

51 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anonymous. 2025. Label-free GUI grounding via confidence-guided negative reinforcement learning. find
  2. In Submitted to The Fourteenth International Conference on Learning Representations. Under review. find
  3. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. Preprint, arXiv:2304.05128. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, and 1 others. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  6. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, and 1 others. 2025a. Retool: Reinforcement learning for strategic tool use in llms. Preprint, arXiv:2504.11536. find
  7. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025b. Group-in-group policy optimization for llm agent training. Preprint, arXiv:2505.10978. find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  9. Jiacheng Guo, Ling Yang, Peter Chen, Qixin Xiao, Yinjie Wang, and 1 others. 2025. Genenv: Difficulty-aligned co-evolution between llm agents and environment simulators. Preprint, arXiv:2512.19682. — [arXiv:2512.19682](https://arxiv.org/abs/2512.19682) find
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, and 1 others. 2021. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.
  11. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, and 1 others. 2025. R-zero: Self-evolving reasoning llm from zero data. Preprint, arXiv:2508.05004. — [arXiv:2508.05004](https://arxiv.org/abs/2508.05004) find
  12. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. Preprint, arXiv:2310.01798.
  13. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, and 1 others. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. Preprint, arXiv:2403.07974.
  14. Yuhua Jiang, Yuwen Xiong, Yufeng Yuan, Chao Xin, Wenyuan Xu, and 1 others. 2025. Pag: Multi-turn reinforced llm self-correction with policy as generative verifier. Preprint, arXiv:2506.10406.
  15. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, and 1 others. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. Preprint, arXiv:2503.09516.
  16. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, and 1 others. 2024. Rewardbench: Evaluating reward models for language modeling. Preprint, arXiv:2403.13787. find
  17. Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack. 2025. Revise: Learning to refine at test-time via intrinsic self-verification. Preprint, arXiv:2502.14565. — [arXiv:2502.14565](https://arxiv.org/abs/2502.14565) find
  18. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang, Ziju Shen, and 1 others. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 13:9. find
  19. Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. 2025. Confidence is all you need: Few-shot rl fine-tuning of language models. Preprint, arXiv:2506.06395. — [arXiv:2506.06395](https://arxiv.org/abs/2506.06395) find
  20. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, and 1 others. 2023. Let’s verify step by step. Preprint, arXiv:2305.20050.
  21. Jia Liu, ChangYi He, YingQiao Lin, MingMin Yang, FeiYang Shen, and ShaoGuo Liu. 2025a. Ettrl: Balancing exploration and exploitation in llm test-time reinforcement learning via entropy mechanism. Preprint, arXiv:2508.11356. — [arXiv:2508.11356](https://arxiv.org/abs/2508.11356) find
  22. Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, and 1 others. 2025b. A simple "try again" can elicit multi-turn llm reasoning. Preprint, arXiv:2507.14295. find
  23. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, and 1 others. 2025c. Agentbench: Evaluating llms as agents. Preprint, arXiv:2308.03688.
  24. Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, and 1 others. 2025d. Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards. Preprint, arXiv:2505.13445. — [arXiv:2505.13445](https://arxiv.org/abs/2505.13445) find
  25. Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, and 1 others. 2025. S$^2$r: Teaching llms to self-verify and self-correct via reinforcement learning. Preprint, arXiv:2502.12853.
  26. Ning Miao, Yee Whye Teh, and Tom Rainforth. 2023. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. Preprint, arXiv:2308.00436. find
  27. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, and 1 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  28. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, and 1 others. 2023. Gpqa: A graduate-level google-proof q&a benchmark. Preprint, arXiv:2311.12022.
  29. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  30. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, and 1 others. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297.
  31. Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, and 1 others. 2025. Too consistent to detect: A study of self-consistent errors in llms. Preprint, arXiv:2505.17656.
  32. Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen, and Tony Mak. 2024. Llms cannot find reasoning errors, but can correct them given the error location. Preprint, arXiv:2311.08516.
  33. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, and 1 others. 2025. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. Preprint, arXiv:2504.20073. find
  34. Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, and 1 others. 2025. Toward training superintelligent software agents through self-play swe-rl. Preprint, arXiv:2512.18552. — [arXiv:2512.18552](https://arxiv.org/abs/2512.18552) find
  35. Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, and 1 others. 2025. Agent0: Unleashing self-evolving agents from zero data via tool-integrated reasoning. Preprint, arXiv:2511.16043. — [arXiv:2511.16043](https://arxiv.org/abs/2511.16043) find
  36. Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. 2025. Self-rewarding correction for mathematical reasoning. Preprint, arXiv:2502.19613. — [arXiv:2502.19613](https://arxiv.org/abs/2502.19613) find
  37. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, and 1 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  38. OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, and 1 others. 2024. Openai o1 system card. Preprint, arXiv:2412.16720.
  39. Ziyi Yang, Weizhou Shen, Ruijun Chen, Chenliang Li, Fanqi Wan, and 1 others. 2025b. Spell: Self-play reinforcement learning for evolving long-context language models. Preprint, arXiv:2509.23863. — [arXiv:2509.23863](https://arxiv.org/abs/2509.23863) find
  40. Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. 2025. Maximizing confidence alone improves reasoning. Preprint, arXiv:2505.22660. — [arXiv:2505.22660](https://arxiv.org/abs/2505.22660) find
  41. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, and 1 others. 2025a. Dapo: An open-source llm reinforcement learning system at scale. Preprint, arXiv:2503.14476.
  42. Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, and 1 others. 2025. Self-consistency preference optimization. Preprint, arXiv:2411.04109. find
  43. Zhaoning Yu, Will Su, Leitian Tao, Haozhu Wang, Aashu Singh, and 1 others. 2025b. Restrain: From spurious votes to signals – self-driven rl with self-penalization. Preprint, arXiv:2510.02172. — [arXiv:2510.02172](https://arxiv.org/abs/2510.02172) find
  44. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2025. Self-rewarding language models. Preprint, arXiv:2401.10020. find
  45. Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-Wei Hong, Duane S. Boning, and Dina Katabi. 2025. Rl tango: Reinforcing generator and verifier together for language reasoning. Preprint, arXiv:2505.15034. — [arXiv:2505.15034](https://arxiv.org/abs/2505.15034) find
  46. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2025a. Generative verifiers: Reward modeling as next-token prediction. Preprint, arXiv:2408.15240.
  47. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, and 1 others. 2025b. The lessons of developing process reward models in mathematical reasoning. Preprint, arXiv:2501.07301.
  48. Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, and 1 others. 2025a. Absolute zero: Reinforced self-play reasoning with zero data. Preprint, arXiv:2505.03335. find
  49. Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. 2025b. Learning to reason without external rewards. Preprint, arXiv:2505.19590. find
  50. Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, and 1 others. 2025. Evolving language models without labels: Majority drives selection, novelty promotes variation. Preprint, arXiv:2509.15194. — [arXiv:2509.15194](https://arxiv.org/abs/2509.15194) find
  51. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, and 1 others. 2025. Ttrl: Test-time reinforcement learning. Preprint, arXiv:2504.16084. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2026-coverrl-breaking-consensus-trap-label/