Elamur Research⌕
☰
home / papers / 2026 / What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

Paper · 2026 · Dong Yan, Jian Liang, Yanbo Wang et al.

Summary

Test-Time Reinforcement Learning (TTRL) lets LLMs improve their reasoning on unlabeled test questions by using the majority-voted answer among sampled rollouts as a pseudo-reward. The authors argue that this approach fails on hard problems or with small rollout budgets. When answers are widely scattered, the "winning" answer is often wrong, and GRPO's group normalization amplifies that error.

They propose SCRL, a GRPO-based framework with three parts:

  • Selective Positive Pseudo-Labeling assigns a positive label only when the top answer has sufficient support (threshold τpos) and a clear lead over the runner-up (τmarg). Otherwise it abstains.
  • Entropy-Gated Negative Pseudo-Labeling, presented as the first negative supervision in TTRL, penalizes answers that are both infrequent and generated with high uncertainty. Rare but potentially correct answers are kept.
  • Dynamic Reward Shaping scales rewards by consensus strength and penalizes uncertain trajectories.

On mathematical reasoning benchmarks, SCRL outperforms TTRL-style baselines, especially on difficult sets and under tight rollout budgets. This makes label-free adaptation of reasoning models more reliable.

Key points

  • On Qwen3-4B with thinking mode on AIME25, SCRL beats TTRL by +2.7 points at a 10,240-token generation limit (53.6 vs 50.9 pass@1). At 15,360 tokens the gain is +0.9 (60.2 vs 59.3).
  • The entropy analysis with Qwen2.5-3B shows that incorrect trajectories consistently have higher average entropy than correct ones. On AIME24 the values are 0.288 vs 0.208, which supports using uncertainty to gate negative labels.
  • In the parameter study on Qwen2.5-Math-7B, τpos=0.375 with τneg=0.125 gave the best results: AIME25 22.8, AMC 67.5, MATH-500 86.2.
  • A noted failure case is Minerva with Qwen2.5-3B, where SCRL trails TTRL. The answers are so fragmented that the strict thresholds rarely allow positive labels.
  • Stated limitations are the lack of theoretical guarantees and limited baseline comparison, since related methods are hard to reproduce. Code is available at github.com/Jasper-Yan/SCRL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Dong Yan, Jian Liang, Yanbo Wang, Shuo Lu, Ran He and Tieniu Tan. 2026. What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1337

@inproceedings{yan2026what,
  title = {{What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time}},
  author = {Dong Yan, Jian Liang, Yanbo Wang, Shuo Lu, Ran He and Tieniu Tan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1337},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1337}
}

References

49 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  2. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, and 1 others. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456.
  3. Yunzhen Feng, Parag Jain, Anthony Hartshorn, Yaqi Duan, and Julia Kempe. 2025. Don’t waste mistakes: Leveraging negative rl-groups via confidence reweighting. arXiv preprint arXiv:2510.08696. — [arXiv:2510.08696](https://arxiv.org/abs/2510.08696) find
  4. Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. 2024. On designing effective rl reward at training time for llm reasoning. arXiv preprint arXiv:2410.15115. — [arXiv:2410.15115](https://arxiv.org/abs/2410.15115) find
  5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  7. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Proc. NeurIPS.
  8. Siyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng, Weiqiang Wang, and Zhouhan Lin. 2024. Mirror-consistency: Harnessing inconsistency in majority voting. In Proc. EMNLP. find
  9. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  10. Dulhan Jayalath, Shashwat Goel, Thomas Foster, Parag Jain, Suchin Gururangan, Cheng Zhang, Anirudh Goyal, and Alan Schelten. 2025. Compute as teacher: Turning inference compute into reference-free supervision. arXiv preprint arXiv:2509.14234. — [arXiv:2509.14234](https://arxiv.org/abs/2509.14234) find
  11. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, and 1 others. 2024. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124.
  12. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, and 1 others. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):… find
  13. Zhiheng Li, Zongyang Ma, Yuntong Pan, Ziqi Zhang, Xiaolei Lv, Bo Li, Jun Gao, Jianing Zhang, Chunfeng Yuan, Bing Li, and Weiming Hu. 2026. Making mllms blind: Adversarial smuggling attacks in mllm content moderation. Preprint, arXiv:2604.06950. find
  14. Jian Liang, Ran He, and Tieniu Tan. 2025. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, 133(1):31–64. find
  15. Jia Liu, ChangYi He, YingQiao Lin, MingMin Yang, FeiYang Shen, and ShaoGuo Liu. 2025a. Ettrl: Balancing exploration and exploitation in llm test-time reinforcement learning via entropy mechanism. arXiv preprint arXiv:2508.11356. — [arXiv:2508.11356](https://arxiv.org/abs/2508.11356) find
  16. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025b. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  17. Shuo Lu, Yanyin Chen, Wei Feng, Jiahao Fan, Fengheng Li, Zheng Zhang, Jingjing Lv, Junjie Shen, Ching Law, and Jian Liang. 2025a. Uni-layout: Integrating human feedback in unified layout generation and evaluation. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 7709–7718… find
  18. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297.
  19. Shuo Lu, Yinuo Xu, Jianjie Cheng, Lingxiao He, Meng Wang, and Jian Liang. 2025b. Deepresearch-slice: Bridging the retrieval-utilization gap via explicit text slicing. arXiv preprint arXiv:2601.03261. — [arXiv:2601.03261](https://arxiv.org/abs/2601.03261) find
  20. Wenlei Shi and Xing Jin. 2025. Heimdall: test-time scaling on the generative verification. arXiv preprint arXiv:2504.10337. — [arXiv:2504.10337](https://arxiv.org/abs/2504.10337) find
  21. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In Proc. NeurIPS. find
  22. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In Proc. NeurIPS.
  23. Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. 2025. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660. — [arXiv:2505.22660](https://arxiv.org/abs/2505.22660) find
  24. Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. 2025. Self-consistency preference optimization. In Proc. ICML. find
  25. Felix Stahlberg, Ilia Kulikov, and Shankar Kumar. 2022. Uncertainty determines the adequacy of the mode and the tractability of decoding in sequence-to-sequence models. In Proc. ACL. find
  26. Chuyi Tan, Peiwen Yuan, Xinglin Wang, Yiwei Li, Shaoxiong Feng, Yueqi Zhang, Jiayi Shi, Ji Zhang, Boyuan Pan, Yao Hu, and 1 others. 2025. Diagnosing and mitigating system bias in self-rewarding rl. arXiv preprint arXiv:2510.08977. — [arXiv:2510.08977](https://arxiv.org/abs/2510.08977) find
  27. Chenwei Tang, Jingyu Xing, Lin Long, Xinyu Liu, Deng Xiong, Wei Ju, Shudong Huang, Jiancheng Lv, and Ziyue Qiao. 2025. Rewarding the journey, not just the destination: A composite path and answer self-scoring reward mechanism for test-time reinforcement learning. arXiv preprint arXiv:2510.17923. — [arXiv:2510.17923](https://arxiv.org/abs/2510.17923) find
  28. Qwen Team and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2(3). find
  29. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Proc. NeurIPS.
  30. Junqiao Wang, Zeng Zhang, Yangfan He, Zihao Zhang, Xinyuan Song, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, and 1 others. 2024. Enhancing code llms with reinforcement learning in code generation: A survey. arXiv preprint arXiv:2412.20367. — [arXiv:2412.20367](https://arxiv.org/abs/2412.20367) find
  31. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First conference on language modeling.
  32. Ru Wang, Wei Huang, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo, and Jiaxian Guo. 2025a. Self-harmony: Learning to harmonize self-supervision and self-play in test-time reinforcement learning. arXiv preprint arXiv:2511.01191. — [arXiv:2511.01191](https://arxiv.org/abs/2511.01191) find
  33. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  34. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, and 1 others. 2025b. Reinforcement learning for reasoning in large language models with one training example. In Proc. NeurIPS. find
  35. Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. In Proc. NeurIPS. find
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  37. Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. 2025. Can large reasoning models self-train? arXiv preprint arXiv:2505.21444. find
  38. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  39. Lai Wei, Yuting Li, Chen Wang, Yue Wang, Linghe Kong, Weiran Huang, and Lichao Sun. 2025. Unsupervised post-training for multi-modal llm reasoning via grpo. In Proc. NeurIPS. find
  40. Wen Wen, Tianwu Zhi, Kanglong Fan, Yang Li, Xinge Peng, Yabin Zhang, Yiting Liao, Junlin Li, and Li Zhang. 2025. Self-evolving vision-language models for image quality assessment via voting and ranking. arXiv preprint arXiv:2509.25787. — [arXiv:2509.25787](https://arxiv.org/abs/2509.25787) find
  41. Jianghao Wu, Yasmeen George, Jin Ye, Yicheng Wu, Daniel F Schmidt, and Jianfei Cai. 2025. Spine: Token-selective test-time reinforcement learning with entropy-band regularization. arXiv preprint arXiv:2511.17938. — [arXiv:2511.17938](https://arxiv.org/abs/2511.17938) find
  42. Dong Yan, Gaochen Wu, and Bowen Zhou. 2025. Mission impossible: Feedback-guided dynamic interactive planning for improving reasoning on llms. arXiv preprint arXiv:2510.05577. — [arXiv:2510.05577](https://arxiv.org/abs/2510.05577) find
  43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  44. Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. 2025b. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590. find
  45. Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang, Haitao Mi, and Dong Yu. 2025. Evolving language models without labels: Majority drives selection, novelty promotes variation. arXiv preprint arXiv:2509.15194. — [arXiv:2509.15194](https://arxiv.org/abs/2509.15194) find
  46. Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. 2025. The surprising effectiveness of negative reinforcement in llm reasoning. In Proc. NeurIPS. find
  47. Ziyi Yang, Weizhou Shen, Ruijun Chen, Chenliang Li, Fanqi Wan, Ming Yan, Xiaojun Quan, and Fei Huang. 2025b. Spell: Self-play reinforcement learning for evolving long-context language models. arXiv preprint arXiv:2509.23863. — [arXiv:2509.23863](https://arxiv.org/abs/2509.23863) find
  48. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, and 1 others. 2025. Ttrl: Test-time reinforcement learning. In Proc. NeurIPS. find
  49. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025a. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yan-2026-what-if-consensus-lies-selective/