Elamur Research⌕
☰
home / papers / 2026 / Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

Paper · 2026 · Weiqin Wang, Yile Wang, Kehao Chen et al.

Summary

Test-time reinforcement learning (TTRL) trains LLMs on reasoning tasks without gold answers. It samples many responses and treats the majority-voted answer as a pseudo-label. This has two weaknesses. Every vote counts equally, so a popular but wrong answer gets reinforced (confirmation bias), and the binary match against one shared label gives sparse rewards. The authors propose SCOPE to address both. First, each response gets a weight from its average step confidence: the response is split into steps at blank lines, token-level top-k log-probability confidence is averaged within each step, and those step scores are averaged across the response. Weighted voting then lets a well-supported minority answer win. Second, the rollout pool is split into subgroups, each with its own local consensus label. The subgroup size is chosen during training by Pareto optimization, which balances label quality against exploration diversity. Training uses GRPO. Across LLaMA3.1, Qwen2.5 and Qwen3 models, SCOPE beats recent unsupervised baselines on math benchmarks. This shows that finer-grained, confidence-aware pseudo-rewards make label-free RL more reliable.

Key points

  • SCOPE gives relative gains of 13.1% on AIME 2025 and 8.1% on AMC over baselines. On Qwen3-8B it improves AIME 2024 by 10.48% and averages a 6.85% gain across benchmarks.
  • Pseudo-label accuracy (PLA) on Qwen2.5-Math-1.5B rises from 25.42 (TTRL) to 30.71 with automatic subgroup sizing, a 20.81% relative improvement. Fixed sizes perform worse (m=1: 26.75; m=64: 29.57).
  • With very small subgroups, labels are noisy. A single global group (m=64) is stable but converges slowly. Adaptive sizing converges fastest and reaches the highest peak.
  • Known limitations: step segmentation relies on a newline heuristic, and the Pareto subgroup search adds roughly 10% compute overhead.
  • The code is released at github.com/szu-tera/SCOPE (Shenzhen University, ACL 2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shenzhen UniversityShenzhen · China16 works · 95 citations🎓Fuzhou UniversityFuzhou · China3 works · 17 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Weiqin Wang, Yile Wang, Kehao Chen and Hui Huang. 2026. Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1727

@inproceedings{wang2026beyond,
  title = {{Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning}},
  author = {Weiqin Wang, Yile Wang, Kehao Chen and Hui Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1727},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1727}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. 2020. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE. find
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  3. Yung-Sung Chuang, Benjamin Cohen-Wang, Zejiang Shen, Zhaofeng Wu, Hu Xu, Xi Victoria Lin, James R. Glass, Shang-Wen Li, and Wen tau Yih. 2025. SelfCite: Self-supervised alignment for context attribution in large language models. In Forty-second International Conference on Machine Learning. find
  4. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  5. Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence. arXiv preprint arXiv:2508.15260. find
  6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
  8. Weiyang Huang, Xuefeng Bai, Kehai Chen, Xinyang Chen, Yibin Chen, Weili Guan, and Min Zhang. 2026. Sat: Balancing reasoning accuracy and efficiency with stepwise adaptive thinking. arXiv:2604.07922. arXiv preprint Vienna, Austria. Association for Computational Linguistics.
  9. Ke Ji, Junying Chen, Anningzhe Gao, Wenya Xie, Xiang Wan, and Benyou Wang. 2025. Unlocking LLMs’ self-improvement capacity with autonomous learning for domain adaptation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21051–21067, Vienna, Austria. Association for Compu… find
  10. Xinyu Tang, Zhenduo Zhang, Yurou Liu, Wayne Xin Zhao, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. 2025. Towards high data efficiency in reinforcement learning with verifiable reward. arXiv preprint arXiv:2509.01321. — [arXiv:2509.01321](https://arxiv.org/abs/2509.01321) find
  11. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9. find
  12. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  13. Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, and Shuangzhi Wu. 2025. AdaCoT: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning. arXiv preprint arXiv:2505.11896. — [arXiv:2505.11896](https://arxiv.org/abs/2505.11896) find
  14. OpenAI. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  15. Vilfredo Pareto. 1964. Cours d’économie politique, volume 1. Librairie Droz. find
  16. Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. 2021. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8558–8567. find
  17. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  18. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In EuroSys, pages 1279– 1297.
  19. Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. 2025. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829. — [arXiv:2503.23829](https://arxiv.org/abs/2503.23829) find
  20. Yutao Sun, Mingshuai Chen, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, and Jianwei Yin. 2025. The self-improvement paradox: Can language models bootstrap reasoning capabilities without external scaffolding? In Findings of the Association for Computational Linguistics: ACL 2025, pages 6501–6512, Carel v… find
  21. Pinzheng Wang, Juntao Li, Zecheng Tang, Haijia Gui, and Min zhang. 2025a. Improving rationality in the reasoning process of language models through self-playing game. In Forty-second International Conference on Machine Learning. find
  22. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025b. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforce…
  23. Weiqin Wang, Yile Wang, and Hui Huang. 2025c. Ranked voting based self-consistency of large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14410– 14426, Vienna, Austria. Association for Computational Linguistics. find
  24. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and yelong shen. 2025d. Reinforcement learning for reasoning in large language models with one training example. In The Thirty-nint… find
  25. Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. 2025. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245. find
  26. Mian Wu, Gavin Zhang, Sewon Min, Sergey Levine, and Aviral Kumar. 2025. RLAC: Reinforcement learning with adversarial critic for free-form generation tasks. arXiv preprint arXiv:2511.01758. — [arXiv:2511.01758](https://arxiv.org/abs/2511.01758) find
  27. Mufan Xu, Kehai Chen, Xuefeng Bai, Zhengyu Niu, Muyun Yang, Tiejun Zhao, and Min Zhang. 2026. Beyond token-level policy gradients for complex reasoning with large language models. arXiv preprint arXiv:2602.14386. — [arXiv:2602.14386](https://arxiv.org/abs/2602.14386) find
  28. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  29. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.
  30. Chenlu Ye, Zhou Yu, Ziji Zhang, Hao Chen, Narayanan Sadagopan, Jing Huang, Tong Zhang, and Anurag Beniwal. 2025. Beyond correctness: Harmonizing process and outcome rewards through rl training. arXiv preprint arXiv:2509.03403. — [arXiv:2509.03403](https://arxiv.org/abs/2509.03403) find
  31. Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al. 2025. RLPR: Extrapolating rlvr to general domains without verifiers. arXiv preprint arXiv:2506.18254. — [arXiv:2506.18254](https://arxiv.org/abs/2506.18254) find
  32. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning. find
  33. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun MA, and Junxian He. 2025. SimpleRL-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. In Second Conference on Language Modeling.
  34. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. 2025. TTRL: Test-time reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Pr… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-beyond-majority-voting-fine-grained/