Elamur Research⌕
☰
home / papers / 2026 / Better LLM Reasoning via Dual-Play

Better LLM Reasoning via Dual-Play

Paper · 2026 · Zhengxin Zhang, Chengyu Huang, Aochong Oliver Li et al.

Summary

Reinforcement learning with verifiable rewards (RLVR) has improved LLM reasoning, but it depends on large sets of curated problems with correct answers. This paper presents PasoDoble, a dual-play framework that reduces that dependence. Two copies of the same base model are trained against each other. The Proposer writes math questions with reference answers, grounded in raw text sampled from a pre-training corpus. The Solver tries each question several times.

The Proposer earns more when the Solver's pass rate is lower. It also earns a diversity bonus based on Jaccard similarity to recent questions. Questions that are almost never solved, or that are too repetitive, get zero reward, which discourages reward hacking. The authors test an online variant, where both models update synchronously, and an offline variant that alternates phases through a question buffer.

On standard math benchmarks, both variants improve Qwen2.5 and Qwen3 base models, and the gains are larger for bigger models. Unlike R-Zero, PasoDoble keeps improving over hundreds of update steps. This makes it a promising way to strengthen reasoning with minimal labelled data.

Key points

  • Evaluation covers AIME, AMC, GSM8K, MATH-500 and OlympiadBench, using Qwen2.5-0.5B/1.5B/3B-Base and Qwen3-0.6B/1.7B/4B-Base.
  • Average pass@1 gains:
    • Qwen3-1.7B-Base: +11.42 online and +12.75 offline.
    • Qwen3-4B-Base: +15.96 online and +15.10 offline.
  • The Proposer's difficulty reward is defined as 1.1 minus the Solver's pass rate, so even easy but valid questions still beat invalid ones.
  • Ablations on Qwen3-1.7B (online), average accuracy relative to the full method's 38.27:
    • Freezing the Proposer lowers performance.
    • Dropping the diversity reward costs about 4 points (34.67).
    • Replacing the Proposer's answers with random rewards collapses accuracy to near zero (0.75).
    • The knowledge base helps larger models but hurts the 0.5B/0.6B models.
  • Limitations:
    • Gains are smaller for small models.
    • There is no clear improvement on out-of-domain GPQA.
    • Training still saturates after a few hundred steps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States134 works · 65,159 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Zhengxin Zhang, Chengyu Huang, Aochong Oliver Li and Claire Cardie. 2026. Better LLM Reasoning via Dual-Play. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1752

@inproceedings{zhang2026better,
  title = {{Better LLM Reasoning via Dual-Play}},
  author = {Zhengxin Zhang, Chengyu Huang, Aochong Oliver Li and Claire Cardie},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1752},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1752}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. 2025. The unreasonable effectiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505.15134. find
  2. Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan-Yee K Wong. 2025a. Spc: Evolving self-play critic via adversarial games for llm reasoning. arXiv preprint arXiv:2504.19162. — [arXiv:2504.19162](https://arxiv.org/abs/2504.19162) find
  3. Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. 2025b. Self-questioning language models. arXiv preprint arXiv:2508.03682. — [arXiv:2508.03682](https://arxiv.org/abs/2508.03682) find
  4. Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557–570.
  5. Pengyu Cheng, Yong Dai, Tianhao Hu, Han Xu, Zhisong Zhang, Lei Han, Nan Du, and Xiaolong Li. 2024. Self-playing adversarial language game enhances llm reasoning. In Advances in Neural Information Processing Systems, volume 37, pages 126515–126543. find
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  7. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Nature, 645:633–638.
  8. Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, and 1 others. 2025. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739.
  9. Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, and Dacheng Tao. 2025. Serl: Self-play reinforcement learning for large language models with limited data. arXiv preprint arXiv:2505.20347. — [arXiv:2505.20347](https://arxiv.org/abs/2505.20347) find
  10. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. In Advances in Neural Information Processing Systems. find
  11. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Me…
  12. Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. 2025a. Efficient test-time scaling via self-calibration. arXiv preprint arXiv:2503.00031. — [arXiv:2503.00031](https://arxiv.org/abs/2503.00031) find
  13. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. 2025b. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004. — [arXiv:2508.05004](https://arxiv.org/abs/2508.05004) find
  14. Jakub Grudzien Kuba, Mengting Gu, Qi Ma, Yuandong Tian, and Vijai Mohan. 2025. Language self-play for data-free training. arXiv preprint arXiv:2509.07414. — [arXiv:2509.07414](https://arxiv.org/abs/2509.07414) find
  15. Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. 2025. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506.06395. — [arXiv:2506.06395](https://arxiv.org/abs/2506.06395) find
  16. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  17. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04), pages 605–612.
  18. Weili Nie, Nina Narodytska, and Ankit Patel. 2019. Relgan: Relational generative adversarial networks for text generation. In International Conference on Learning Representations. find
  19. Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. 2017. Robust adversarial reinforcement learning. In International Conference on Machine Learning. find
  20. Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. 2025. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660. — [arXiv:2505.22660](https://arxiv.org/abs/2505.22660) find
  21. Qwen. 2025a. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  22. Qwen. 2025b. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  23. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  24. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  25. Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, and 1 others. 2025. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. arXiv preprint arXiv:2507.10532. — [arXiv:2507.10532](https://arxiv.org/abs/2507.10532) find
  26. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In The Thirty-First AAAI Conference on Artificial Intelligence. find
  27. Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017. Adversarial feature matching for text generation. In International Conference on Machine Learning. find
  28. Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. 2025. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335.
  29. Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. 2025. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.109…
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024a. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  31. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. 2024b. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  32. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018. A general reinforcement learning algorithm that masters chess, shogi, and go thr… find
  33. Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. 2025. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829. — [arXiv:2503.23829](https://arxiv.org/abs/2503.23829) find
  34. Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. 2025. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512. — [arXiv:2506.20512](https://arxiv.org/abs/2506.20512) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-better-llm-reasoning-dual-play/