Elamur Research⌕
☰
home / papers / 2026 / Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning

Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning

Paper · 2026 · Zelin Tan, Hejia Geng, Xiaohang Yu et al.

Summary

Scaling laws are well established for LLM pre-training, but little is known about how they behave when reinforcement learning is applied after pre-training. This study addresses that gap for mathematical reasoning. The authors trained 63 models from the Qwen2.5 family, ranging from 0.5B to 72B parameters and covering both base and instruct variants. Training used GRPO in the VeRL framework on more than 50k math problems from the guru-RL-92k collection, ordered from easiest to hardest. Performance was measured as test loss, defined as one minus the pass rate. They found that the logarithm of test loss falls linearly with the logarithm of compute or data. The slope of that line, which measures learning efficiency, improves with model size but levels off toward an upper bound. When data is limited, the total number of training samples seen matters more than how many of them are unique, so reusing data is a practical option. Fitting the law on early training steps or on smaller models predicts later results well. This helps teams plan RL budgets before committing compute.

Key points

  • The proposed law is log L = −k(N)·log X + E(N), where X is either compute (FLOPs) or data. The efficiency term k(N) saturates toward a ceiling K_max as model size N grows.
  • Coefficients fitted on the 0.5B–32B models correctly forecast the efficiency of the held-out 72B model. Fits on early training steps predict a model's later loss trajectory.
  • Evaluation used two kinds of data:
    • a held-out in-domain set of 500 math problems;
    • transfer benchmarks: AIME2024, AMC2023, GSM8K, MATH500, HumanEval, Zebra Puzzle and SuperGPQA.
  • Training rewards were binary, based on whether the final answer matched. Larger models were more compute- and data-efficient, and the findings also held on Llama 3.
  • The authors note several limitations:
    • only math-domain RL was studied;
    • no models above 72B were tested;
    • only dense architectures were covered, so MoE models remain untested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations🎓Imperial College LondonLondon · United Kingdom48 works · 12,181 citations🎓Wuhan UniversityWuhan · China35 works · 1,127 citations🎓University of GeorgiaAthens · United States17 works · 26,331 citations🎓Dalian University of TechnologyDalian · China12 works · 46 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Zelin Tan, Hejia Geng, Xiaohang Yu, Mulei Zhang, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhong-Zhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang, Zhenfei Yin, Philip Torr and Lei Bai. 2026. Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1444

@inproceedings{tan2026scaling,
  title = {{Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning}},
  author = {Zelin Tan, Hejia Geng, Xiaohang Yu, Mulei Zhang, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhong-Zhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang, Zhenfei Yin, Philip Torr and Lei Bai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1444},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1444}
}

References

58 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages…
  2. Yuchen Cai, Ding Cao, Xin Xu, Zijun Yao, Yuqing Huang, Zhenyu Tan, Benyi Zhang, Guangzhong Sun, Guiquan Liu, and Junfeng Fang. 2025. On predictability of reinforcement learning dynamics for large language models. arXiv preprint arXiv:2510.00553. — [arXiv:2510.00553](https://arxiv.org/abs/2510.00553) find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  4. Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, and 5 others. 2025. Revisiting reinforcement learning for llm reason… — [arXiv:2506.14965](https://arxiv.org/abs/2506.14965) find
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  6. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. 2025. The entropy mechanism of reinforcement learning for reasoning language models. Preprint, a… find
  7. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Yuchen Fan*, Kaiyan Zhang*, Heng Zhou*, Yuxin Zuo, Yanxu Chen, Yu Fu, Xuekai Zhu, Che Jiang, Yuchen Zhang, Li Kang, Gang Chen, Cheng Huang, Zhizhou He, Bingning Wang, Lei Bai, Ning Ding, and Bowen Zhou. 2025. Ssrl: Self-search reinforcement learning. find
  9. Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. 2025. From llm reasoning to autonomous ai agents: A comprehensive review. arXiv preprint arXiv:2504.19678. find
  10. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, and 1 others. 2022. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487.
  11. Jacob Hilton, Jie Tang, and John Schulman. 2023. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301.13442. find
  12. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  13. Sam Houliston, Ambroise Odonnat, Charles Arnal, and Vivien Cabannes. 2025. Provable benefits of in-tool learning for large language models. Preprint, arXiv:2508.20755. — [arXiv:2508.20755](https://arxiv.org/abs/2508.20755) find
  14. Yichen Huang and Lin F. Yang. 2025. Gemini 2.5 pro capable of winning gold at imo 2025. Preprint, arXiv:2507.15855. — [arXiv:2507.15855](https://arxiv.org/abs/2507.15855) find
  15. Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. 2024. Scaling laws for downstream task performance in machine translation. arXiv preprint arXiv:2402.04177. — [arXiv:2402.04177](https://arxiv.org/abs/2402.04177) find
  16. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, and 1 others. 2021. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583–589.
  17. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2023. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358–50376. find
  18. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  19. Shuaiyi Nie, Siyu Ding, Wenyuan Zhang, Linhao Yu, Tianmeng Yang, Yao Chen, Tingwen Liu, Weichong Yin, Yu Sun, and Hua Wu. 2026. Attnpo: Attention-guided process supervision for efficient reasoning. arXiv preprint arXiv:2602.09953.
  20. Kimi Team. 2025. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599.
  21. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b and gpt-oss-20b mod…
  22. KnovelEng. 2025. Knovel engineering amc-23 dataset. https://huggingface.co/datasets/ knoveleng/AMC-23. Accessed: 2025-09-23. ↗
  23. Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. 2025a. (mis)fitting scaling laws: A survey of scaling law fitting techniques in deep learning. In The Thirteenth International Conference on Learning Representations. find
  24. Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025b. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886. find
  25. Yu Li, Zhuoshi Pan, Honglin Lin, Mengyuan Sun, Conghui He, and Lijun Wu. 2025c. Can one domain help others? a data-centric study on multi-domain reasoning via reinforcement learning. arXiv preprint arXiv:2507.17512. — [arXiv:2507.17512](https://arxiv.org/abs/2507.17512) find
  26. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. Preprint, arXiv:2305.20050.
  27. Bill Yuchen Lin. 2024. ZeroEval: A Unified Framework for Evaluating Language Models. find
  28. Heng Lin and Zhongwen Xu. 2025. Understanding tool-integrated reasoning. Preprint, arXiv:2508.19201. — [arXiv:2508.19201](https://arxiv.org/abs/2508.19201) find
  29. Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. 2025. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864. find
  30. Rohan Pandey. 2024. gzip predicts data-dependent scaling laws. arXiv preprint arXiv:2405.16684. — [arXiv:2405.16684](https://arxiv.org/abs/2405.16684) find
  31. Bhrij Patel, Souradip Chakraborty, Wesley A. Suttle, Mengdi Wang, Amrit Singh Bedi, and Dinesh Manocha. 2024. Aime: Ai system optimization via multiple llm evaluators. Preprint, arXiv:2410.03131. — [arXiv:2410.03131](https://arxiv.org/abs/2410.03131) find
  32. Tim Pearce and Jinyeop Song. 2024. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406.12907. — [arXiv:2406.12907](https://arxiv.org/abs/2406.12907) find
  33. Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. 2024. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37:100535–100570.
  34. Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R Fung, and 1 others. 2025. Scaling laws of synthetic data for language models. arXiv preprint arXiv:2503.19551. — [arXiv:2503.19551](https://arxiv.org/abs/2503.19551) find
  35. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. find
  36. Xinji Mai, Haotian Xu, Zhong-Zhi Li, Xing W, Weinong Wang, Jian Hu, Yingying Zhang, and Wenqiang Zhang. 2025a. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. Preprint, arXiv:2505.07773. — [arXiv:2505.07773](https://arxiv.org/abs/2505.07773) find
  37. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  38. Xinji Mai, Haotian Xu, Weinong Wang, Jian Hu, Yingying Zhang, Wenqiang Zhang, and 1 others. 2025b. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773. — [arXiv:2505.07773](https://arxiv.org/abs/2505.07773) find
  39. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256.
  40. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2017. Mastering chess and shogi by self-play with a general reinforcement learning alg…
  41. Haohan Yuan and Haopeng Zhang. 2025. Understanding llm reasoning for abstractive summarization. Preprint, arXiv:2512.03503.
  42. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
  43. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837.
  44. Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. 2022. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523–19536.
  45. Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA. find
  46. P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, and 78 others. 2025. Supergpqa: Scaling llm evaluation across 285 gradu… find
  47. Hugo Touvron, Albert Q. Jiang, Nan Du, and et al. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825.
  48. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, and 1 others. 2025. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571. find
  49. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  50. Xiangyuan Xue, Yifan Zhou, Guibin Zhang, Zaibin Zhang, Yijiang Li, Chen Zhang, Zhenfei Yin, Philip Torr, Wanli Ouyang, and Lei Bai. 2026. Comas: Co-evolving multi-agent systems via interaction rewards. Preprint, arXiv:2510.08529. — [arXiv:2510.08529](https://arxiv.org/abs/2510.08529) find
  51. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  52. Qiwei Ye, Chenxin Qian, Qingxiu Song, and et al. 2024. Skywork-math: Data scaling laws for mathematical reasoning in large language models — the story goes on. arXiv preprint arXiv:2407.08348.
  53. Xiang Yue, Fanghua Liu, Yuxuan Zhang, and et al. 2023. Mammoth: Building math generalist models. arXiv preprint arXiv:2309.05653.
  54. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025a. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.
  55. Yifan Zeng, Tianyu Guo, Yuqing Wang, and et al. 2025b. Agent rl scaling law: Spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773. — [arXiv:2505.07773](https://arxiv.org/abs/2505.07773) find
  56. Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, and 6 others. 2025a. The landscape of agentic reinforcement lea… — [arXiv:2509.02547](https://arxiv.org/abs/2509.02547) find
  57. Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, and 20 others. 2025b. A survey of reinforcement learning for large reasoning… — [arXiv:2509.08827](https://arxiv.org/abs/2509.08827) find
  58. Heng Zhou*, Hejia Geng*, Xiangyuan Xue, Li Kang, Yiran Qin, Zhiyong Wang, Zhenfei Yin, and Lei Bai. 2025. Reso: A reward-driven self-organizing llm-based multi-agent system for reasoning tasks. EMNLP2025 SAC Highlight. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tan-2026-scaling-behaviors-llm-reinforcement-learning/