Elamur Research⌕
☰
home / papers / 2025 / GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models

GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models

Paper · 2025 · Jixiao Zhang and Chunsheng Zuo

Summary

The paper targets three weaknesses of Group Relative Policy Optimization (GRPO), the RL method behind R1-like math reasoners. Binary correctness rewards give sparse signals, they do not discourage overly long chains of thought, and they treat easy and hard problems the same. The authors propose GRPO-LEAD, which changes the reward and the advantage calculation in three ways:

  • Length-dependent reward. A correct answer is rewarded according to how short it is compared with the other correct answers in its group.
  • Explicit penalty. A wrong answer gets a negative reward instead of zero, so guessing no longer pays off.
  • Difficulty reweighting. Advantages are rescaled with a logistic function of each question's empirical solve rate, which gives harder questions more weight.

Ablations at the 7B scale indicate that each component helps. At 14B, the final model is compared with DeepSeek-14B and Light-R1-14B-DS on AIME24 and AIME25 under a 14k-token budget. It reaches the best Pass@1 on both benchmarks while producing noticeably shorter solutions. The work shows that careful reward shaping can make math-reasoning models both more accurate and more efficient at modest compute cost. Code, generated data and models are released publicly.

Key points

  • Reward design.

    • A correct response receives exp(−αz), where z is its standardized token length among the group's correct responses.
    • An incorrect response receives −1.
    • With this scheme the expected reward is roughly 2P(correct)−1, so it turns positive only when P(correct) > 0.5.

  • Ablation of the explicit penalty (7B scale).

    • On AIME24, Pass@1 rose from 0.458 to 0.470 and Cons@32 from 0.767 to 0.800.
    • On AIME25, Pass@1 rose from 0.325 to 0.345.
    • The cost was longer outputs on AIME24, from about 5,300 to 6,104 tokens.

  • LEAD-stage2 (14B) vs. the strongest baseline, Light-R1-14B-DS:

    BenchmarkCons@32Pass@1Avg. length (tokens)
    AIME240.867 (vs. 0.833)0.650 (vs. 0.641)8,267 (vs. 9,571)
    AIME250.767 (tie)0.539 (vs. 0.505)8,668 (vs. 10,194)

  • Training efficiency. LEAD-stage1 trained for only 100 steps, about 24 hours on eight H20 GPUs. It already matches Light-R1-14B-DS on Cons@32 and beats it on AIME25 Pass@1.

  • Limitations. Results cover only mathematical reasoning. The authors did no formal hyperparameter search, and some comparisons (the full GRPO baseline, one 7B validation curve) are missing because of limited compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputational Theory and Mathematics · 252 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Jixiao Zhang and Chunsheng Zuo. 2025. GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.287

@inproceedings{zhang2025grpo,
  title = {{GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models}},
  author = {Jixiao Zhang and Chunsheng Zuo},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.287},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.287}
}

References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranjal Aggarwal and Sean Welleck. 2025. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697.
  2. Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. 2025. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546. — [arXiv:2504.02546](https://arxiv.org/abs/2504.02546) find
  3. Muzhi Dai, Chenxu Yang, and Qingyi Si. 2025. S-grpo: Early exit via reinforcement learning in reasoning models. arXiv preprint arXiv:2505.07686. — [arXiv:2505.07686](https://arxiv.org/abs/2505.07686) find
  4. Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. 2017. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417. — [arXiv:1705.08417](https://arxiv.org/abs/1705.08417) find
  5. Simon Frieder, Sam Bealing, Arsenii Nikolaiev, Geoff C. Smith, Kevin Buzzard, Timothy Gowers, Peter J. Liu, Po-Shen Loh, Lester Mackey, Leonardo de Moura, Dan Roberts, D. Sculley, Terence Tao, David Balduzzi, Simon Coyle, Alex Gerko, Ryan Holbrook, Addison Howard, and XTX Markets. 2024. Ai mathemati… find
  6. Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR.
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025a. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290.
  9. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025b. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290.
  10. Xuying Li, Zhuo Li, Yuji Kosuga, and Victor Bian. 2025. Optimizing safe and aligned language generation: A multi-objective grpo approach. arXiv preprint arXiv:2503.21819. — [arXiv:2503.21819](https://arxiv.org/abs/2503.21819) find
  11. Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. 2025a. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470. — [arXiv:2505.05470](https://arxiv.org/abs/2505.05470) find
  12. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025b. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  13. Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025a. Deepcoder: A fully open-source 14b coder at o3-mini level. . Notion Blog. find
  14. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025b. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. . Notion Blog. find
  15. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741.
  16. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  17. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  18. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256.
  19. Zhiqing Sun. 2024. Aimo-cmu-math/cmu-math-aimo. find
  20. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — Kimi k1.5: Scaling Reinforcement Learning with LLMs
  21. Qwen Team. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  22. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, and 1 others. 2025. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503.10460. find
  23. Lilian Weng. 2024. Reward hacking in reinforcement learning. lilianweng.github.io. find
  24. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
  25. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-grpo-lead-difficulty-aware-reinforcement/