Elamur Research⌕
☰
home / papers / 2026 / MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning

MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning

Paper · 2026 · Xiaoliang Fu, Jiaye Lin, Yangyi Fang et al.

Summary

RLVR methods like GRPO inherit PPO-style hard clipping, which the authors argue is poorly suited to how LLMs learn reasoning. They identify three problems. First, the binary clip boundary throws away useful gradients from tokens just outside the trust region, yet still lets harmful updates through inside it. Second, a single ratio range treats frequent and rare tokens alike, which is too permissive for head tokens and too strict for long-tail ones. Third, positive and negative rewards are weighted equally, even though failed responses give much noisier credit assignment.

MASPO (Mass-Adaptive Soft Policy Optimization) handles all three in one objective. A differentiable soft Gaussian gate replaces the box constraint. A mass-adaptive limiter widens the trust region as token probability decreases. An asymmetric risk controller scales updates according to how reliable the advantage signal is.

The authors trained DeepSeek-R1-Distill-Qwen models on math data and evaluated on AIME and MATH500. MASPO beat GRPO at every model size and kept policy entropy higher during training. This suggests probability-aware, risk-sensitive trust regions are a practical step toward more sample-efficient reasoning training.

Key points

  • Setup: Experiments use DeepSeek-R1-Distill-Qwen at 1.5B, 7B and 14B, with DAPO-Math-17k for training. Benchmarks include AIME24, AIME25 and MATH500, reported as Avg@32 and Pass@32.
  • Gains over GRPO (average Avg@32 / Pass@32):
    • 1.5B: 48.4/70.1 → 51.4/72.9
    • 7B: 58.9/77.2 → 61.8/79.8
    • 14B: 53.6/67.4 → 56.4/71.1, a gain of +2.8 Avg@32 and +3.7 Pass@32
  • Robustness: MASPO stays ahead of GRPO across a range of α and β settings. The 14B run used α = 0.5 and β = 0.03.
  • Limitations:
    • Evaluation covers only math tasks with deterministic answers.
    • Models beyond 14B and MoE architectures were not tested.
    • The asymmetric controller assumes positive rewards are trustworthy, so domains with subjective rewards would need adaptation.
  • Code: Available at github.com/FlyTune/MASPO-RL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Binbin Zheng, Chaowen Hu, Zekai Shao, Cong Qin, Lu Pan, Ke Zeng and Xunliang Cai. 2026. MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1918

@inproceedings{fu2026maspo,
  title = {{MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning}},
  author = {Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Binbin Zheng, Chaowen Hu, Zekai Shao, Cong Qin, Lu Pan, Ke Zeng and Xunliang Cai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1918},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1918}
}

References

48 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sikai Bai, Haoxi Li, Jie Zhang, Yongjiang Liu, and Song Guo. 2026. Ttvs: Boosting self-exploring reinforcement learning via test-time variational synthesis. arXiv preprint arXiv:2604.08468. find
  2. Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, and 1 others. 2025a. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585.
  3. Zhiwei Chen, Yupeng Hu, Zhiheng Fu, Zixu Li, Jiale Huang, Qinlei Huang, and Yinwei Wei. 2026. Intent: Invariance and discrimination-aware noise mitigation for robust composed image retrieval. In AAAI Conference on Artificial Intelligence (AAAI), pages 20463–20471. find
  4. Zhiwei Chen, Yupeng Hu, Zixu Li, Zhiheng Fu, Xuemeng Song, and Liqiang Nie. 2025b. Offset: Segmentation-based focus shift revision for composed image retrieval. In ACM International Conference on Multimedia (ACM MM), pages 6113–6122. find
  5. Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. 2025. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758. find
  6. Haonan Dong, Kehan Jiang, Haoran Ye, Wenhao Zhu, Zhaolu Kang, and Guojie Song. 2026. Neureasoner: Towards explainable, controllable, and unified reasoning via mixture-of-neurons. Preprint, arXiv:2604.02972.
  7. Haonan Dong, Wenhao Zhu, Guojie Song, and Liang Wang. 2025. AuroRA: Breaking low-rank bottleneck of loRA with nonlinear mapping. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  8. Benjamin Eysenbach and Sergey Levine. 2021. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257. — [arXiv:2103.06257](https://arxiv.org/abs/2103.06257) find
  9. Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. 2025. Soft adaptive policy optimization. arXiv preprint arXiv:2511.20347. — [arXiv:2511.20347](https://arxiv.org/abs/2511.20347) find
  10. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
  11. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, and 1 others. 2025. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312. find
  12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  13. Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, Xue Gong, Siheng Li, Ruibin Xiong, Kejiao Li, Yuhao Jiang, and Bo Zhou. 2025. Low-probability tokens sustain exploration in reinforcement learning with verifiable reward. arXiv preprint arXiv:2510.03222. find
  14. Edwin T Jaynes. 1957a. Information theory and statistical mechanics. Physical review, 106(4):620. find
  15. Edwin T Jaynes. 1957b. Information theory and statistical mechanics. ii. Physical review, 108(2):171. find
  16. Kehan Jiang, Haonan Dong, Zhaolu Kang, Zhengzhou Zhu, and Guojie Song. 2026. Foe: Forest of errors makes the first solution the best in large reasoning models. Preprint, arXiv:2604.02967.
  17. Hynek Kydlíček. 2024. Math-Verify: Math Verification Library. find
  18. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857…
  19. Zixu Li, Yupeng Hu, Zhiwei Chen, Shiqi Zhang, Qinlei Huang, Zhiheng Fu, and Yinwei Wei. 2026. Habit: Chrono-synergia robust progressive learning framework for composed image retrieval. In AAAI Conference on Artificial Intelligence (AAAI), pages 6762– 6770. find
  20. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023a. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  21. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023b. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  22. Peiyang Liu. 2024. Unsupervised corrupt data detection for text training. Expert Systems with Applications, 248:123335. find
  23. Peiyang Liu, Ziqiang Cui, Di Liang, and Wei Ye. 2025. Who stole your data? a method for detecting unauthorized rag theft. arXiv preprint arXiv:2510.07728. — [arXiv:2510.07728](https://arxiv.org/abs/2510.07728) find
  24. Peiyang Liu, Jinyu Yang, Lin Wang, Sen Wang, Yunlai Hao, and Huihui Bai. 2023. Retrieval-based unsupervised noisy label detection on text data. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4099–4104. find
  25. MAA. 2023. American mathematics competitions -amc. Accessed: 2023. find
  26. MAA. 2024. American invitational mathematics examination - aime. Accessed: 2024. find
  27. MAA. 2025. American invitational mathematics examination - aime. Accessed: 2025. find
  28. Brendan O’Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. 2016. Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626. find
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  30. Doina Precup, Richard S Sutton, and Satinder P Singh. 2000. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759–766. find
  31. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  32. Zekai Shao, Yi Shan, Yixuan He, Yuxuan Yao, Junhong Wang, Xiaolong Zhang, Yu Zhang, and Siming Chen. 2025a. Do language model agents align with humans in rating visualizations? an empirical study. IEEE Computer Graphics and Applications. find
  33. Zekai Shao, Leixian Shen, Haotian Li, Yi Shan, Huamin Qu, Yun Wang, and Siming Chen. 2025b. Narrative player: Reviving data narratives with visuals. IEEE Transactions on Visualization and Computer Graphics, 31(10):6781–6795. find
  34. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  35. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297.
  36. Zhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Wenping Hu, Fuzheng Zhang, Kun Gai, and Guorui Zhou. 2025a. Klear-reasoner: Advancing reasoning capability via gradient-preserving clipping policy optimization. arXiv preprint arXiv:2508.07629. — [arXiv:2508.07629](https://arxiv.org/abs/2508.07629) find
  37. Zhenpeng Su, Leiyu Pan, Minxuan Lv, Yuntao Li, Wenping Hu, Fuzheng Zhang, Kun Gai, and Guorui Zhou. 2025b. Ce-gppo: Coordinating entropy via gradient-preserving clipping policy optimization in reinforcement learning. arXiv preprint arXiv:2509.20712.
  38. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275.
  39. Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. 2025. Aspo: Asymmetric importance sampling policy optimization. arXiv preprint arXiv:2510.06062. — [arXiv:2510.06062](https://arxiv.org/abs/2510.06062) find
  40. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256. find
  41. Ronald J Williams and Jing Peng. 1991. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268. find
  42. Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, and 1 others. 2025. Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping. arXiv preprint arXiv:2510.18927. — [arXiv:2510.18927](https://arxiv.org/abs/2510.18927) find
  43. Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and 1 others. 2025. A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343. — [arXiv:2504.11343](https://arxiv.org/abs/2504.11343) find
  44. Shihui Yang, Chengfeng Dou, Peidong Guo, Kai Lu, Qiang Ju, Fei Deng, and Rihui Xin. 2025a. Dcpo: Dynamic clipping policy optimization. arXiv preprint arXiv:2509.02333. — [arXiv:2509.02333](https://arxiv.org/abs/2509.02333) find
  45. Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. 2025b. Do not let low-probability tokens over-dominate in rl for llms. arXiv preprint arXiv:2505.12929. — [arXiv:2505.12929](https://arxiv.org/abs/2505.12929) find
  46. Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, and 1 others. 2020. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, pages 6672–6679. find
  47. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
  48. Naixin Zhai, Pengyang Shao, Binbin Zheng, Yonghui Yang, Fei Shen, Long Bai, and Xun Yang. 2026. Maximizing local entropy where it matters: Prefix-aware localized llm unlearning. arXiv preprint arXiv:2601.03190.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fu-2026-maspo-unifying-gradient-utilization-probability/