Elamur Research⌕
☰
home / papers / 2026 / SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

Paper · 2026 · Tianyi Wang, Yixia Li, Long Li et al.

Summary

Reinforcement learning with verifiable rewards is a common way to improve LLM reasoning, but the two main options each have a cost. Token-level PPO depends on a critic and GAE, and these struggle to pass a sparse final reward back through very long Chain-of-Thought outputs. The authors show the critic separates correct from incorrect answers only near the end of a sequence, so the advantage signal fades. GRPO avoids the critic but must sample many responses per prompt, which slows training. The authors argue that GRPO works because it implicitly treats each full response as one action in a contextual bandit. They propose SPPO, which makes this view explicit. A scalar value model, trained with BCE, predicts how likely a prompt is to be solved. The advantage is the binary reward minus this prediction, and it is applied to every token of a single sampled response. On math benchmarks, SPPO beats standard PPO and matches GRPO at a fraction of the compute. This makes RL alignment of reasoning models cheaper.

Key points

  • Evaluated on AIME24/25, AMC23, MATH and Minerva. SPPO reaches GRPO's peak performance with a reported 5.9× training speedup, using one sample per prompt (N = 1).
  • With a Decoupled Critic, a small 1.5B value model guides a 7B policy and reduces memory use by 12.8%.
  • The "tail effect" diagnosis comes from a PPO critic trained with an 8192-token context. Its value estimates track token position rather than the semantic contribution of each step.
  • Additional sparse-reward control experiments cover Hopper, MountainCar, LunarLander, Pendulum and Precision CartPole. In these, SPPO matches or exceeds standard PPO, which fails or becomes unstable on several tasks.
  • The method requires verifiable outcomes, so open-ended generation is left for future work. Code is available at github.com/sustech-nlp/SPPO.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🏢Microsoft Research AsiaBeijing · China73 works · 60,539 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China57 works · 32,843 citations🎓Southern University of Science and TechnologyShenzhen · China19 works · 1,337 citations🎓Shanghai University of Finance and EconomicsShanghai · China13 works · 825 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu and Guanhua Chen. 2026. SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.356

@inproceedings{wang2026sppo,
  title = {{SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks}},
  author = {Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu and Guanhua Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.356},
  url = {https://doi.org/10.18653/v1/2026.acl-long.356}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. Preprint, arXiv:2402.14740. find
  2. Art of Problem Solving. 2025a. Aime problems and solutions. Accessed: 2026-01-04. find
  3. Art of Problem Solving. 2025b. Amc problems and solutions. Accessed: 2026-01-04. find
  4. Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P. Xing,… — [arXiv:2506.14965](https://arxiv.org/abs/2506.14965) find
  5. Tianqing Fang, Zhisong Zhang, Xiaoyang Wang, Rui Wang, Can Qin, Yuxuan Wan, Jun-Yu Ma, Ce Zhang, Jiaqi Chen, Xiyun Li, Hongming Zhang, Haitao Mi, and Dong Yu. 2025. Cognitive kernel-pro: A framework for deep research agents and agent foundation models training. Preprint, arXiv:2508.00414. — [arXiv:2508.00414](https://arxiv.org/abs/2508.00414) find
  6. Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, and Xunliang Cai. 2026a. How to allocate, how to learn? dynamic rollout allocation and advantage modulation for policy optimization. arXiv preprint arXiv:2602.19208. find
  7. Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chang Liu, and Peilin Zhao. 2026b. Proximity-based multi-turn optimization: Practical credit assignment for llm agent training. arXiv preprint arXiv:2602.19225. find
  8. Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. 2025. Segment policy optimization: Effective segment-level credit assignment in rl for large language models. Preprint, arXiv:2505.23564. — [arXiv:2505.23564](https://arxiv.org/abs/2505.23564) find
  9. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, and Dong Yu. 2025. OpenWebVoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computatio…
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.
  11. Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. 2025. Vineppo: Refining credit assignment in rl training of llms. Preprint, arXiv:2410.01679. find
  12. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. Preprint, arXiv:2…
  13. Chen Li, Nazhou Liu, and Kai Yang. 2025a. Adaptive group policy optimization: Towards stable training and token-efficient reasoning. Preprint, arXiv:2503.15952. — [arXiv:2503.15952](https://arxiv.org/abs/2503.15952) find
  14. Long Li, Xuzheng He, Haozhe Wang, Linlin Wang, and Liang He. 2024. How do humans write code? large models do it the same way too. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page 4638–4649. Association for Computational Linguistics. find
  15. Long Li, Zhijian Zhou, Jiaran Hao, Jason Klein Liu, Yanting Miao, Wei Pang, Xiaoyu Tan, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, and Yuan Qi. 2026a. The choice of divergence: A neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward. Preprint, arXiv:2509.07430. — [arXiv:2509.07430](https://arxiv.org/abs/2509.07430) find
  16. Long Li, Zhijian Zhou, Jiangxuan Long, Peiyang Liu, Weidi Xu, Zhe Wang, Shirui Pan, and Chao Qu. 2026b. Sql-astra: Alleviating sparse feedback in agentic sql via column-set matching and trajectory aggregation. Preprint, arXiv:2603.16161. find
  17. Long Li, Zhijian Zhou, Tianyi Wang, Weidi Xu, Zuming Huang, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, and Yuan Qi. 2026c. Dyjr: Preserving diversity in reinforcement learning with verifiable rewards via dynamic jensen-shannon replay. Preprint, arXiv:2603.16157. — [arXiv:2603.16157](https://arxiv.org/abs/2603.16157) find
  18. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. Preprint, arXiv:2503.20783.
  19. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio -b75.notion.site/DeepScaleR-Surpassing-O 1-Preview-w… find
  20. Weijian Ma, Shizhao Sun, Tianyu Yu, Ruiyu Wang, Tat-Seng Chua, and Jiang Bian. 2026. Thinking with blueprints: Assisting vision-language models in spatial reasoning via structured object representation. Preprint, arXiv:2601.01984. — [arXiv:2601.01984](https://arxiv.org/abs/2601.01984) find
  21. Zhiwen Ruan, Yun Chen, Yutao Hou, Peng Li, Yang Liu, and Guanhua Chen. 2025a. Unveiling over-memorization in finetuning llms for reasoning tasks. Preprint, arXiv:2508.04117. — [arXiv:2508.04117](https://arxiv.org/abs/2508.04117) find
  22. Zhiwen Ruan, Yixia Li, He Zhu, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. 2025b. Enhancing large language model reasoning via selective critical token fine-tuning. Preprint, arXiv:2510.10974. — [arXiv:2510.10974](https://arxiv.org/abs/2510.10974) find
  23. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  24. Mukai Li, Qingcheng Zeng, Tianqing Fang, Zhenwen Liang, Linfeng Song, Qi Liu, Haitao Mi, and Dong Yu. 2026d. Verified critical step optimization for llm agents. Preprint, arXiv:2602.03412. find
  25. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  26. Xiaozhe Li, Xinyu Fang, Shengyuan Ding, Linyang Li, Haodong Duan, Qingwen Liu, and Kai Chen. 2025b. Np-engine: Empowering optimization reasoning in large language models with verifiable synthetic np problems. Preprint, arXiv:2510.16476. find
  27. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM.
  28. Zhicong Li, Lingjie Jiang, Yulan Hu, Xingchen Zeng, Yixia Li, Xiangwen Zhang, Guanhua Chen, Zheng Pan, Xin Li, and Yong Liu. 2026e. No more stale feedback: Co-evolving critics for open-world agent learning. Preprint, arXiv:2601.06794.
  29. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press. find
  30. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. Preprint, arXiv:2305.20050.
  31. Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. 2025. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. Preprint, arXiv:2503.22342. find
  32. Chengbing Wang, Yang Zhang, Wenjie Wang, Xiaoyan Zhao, Fuli Feng, Xiangnan He, and Tat-Seng Chua. 2026a. Think-while-generating: On-the-fly reasoning for personalized long-form generation. Preprint, arXiv:2512.06690. — [arXiv:2512.06690](https://arxiv.org/abs/2512.06690) find
  33. Chengbing Wang, Wuqiang Zheng, Yang Zhang, Fengbin Zhu, Junyi Cheng, Yi Xie, Wenjie Wang, and Fuli Feng. 2026b. Perm: Psychology-grounded empathetic reward modeling for large language models. Preprint, arXiv:2601.10532.
  34. Hu Wang, Congbo Ma, Ian Reid, and Mohammad Yaqub. 2025a. Kalman filter enhanced grpo for reinforcement learning-based language model reasoning. Preprint, arXiv:2505.07527. — [arXiv:2505.07527](https://arxiv.org/abs/2505.07527) find
  35. He Zhu, Junyou Su, Peng Lai, Ren Ma, Wenjia Zhang, Linyi Yang, and Guanhua Chen. 2026. Anchored supervised fine-tuning. In The Fourteenth International Conference on Learning Representations. find
  36. Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. 2025b. Aspo: Asymmetric importance sampling policy optimization. Preprint, arXiv:2510.06062. — [arXiv:2510.06062](https://arxiv.org/abs/2510.06062) find
  37. Wenjie Zhu, Yabin Zhang, Xin Jin, Wenjun Zeng, and Lei Zhang. 2025. Ants: Adaptive negative textual space shaping for ood detection via test-time mllm understanding and reasoning. arXiv preprint arXiv:2509.03951. — [arXiv:2509.03951](https://arxiv.org/abs/2509.03951) find
  38. Tianyi Wang, Long Li, Hongcan Guo, Yibiao Chen, Yixia Li, Yong Wang, Yun Chen, and Guanhua Chen. 2026c. Anchored policy optimization: Mitigating exploration collapse via support-constrained rectification. Preprint, arXiv:2602.05717. — [arXiv:2602.05717](https://arxiv.org/abs/2602.05717) find
  39. Cehua Yang, Dongyu Xiao, Junming Lin, Yuyang Song, Hanxu Yan, Shawn Guo, Wei Zhang, Jian Yang, Mingjie Tang, and Bryan Dai. 2025. Agro-sql: Agentic group-relative optimization with high-fidelity data synthesis. Preprint, arXiv:2512.23366. — [arXiv:2512.23366](https://arxiv.org/abs/2512.23366) find
  40. Jian Yang, Wei Zhang, Jiajun Wu, Junhang Cheng, Tuney Zheng, Fanglin Xu, Weicheng Gu, Lin Jing, Yaxin Du, Joseph Li, Yizhi Li, Yan Xing, Chuan Hao, Ran Tao, Ruihao Gong, Aishan Liu, Zhoujun Li, Mingjie Tang, Chenghua Lin, Siheng Chen, Wayne Xin Zhao, Xianglong Liu, Ming Zhou, Bryan Dai, and Weifeng… find
  41. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli… — DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  42. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. 2025. What’s behind ppo’s collapse in long-cot? value optimization holds the secret. Preprint, arXiv:2503.01491. — [arXiv:2503.01491](https://arxiv.org/abs/2503.01491) find
  43. Xuan Zhang, Ruixiao Li, Zhijian Zhou, Long Li, Yulei Qin, Ke Li, Xing Sun, Xiaoyu Tan, Chao Qu, and Yuan Qi. 2026. Count counts: Motivating exploration in LLM reasoning with count-based intrinsic rewards. In The Fourteenth International Conference on Learning Representations. find
  44. Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, Fang Wan, and Furu Wei. 2025. Geometric-mean policy optimization. Preprint, arXiv:2507.20673. — [arXiv:2507.20673](https://arxiv.org/abs/2507.20673) find
  45. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. Group sequence policy optimization. Preprint, arXiv:2507.18071.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-sppo-sequence-level-ppo-long/