Elamur Research⌕
☰
home / papers / 2026 / CRPS: Curriculum Replay via Progressive Suffixes from Successful Trajectories for Long-Horizon LLM Agents

CRPS: Curriculum Replay via Progressive Suffixes from Successful Trajectories for Long-Horizon LLM Agents

Paper · 2026 · Zijing Zhang and Xiajie Yang

Summary

Reinforcement learning for long-horizon LLM agents is hard when the only reward arrives at the end of an episode. Group-normalized optimizers such as GRPO make this worse: if every rollout in a group fails, or every rollout succeeds, the normalized advantages are near zero, so those environment interactions produce almost no gradient. CRPS is a lightweight add-on that addresses this by reusing the rare successes the agent stumbles on. It stores successful trajectories in a buffer, replays a prefix to restore an intermediate state, and lets the current policy finish the remaining suffix of length k. A small k gives an easy near-goal subproblem; a larger k gives a harder one. An online controller tunes k so that group success rates stay in a moderate, learnable range. On ALFWorld and WebShop, with several foundation models, CRPS beats full-episode GRPO and naive experience replay. It also cuts the share of degenerate groups and makes outcomes within groups more varied. The method needs no dense reward shaping and no external task generator.

Key points

  • Core mechanism: replay restarts at index t₀ = T − k of a stored successful trajectory. The suffix length k serves as the single difficulty knob, and an online controller adapts it during training.
  • Diagnostics on WebShop: the authors track the fraction of all-zero groups and the group reward entropy (in bits). Compared with GRPO, CRPS lowers the all-zero fraction quickly and reaches higher entropy earlier, which matters most during cold-start training.
  • Ablations (ALFWorld, Qwen2.5-1.5B):
    • Dynamic k outperforms a fixed suffix length in early learning and keeps entropy higher.
    • Of the replay ratios tested (p_replay ∈ {0.05, 0.2, 0.8}), the default of 0.2 works best.
    • A ratio of 0.05 performs close to plain on-policy training.
    • A ratio of 0.8 speeds up early learning but limits exploration.
  • Compatibility: CRPS is designed for GRPO-style optimizers and also works with RLOO-type objectives. It mixes replayed rollouts with fresh exploration.
  • Limitation: prefix replay assumes roughly deterministic environment transitions. Stochastic dynamics or side effects that cannot be replayed may cause state mismatch.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Zijing Zhang and Xiajie Yang. 2026. CRPS: Curriculum Replay via Progressive Suffixes from Successful Trajectories for Long-Horizon LLM Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.680

@inproceedings{zhang2026crps,
  title = {{CRPS: Curriculum Replay via Progressive Suffixes from Successful Trajectories for Long-Horizon LLM Agents}},
  author = {Zijing Zhang and Xiajie Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.680},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.680}
}

References

41 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mc-Grew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight experience replay. Advances in Neural Information Processing Systems, 30.
  2. Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. — FireAct: Toward Language Agent Fine-tuning
  3. Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. 2021. First return, then explore. In Nature, volume 590, pages 580–586. Nature Publishing Group. find
  4. Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhen Zhang. 2019. Curriculum-guided hindsight experience replay. Advances in Neural Information Processing Systems, 32. find
  5. Erhu Feng, Wenbo Zhou, Zibin Liu, Le Chen, Yunpeng Dong, Cheng Zhang, Yisheng Zhao, Dong Du, Zhichao Hua, Yubin Xia, and 1 others. 2025a. Get experience from practice: Llm agents with record & replay. arXiv preprint arXiv:2505.17716. — [arXiv:2505.17716](https://arxiv.org/abs/2505.17716) find
  6. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025b. Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978.
  7. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856.
  8. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919.
  9. Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. 2018. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933. — [arXiv:1803.00933](https://arxiv.org/abs/1803.00933) find
  10. Jonas Hübotter, Leander Diaz-Bone, Ido Hakimi, Andreas Krause, and Moritz Hardt. 2025. Learning on the job: Test-time curricula for targeted reinforcement learning. arXiv preprint arXiv:2510.04786. — [arXiv:2510.04786](https://arxiv.org/abs/2510.04786) find
  11. Wouter Kool, Herke van Hoof, and Max Welling. 2019. Buy 4 reinforce samples, get a baseline for free! In ICLR 2019 Workshop. find
  12. Yitao Liu, Chenglei Si, Karthik Narasimhan, and Shunyu Yao. 2025. Contextual experience replay for self-improvement of language agents. arXiv preprint arXiv:2506.06698. Accepted to ACL 2025. find
  13. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41–48.
  14. Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. 2025. Arpo: End-to-end policy optimization for gui agents with experience replay. arXiv preprint arXiv:2505.16282. — [arXiv:2505.16282](https://arxiv.org/abs/2505.16282) find
  15. Gaurav Chaudhary and Laxmidhar Behera. 2025. Teach: Temporal variance-driven curriculum for reinforcement learning. arXiv preprint arXiv:2512.22824. — [arXiv:2512.22824](https://arxiv.org/abs/2512.22824) find
  16. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, and 1 others. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533. find
  17. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. find
  18. Fireact: Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. 2020. Curriculum learning for reinforcement learning domains: A framework and survey. In Journal of Machine Learning Research, volume 21, pages 7382– 7431. find
  19. Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. 2018. Self-imitation learning. In International Conference on Machine Learning, pages 3878–3887. PMLR. find
  20. Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pages 2778–2787. PMLR. find
  21. Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. 2020. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664. find
  22. Anirudh Satheesh, Keenan Powell, and Vaneet Aggarwal. 2025a. Distributionally robust self paced curriculum reinforcement learning. arXiv preprint arXiv:2511.05694. — [arXiv:2511.05694](https://arxiv.org/abs/2511.05694) find
  23. Anirudh Satheesh, Keenan Powell, and Hua Wei. 2025b. cmalc-d: Contextual multi-agent llm-guided curriculum learning with diversity-based context blending. arXiv preprint arXiv:2508.20818. Accepted to CIKM 2025. — [arXiv:2508.20818](https://arxiv.org/abs/2508.20818) find
  24. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015. Prioritized experience replay. arXiv preprint arXiv:1511.05952. find
  25. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551.
  26. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  27. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. DeepSeek-Math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  28. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations.
  29. Weiting Tan, Xinghua Qu, Ming Tu, Meng Ge, Andy T Liu, Philipp Koehn, and Lu Lu. 2025. Process-supervised reinforcement learning for interactive multimodal tool-use agents. arXiv preprint arXiv:2509.14480. — [arXiv:2509.14480](https://arxiv.org/abs/2509.14480) find
  30. Georgios Tzannetos, Parameswaran Kamalaruban, and Adish Singla. 2025. Curriculum design for trajectory-constrained agent: Compressing chain-of-thought tokens in llms. Advances in Neural Information Processing Systems, 38. find
  31. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, and 1 others. 2025. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073.
  32. Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, and 1 others. 2025. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning. arXiv preprint arXiv:2505.16421.
  33. Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, and 1 others. 2025. Agentgym: Evaluating and training large language model-based agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Associatio… find
  34. Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, and Huaxiu Yao. 2025. Agent0: Unleashing self-evolving agents from zero data via tool-integrated reasoning. arXiv preprint arXiv:2511.16043. — [arXiv:2511.16043](https://arxiv.org/abs/2511.16043) find
  35. Zidi Xiong, Yuping Lin, Wenya Xie, Pengfei He, Zirui Liu, Jiliang Tang, Himabindu Lakkaraju, and Zhen Xiang. 2025. How memory management impacts llm agents: An empirical study of experience-following behavior. arXiv preprint arXiv:2505.16067.
  36. Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng. 2024. Direct multi-turn preference optimization for language agents. arXiv preprint arXiv:2406.14868. find
  37. Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, and Yuxiao Dong. 2025. Mobilerl: Online agentic reinforcement learning for mobile gui agents. arXiv preprint arXiv:2509.18119. — [arXiv:2509.18119](https://arxiv.org/abs/2509.18119) find
  38. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652.
  39. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
  40. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations.
  41. Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. AgentTuning: Enabling generalized agent abilities for LLMs. arXiv preprint arXiv:2310.12823.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-crps-curriculum-replay-progressive-suffixes/