Elamur Research⌕
☰
home / papers / 2026 / SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility

SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility

Paper · 2026 · Xuyang Zhi, Peilun Zhou, Chengqiang Lu et al.

Summary

LLM post-training increasingly targets open-ended tasks, where rewards come from several criteria scored by an LLM judge. The usual approach merges these criteria with fixed weights. That ignores two facts: different capabilities improve and plateau at different times, and a training example useful for one criterion may help little with another. SPARD replaces fixed weighting with an automatic, self-paced curriculum built on top of GRPO. It has two coupled parts:

  • Progress-Aware Weight Adaptation. The method tracks a noise-filtered measure of progress for each reward dimension, a "Reliable Performance Gain" based on the Lower Confidence Bound idea. It then shifts weight toward dimensions that are still improving steadily, using an exponentiated-gradient update.
  • Reward-Attributed Data Rebalancing. A reward-attribute matrix, derived from score dispersion, links data categories to rewards. Categories that best serve the currently prioritized objectives receive higher importance.

Experiments on Qwen2.5 Instruct models of several sizes show consistent gains over the base models and over a GRPO baseline with averaged rewards. This suggests that progress-aware scheduling makes better use of limited training compute when alignment has many objectives.

Key points

  • The baseline, GRPO_avg, averages the per-dimension rewards. Evaluation covers instruction following (IF), GPQA, LCB (code), creative writing (CW) and multi-turn dialogue (MT).
  • Results for the 14B model:
    • MT: 79.84 (base), 82.25 (GRPO_avg), 84.88 (SPARD).
    • CW: 55.80 (base), 60.02 (GRPO_avg), 60.63 (SPARD).
  • Results for the 32B model:
    • GPQA: 47.47 (base), 48.99 (GRPO_avg), 49.49 (SPARD).
    • MT: 84.68 (base), 85.31 (GRPO_avg), 86.00 (SPARD).
    • On saturated skills such as IF and code, SPARD stays roughly level with GRPO_avg.
  • The learned schedule changes over training:
    • Text-transformation data dominates early.
    • Code data peaks early and then fades.
    • Knowledge QA and creative writing gain weight in later stages.
  • Stated limitations:
    • Strong LLM judges add latency and compute cost.
    • Reward aggregation is still linear.
    • Judge biases may be amplified.
  • Code is available at github.com/USTC-StarTeam/SPARD (USTC and Xiaohongshu, ACL 2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🏢Xiaohongshu (Little Red Book)Shanghai · China11 works · 107 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Xuyang Zhi, Peilun Zhou, Chengqiang Lu, Hang Lv, Yiwei Liang, Rongyang Zhang, Yan Gao, Yiwu, Yao Hu, Hongchao Gu, Defu Lian, Hao Wang and Enhong Chen. 2026. SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2191

@inproceedings{zhi2026spard,
  title = {{SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility}},
  author = {Xuyang Zhi, Peilun Zhou, Chengqiang Lu, Hang Lv, Yiwei Liang, Rongyang Zhang, Yan Gao, Yiwu, Yao Hu, Hongchao Gu, Defu Lian, Hao Wang and Enhong Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2191},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2191}
}

References

48 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \#\# Ethical Considerations Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. 2025c. Rm-r1: Reward modeling as reasoning. Preprint, arXiv:2505.02387.
  2. This work introduces a curriculum-based RL alignment framework that relies on LLM-as-a-judge signals to optimize open-ended generation. A central ethical concern is that judge models may encode subjective preferences, annotation artifacts, and Adithya Bhaskar, Xi Ye, and Danqi Chen. 2025. Language m… find
  3. Nuo Chen, Yufei Gao, Yongnan Jin, Yan Hu, Anningzhe Gao, Lingyong Yan, and Benyou Wang. 2025a. DRBO: Mitigating the bottleneck effect via dynamic reward balancing in multi-reward LLM optimization. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 8817– 8841, Suzhou, Chi… find
  4. Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. 2025b. Self-evolving curriculum for llm reasoning. Preprint, arXiv:2505.14970. find
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  6. Jacob Dineen, Aswin Rrv, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, and Ben Zhou. 2025. Qa-lign: Aligning llms through constitutionally decomposed qa. In Findings of the Association for Computational Linguistics: EMNLP 2025, page 20619–20642. Association… find
  7. Hongchao Gu, Dexun Li, Kuicai Dong, Hao Zhang, Hang Lv, Hao Wang, Defu Lian, Yong Liu, and Enhong Chen. 2025. Rapid: Efficient retrieval-augmented long text generation with writing planning and information discovery. arXiv preprint arXiv:2503.00751. find
  8. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. Preprint, arXiv:2507.17746. find
  9. Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G. Patil, Qi Qi, Shengyu Feng, Julian Katz-Samuels, Richard Yuanzhe Pang, Sujan Gonugondla, Hunter Lang, Yue Yu, and 6 others. 2025. Advancedif: Rub…
  10. Yuqing Huang, Rongyang Zhang, Xuesong He, Xuyang Zhi, Hao Wang, Xin Li, Feiyang Xu, Deguang Liu, Huadong Liang, Yi Li, Jian Cui, Zimu Liu, Shijin Wang, Guoping Hu, Guiquan Liu, Qi Liu, Defu Lian, and Enhong Chen. 2024. Chemeval: A comprehensive multi-level chemical evaluation for large language mode… — [arXiv:2409.13989](https://arxiv.org/abs/2409.13989) find
  11. Yuqing Huang, Rongyang Zhang, Qimeng Wang, Chengqiang Lu, Yan Gao, Yi Wu, Yao Hu, Xuyang Zhi, Guiquan Liu, Xin Li, Hao Wang, and Enhong Chen. 2025a. Selfaug: Mitigating catastrophic forgetting in retrieval-augmented generation via distribution self-alignment. Preprint, arXiv:2509.03934. find
  12. Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, Xijun Gu, Peiyi Tu, Jiaxin Liu, Wenyu Chen, Yuzhuo Fu, Zhiting Fan, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, and 2 others. 2025b. Reinforcement learning with rubric anchors. Prepr… — [arXiv:2508.12790](https://arxiv.org/abs/2508.12790) find
  13. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. Preprint, arXiv:2403.07974.
  14. Mengzhao Jia, Zhihan Zhang, Ignacio Cases, Zheyuan Liu, Meng Jiang, and Peng Qi. 2025a. Autorubric-r1v: Rubric-based generative rewards for faithful multimodal reasoning. Preprint, arXiv:2510.14738. — [arXiv:2510.14738](https://arxiv.org/abs/2510.14738) find
  15. Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. 2025b. Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards. Preprint, arXiv:2506.00103. — [arXiv:2506.00103](https://arxiv.org/abs/2506.00103) find
  16. Zae Myung Kim, Chanwoo Park, Vipul Raheja, Suin Kim, and Dongyeop Kang. 2025. Toward evaluative thinking: Meta policy optimization with evolving reward models. Preprint, arXiv:2504.20157. — [arXiv:2504.20157](https://arxiv.org/abs/2504.20157) find
  17. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, and 4 other… — Tulu 3: Pushing Frontiers in Open Language Model Post-Training
  18. Derek Li, Jiaming Zhou, Leo Maxime Brunswic, Abbas Ghaddar, Qianyi Sun, Liheng Ma, Yu Luo, Dong Li, Mark Coates, Jianye Hao, and Yingxue Zhang. 2025. Omni-thinker: Scaling multi-task rl in llms with hybrid reward and task scheduling. Preprint, arXiv:2507.14783. — [arXiv:2507.14783](https://arxiv.org/abs/2507.14783) find
  19. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024a. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. Preprint, arXiv:2406.11939.
  20. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024b. From live data to high-quality benchmarks: The arena-hard pipeline. LMSYS Blog. find
  21. Sheng Liang, Hang Lv, Zhihao Wen, Yaxiong Wu, Yongyue Zhang, Hao Wang, and Yong Liu. 2025. Adaptive schema-aware event extraction with retrieval-augmented generation. arXiv e-prints, pages arXiv–2505. find
  22. Jianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, and Runzhi Shi. 2025. Rlmr: Reinforcement learning with mixed rewards for creative writing. Preprint, arXiv:2508.18642. — [arXiv:2508.18642](https://arxiv.org/abs/2508.18642) find
  23. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. Preprint, arXiv:2406.04770.
  24. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in llms. Preprint, arXiv:2410.18451.
  25. Tingjia Shen, Hao Wang, Chuan Qin, Ruijun Sun, Yang Song, Defu Lian, Hengshu Zhu, and Enhong Chen. 2025. Prompting is not enough: Exploring knowledge integration and controllable generation on large language models. Big Data Mining and Analytics. find
  26. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2025a. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment. Preprint, arXiv:2510.07743.
  27. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, and 77 others. 2025. Kimi k1.5: Scaling reinforcement l…
  28. Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025b. Inference-time scaling for generalist reward modeling. Preprint, arXiv:2504.02495.
  29. Do June Min, Veronica Perez-Rosas, Kenneth Resnicow, and Rada Mihalcea. 2024. Dynamic reward adjustment in multi-reward reinforcement learning for counselor reflection generation. Preprint, arXiv:2403.13578. find
  30. Samuel J. Paech. 2024. Eq-bench: An emotional intelligence benchmark for large language models. Preprint, arXiv:2312.06281.
  31. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  32. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  33. Sangwon Ryu, Heejin Do, Yunsu Kim, Gary Lee, and Jungseul Ok. 2024. Multi-dimensional optimization for text summarization via reinforcement learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5858–5871, Bangkok, Thailand… find
  34. Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, and 2 others. 2025. D… — DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
  35. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  36. Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. 2025. Checklists are better than reward models for aligning language models. Preprint, arXiv:2507.18624.
  37. Zhenting Wang, Guofeng Cui, Yu-Jhe Li, Kun Wan, and Wentian Zhao. 2025. Dump: Automated distribution-level curriculum learning for rl-based llm post-training. Preprint, arXiv:2504.09710. — [arXiv:2504.09710](https://arxiv.org/abs/2504.09710) find
  38. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. 2025. Lightr1: Curriculum sft, dpo and rl for long cot from scratch and beyond. Preprint, arXiv:2503.10460. find
  39. Wei Wu, Peilun Zhou, Liyi Chen, Qimeng Wang, Chengqiang Lu, Yan Gao, Yi Wu, Yao Hu, and Hui Xiong. 2026. Aligning large language models with searcher preferences. Preprint, arXiv:2603.10473. — [arXiv:2603.10473](https://arxiv.org/abs/2603.10473) find
  40. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  41. Haocheng Yu, Yaxiong Wu, Hao Wang, Wei Guo, Yong Liu, Yawen Li, Yuyang Ye, Junping Du, and Enhong Chen. 2025a. Thought-augmented planning for llm-powered interactive recommender agent. arXiv preprint arXiv:2506.23485. — [arXiv:2506.23485](https://arxiv.org/abs/2506.23485) find
  42. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025b. Dapo: An open-source llm reinforcement learning…
  43. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. Preprint, arXiv:2503.18892.
  44. Rongyang Zhang, Yuqing Huang, Chengqiang Lu, Qimeng Wang, Yan Gao, Yi Wu, Yao Hu, Yin Xu, Wei Wang, Hao Wang, and 1 others. 2025. Rag-igbench: Innovative evaluation for rag-based interleaved generation in open-domain question answering. arXiv e-prints, pages arXiv–2512. find
  45. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. Preprint, arXiv:2405.01470.
  46. Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Hong Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. 2025. Swift:a scalable lightweight infrastructure for fine-tuning. Preprint, arXiv:2408.05517. — [arXiv:2408.05517](https://arxiv.org/abs/2408.05517) find
  47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  48. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhi-2026-spard-self-paced-curriculum-rl/