Elamur Research⌕
☰
home / papers / 2026 / Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration

Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration

Paper · 2026 · Yang Zhao, Yangou Ouyang, Xiao Ding et al.

Summary

The usual recipe for training LLM agents runs a supervised fine-tuning (SFT) stage first and a reinforcement learning (RL) stage after it. It is still unclear which training examples should go to which stage. Existing approaches send the whole corpus through both stages, send everything to RL, or filter by outcome signals such as accuracy, and none of these reflect what the model itself finds hard. Drawing on Schema Theory, the authors propose PRISM. It measures how strongly a sample conflicts with the model's current knowledge by looking at how gradient energy is distributed across parameter groups.

A single backward pass on the reference trajectory, with no weight update, produces per-matrix gradient norms. Concentration statistics (Gini, kurtosis, coefficient of variation) then score each sample. Samples above the median, where updates are concentrated, go to RL. The rest, where updates are spread out, go to SFT.

On WebShop and ALFWorld, PRISM beats strong hybrid baselines and sets a new best ALFWorld result. It also cuts training time substantially. This suggests that routing data by internal optimization signals is a cheaper and more reliable way to post-train agents.

Key points

  • Probing: Gradients are probed over 7L parameter groups: the attention and FFN projection matrices in each of L layers. Frobenius norms are computed on the fly, so the memory cost is close to that of a forward pass. Probing took about 2 minutes on 8× NVIDIA A100 (80GB).
  • Results and speed:
    • PRISM reaches 95.31 success rate on ALFWorld.
    • Compared with full-RL GRPO, training is up to 3.22× faster on ALFWorld (Gini variant) and up to 1.95× faster on WebShop (Kurtosis variant).
  • Concentration vs. magnitude: Routing by spatial concentration (Gini) outperforms routing by plain L2 gradient magnitude:
    • WebShop score: 85.15 vs 79.75.
    • ALFWorld success rate: 95.31 vs 90.63.
  • SFT/RL split: Varying the share of data sent to RL gives an inverted-U curve, with the best result at a 50/50 split (Qwen3-8B on WebShop, CV metric). The gains hold across Qwen and Llama backbones.
  • Limitations:
    • Only 7B–8B models were tested.
    • Routing is static, computed once at the start of training.
    • Evaluation covers only agent benchmarks.
    • Code is available at github.com/zy125413/PRISM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China185 works · 29,140 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yang Zhao, Yangou Ouyang, Xiao Ding, Hepeng Wang, Bibo Cai, Kai Xiong, Jinglong Gao, Zhouhao Sun, Li Du, Bing Qin and Ting Liu. 2026. Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1097

@inproceedings{zhao2026consolidation,
  title = {{Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration}},
  author = {Yang Zhao, Yangou Ouyang, Xiao Ding, Hepeng Wang, Bibo Cai, Kai Xiong, Jinglong Gao, Zhouhao Sun, Li Du, Bing Qin and Ting Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1097},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1097}
}

References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chirag Agarwal, Daniel D’souza, and Sara Hooker. 2022. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10368–10378. find
  2. Lenaic Chizat, Edouard Oyallon, and Francis Bach. 2019. On lazy training in differentiable programming. Advances in neural information processing systems, 32. find
  3. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. 2025. Sft memorizes, rl generalizes: A comparative study of Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories… find
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  5. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  6. Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, Ning Ding, and 1 others. 2025. Towards a unified view of large language model post-training. arXiv preprint arXiv:2509.04419. — [arXiv:2509.04419](https://arxiv.org/abs/2509.04419) find
  7. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. find
  8. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  9. Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. 2021. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596– 20607. 23937 9 find
  10. Jean Piaget. 1952. The origins of intelligence in children, volume 8. International Universities Press New York. find
  11. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. 2025. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958.
  12. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, and 1 others. 2024. Toolllm: Facilitating large language models to master 16000+ real-world apis. In ICLR.
  13. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  14. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  15. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In ICLR.
  16. Umut Simsekli, Levent Sagun, and Mert Gurbuzbala-ban. 2019. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827–5837. PMLR. find
  17. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  18. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
  19. Minxing Zhang, Yi Yang, Roy Xie, Bhuwan Dhingra, Shuyan Zhou, and Jian Pei. 2025. Generalizability of large language model-based agents: A comprehensive survey. arXiv preprint arXiv:2509.16330. — [arXiv:2509.16330](https://arxiv.org/abs/2509.16330) find
  20. Yang Zhao, Li Du, Xiao Ding, Yangou Ouyang, Hepeng Wang, Kai Xiong, Jinglong Gao, Zhouhao Sun, Dongliang Xu, Qing Yang, and 1 others. 2025. Beyond similarity: A gradient-based graph method for instruction tuning data selection. In Proceedings of the 63rd Annual Meeting of the Association for Computa… find
  21. Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Shi Jun, Ting Liu, and Bing Qin. 2024. Deciphering the impact of pretraining data on large language models through machine unlearning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9386–9406. find
  22. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations), pages 400–410.
  23. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, and 1 others. 2023. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-consolidation-or-adaptation-prism-disentangling/