Elamur Research⌕
☰
home / papers / 2026 / Plug-and-Play Data Module for Code RL: Adaptive Ambiguity Replay

Plug-and-Play Data Module for Code RL: Adaptive Ambiguity Replay

Paper · 2026 · Jianqing Zhang, Wei Xia, Zhezheng Hao et al.

Summary

Reinforcement learning helps LLMs generate and edit code, but good training data is scarce. Experience replay can stretch limited data. However, current LLM replay methods such as RePO and RLEP choose what to replay using static, single-epoch signals like reward or variance, and they can feed the model stale experience. The authors apply dataset cartography to RL training dynamics. They find that "ambiguous" samples, which matter most for generalization, quickly drift into the "easy-to-learn" region and stop being useful. Their remedy is Adaptive Ambiguity Replay (A2R), a plug-in data module that runs separately from the main training loop. A2R does three things:

  • It keeps a FIFO buffer of per-prompt rewards across epochs, along with their rollouts.
  • It estimates each prompt's cartography position from the mean and standard deviation of those rewards, then replays the top-K most ambiguous prompts.
  • It down-weights old rollouts according to how far the current policy has moved from the policy that produced them.

Added to GRPO and DAPO, A2R improves exact match accuracy on real-world code editing, both in learned and unseen domains. The authors conclude that ambiguity tracked across epochs is a useful criterion for choosing what to replay.

Key points

  • Tested on nine general-purpose and code-specialized LLMs (3B–14B), including Qwen2.5-Coder and Qwen3. A2R plugs into GRPO and DAPO without changing their rollouts or advantage computation. Code is published as verl-A2R.
  • On the learned-domain test set, DAPO + A2R improved over DAPO for every model reported:
    • Qwen2.5-Coder-3B: 38.74 → 42.18
    • Qwen2.5-Coder-7B: 41.64 → 43.36
    • Qwen3-4B: 37.99 → 40.31
    • Qwen3-8B: 39.80 → 42.91
  • In the ablations, choosing replay samples by high reward, low reward, or within-epoch variance did worse than plain DAPO. Replaying hard (low-reward) samples hurt the most. The orthogonal-prompt filter and adaptive scaling each added further gains.
  • The default hyperparameters are K = 128 replayed prompts, which balances accuracy against cost, and λ = 5 for importance-scaling strength, which sits at the peak of a unimodal curve.
  • A variant that regenerates rollouts for the selected prompts, instead of reusing cached ones, adds up to +2.12 points, mostly for GRPO. This addresses the method's main limitation, outdated rollouts, at the price of extra compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🏢TencentShenzhen · China163 works · 21,032 citations🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations🎓Hong Kong Polytechnic UniversityHong Kong · China106 works · 5,190 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

Cite

Cite as Jianqing Zhang, Wei Xia, Zhezheng Hao, Hong Wang, Hande Dong, Qiang Lin, Yang Liu, Jian Cao and Qiang Yang. 2026. Plug-and-Play Data Module for Code RL: Adaptive Ambiguity Replay. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.886

@inproceedings{zhang2026plug,
  title = {{Plug-and-Play Data Module for Code RL: Adaptive Ambiguity Replay}},
  author = {Jianqing Zhang, Wei Xia, Zhezheng Hao, Hong Wang, Hande Dong, Qiang Lin, Yang Liu, Jian Cao and Qiang Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.886},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.886}
}

References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Natasha Butt, Blazej Manczak, Auke Wiggers, Corrado Rainone, David W Zhang, Michaël Defferrard, and Taco Cohen. 2024. Codeit: Self-improving language models with prioritized hindsight replay. In International Conference on Machine Learning. find
  2. Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, and 1 others. 2024. Can it edit? evaluating the ability of large language models to follow code editing instructions. In First Conference… find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. 2023. Aligning offline metrics and human judgments of value for code generation models. In Findings of… find
  4. Shihan Dou, Muling Wu, Jingwen Xu, Rui Zheng, Tao Gui, and Qi Zhang. 2025. Improving rl exploration for llm reasoning through retrospective replay. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 594–606. Springer. find
  5. William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. 2020. Revisiting fundamentals of experience replay. In International conference on machine learning. find
  6. Cuiyun Gao, Xing Hu, Shan Gao, Xin Xia, and Zhi Jin. 2025. The current challenges of software engineering in the era of large language models. ACM Transactions on Software Engineering and Methodology, 34(5):1–30. find
  7. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024a. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. find
  8. Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, Yue Wang, and 1 others. 2024b. Codeeditorbench: Evaluating code editing capability of large language models. arXiv preprint arXiv:2404.03543. — [arXiv:2404.03543](https://arxiv.org/abs/2404.03543) find
  9. Hugging Face. 2025. Hugging face — the ai community building the future. https://huggingface.co/. Accessed: 2025-10-08. ↗
  10. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — Qwen2.5-Coder Technical Report
  11. David Isele and Akansel Cosgun. 2018. Selective experience replay for lifelong learning. In Proceedings of the AAAI conference on artificial intelligence. find
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  13. VI Lcvenshtcin. 1966. Binary coors capable or ‘correcting deletions, insertions, and reversals. In Soviet physics-doklady. find
  14. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems.
  15. Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. 2025. Repo: Replay-enhanced policy optimization. arXiv preprint arXiv:2506.09340. — [arXiv:2506.09340](https://arxiv.org/abs/2506.09340) find
  16. Zichuan Liu, Jinyu Wang, Lei Song, and Jiang Bian. 2025. Sample-efficient llm optimization with reset replay. arXiv preprint arXiv:2508.06412. — [arXiv:2508.06412](https://arxiv.org/abs/2508.06412) find
  17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems.
  18. Leonard S Pleiss, Tobias Sutter, and Maximilian Schiffer. 2025. Reliability-adjusted prioritized experience replay. arXiv preprint arXiv:2506.18482. — [arXiv:2506.18482](https://arxiv.org/abs/2506.18482) find
  19. Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  20. Santhosh Kumar Ravindran. 2025. Cosmocore affective dream-replay reinforcement learning for code generation. arXiv preprint arXiv:2510.18895. — [arXiv:2510.18895](https://arxiv.org/abs/2510.18895) find
  21. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  23. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems.
  24. Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. 2025a. Improving data efficiency for llm reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay. arXiv preprint arXiv:2506.05316. — [arXiv:2506.05316](https://arxiv.org/abs/2506.05316) find
  25. Yiyou Sun, Yuhan Cao, Pohao Huang, Haoyue Bai, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. 2025b. Delta: How does rl unlock and transfer new algorithms in llms? In The 5th Workshop on Mathematical Reasoning and AI at NeurIPS 2025. find
  26. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  27. Harm Vanseijen and Rich Sutton. 2015. A deeper look at planning as learning from replay. In International conference on machine learning. find
  28. Chao Wang, Tao Yang, Hongtao Tian, Yunsheng Shi, Qiyao Ma, Xiaotao Liu, Ting Yao, and Wenbo Ding. 2025a. Learning more with less: A dynamic dual-level down-sampling framework for efficient policy optimization. arXiv preprint arXiv:2509.22115. — [arXiv:2509.22115](https://arxiv.org/abs/2509.22115) find
  29. Chen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao, Zedong Dan, Weiran Huang, Yue Wang, and Yuzhi Zhang. 2025b. Eframe: Deeper reasoning via exploration-filtering-replay reinforcement learning framework. arXiv preprint arXiv:2506.22200. — [arXiv:2506.22200](https://arxiv.org/abs/2506.22200) find
  30. Hong Wang, Zhezheng Hao, Jian Luo, Chenxing Wei, Yao Shu, Lei Liu, Qiang Lin, Hande Dong, and Jiawei Chen. 2025c. Scheduling your llm reinforcement learning with reasoning trees. arXiv preprint arXiv:2510.24832. — [arXiv:2510.24832](https://arxiv.org/abs/2510.24832) find
  31. Junqiao Wang, Zeng Zhang, Yangfan He, Zihao Zhang, Xinyuan Song, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, and 1 others. 2024. Enhancing code llms with reinforcement learning in code generation: A survey. arXiv preprint arXiv:2412.20367. — [arXiv:2412.20367](https://arxiv.org/abs/2412.20367) find
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv:2505.09388. arXiv preprint John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir P…
  33. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
  34. Zed Industries. 2025. Zeta: Repository-level code edit prediction dataset. https://huggingface.co/ datasets/zed-industries/zeta. Dataset, Hugging Face; Apache-2.0 license. ↗
  35. Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, and Guorui Zhou. 2025a. Rlep: Reinforcement learning with experience replay for llm reasoning. arXiv preprint arXiv:2507.07451. — [arXiv:2507.07451](https://arxiv.org/abs/2507.07451) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-plug-play-data-module-code/