Elamur Research⌕
☰
home / papers / 2026 / d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models

d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models

Paper · 2026 · Leyi Pan, Shuchang Tao, Yunpeng Zhai et al.

Summary

Reinforcement learning for diffusion large language models (dLLMs) has two weak spots. Rewards are usually sparse outcome scores or unverified process signals, and per-token log-probabilities are approximated without accounting for the expectation over all possible decoding orders. d-TreeRPO addresses both. It organizes rollouts as a tree: denoising steps are grouped into tree steps, and each internal node branches into several samples. Verifiable rewards at the leaves are propagated upward, so each step's advantage is the value of the child minus the value of its parent. This yields step-level credit that remains checkable. The authors also prove that the error of a single-forward-pass probability estimate shrinks as the policy grows more confident. However, high confidence early in training hurts exploration. To balance this, they add a self-distillation loss whose weight rises over training. The method was tested on LLaDA-8B-Instruct and LLaDA-MoE-7BA1B-Instruct across Sudoku, Countdown, GSM8K and Math500. It beats Diffu-GRPO, wd1 and LLaDA-1.5, making dLLM RL more dependable.

Key points

  • On LLaDA-8B-Instruct, the gains over the base model are +86.2% on Sudoku (reaching 92.9%), +51.6% on Countdown, +4.5% on GSM8K and +5.3% on Math500.
  • The self-distillation schedule raises λ(t) and lowers τ(t) during training. A reversed schedule improves faster at first but degrades once reward reaches about 0.75.
  • Estimation error was checked by Monte Carlo over 32 random decoding orders, with 100 samples per task. The variant with self-distillation had the smallest error, while a diversity-promoting loss performed worst.
  • Cost was measured on 8 H20 GPUs for Sudoku. d-TreeRPO takes about 598 s per batch, 9.96 s per update and roughly 48 h to converge. It is slower than Diffu-GRPO and wd1, comparable to GDPO and TraceRL, and faster than SAPO.
  • Code is available at github.com/THU-BPM/d-TreeRPO, and checkpoints are published on Hugging Face (Leyiii/dtreerpo-checkpoints).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓University of Illinois ChicagoChicago · United States61 works · 20,107 citations🏢Alibaba Group (Cayman Islands)George Town · Cayman Islands32 works · 3,389 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Leyi Pan, Shuchang Tao, Yunpeng Zhai, Zheyu Fu, Liancheng Fang, Minghua He, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu and Lijie Wen. 2026. d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.994

@inproceedings{pan2026d,
  title = {{d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models}},
  author = {Leyi Pan, Shuchang Tao, Yunpeng Zhai, Zheyu Fu, Liancheng Fang, Minghua He, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu and Lijie Wen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.994},
  url = {https://doi.org/10.18653/v1/2026.acl-long.994}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. 2025. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573. — [arXiv:2503.09573](https://arxiv.org/abs/2503.09573) find
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  3. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, and 1 others. 2025. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. find
  4. Liancheng Fang, Aiwei Liu, Henry Peng Zou, Yankai Chen, Enze Ma, Leyi Pan, Chunyu Miao, Wei-Chieh Huang, Xue Liu, and Philip S Yu. 2026. Locally confident, globally stuck: The quality-exploration dilemma in diffusion language models. arXiv preprint arXiv:2604.00375. — [arXiv:2604.00375](https://arxiv.org/abs/2604.00375) find
  5. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  6. Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. 2025. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639. — [arXiv:2506.20639](https://arxiv.org/abs/2506.20639) find
  7. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  8. Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, and Guo-Jun Qi. 2025. Reinforcing the diffusion chain of lateral thought with diffusion language models. arXiv preprint arXiv:2505.10446. find
  9. Xianzhi Li, Ethan Callanan, Xiaodan Zhu, Mathieu Sibue, Antony Papadimitriou, Mahmoud Mahfouz, Zhiqiang Ma, and Xiaomo Liu. 2025. Entropy-aware branching for improved mathematical reasoning. arXiv preprint arXiv:2503.21961. — [arXiv:2503.21961](https://arxiv.org/abs/2503.21961) find
  10. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  11. Nianyi Lin, Jiajie Zhang, Lei Hou, and Juanzi Li. 2025. Boundary-guided policy optimization for memory-efficient rl of diffusion large language models. arXiv preprint arXiv:2510.11683. find
  12. Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, and 1 others. 2025. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193.
  13. Hongze Tan, Jianfei Pan, Jinghao Lin, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang. 2025. Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508.04349. — [arXiv:2508.04349](https://arxiv.org/abs/2508.04349) find
  14. Xiaohang Tang, Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. 2025. wd1: Weighted policy optimization for reasoning in diffusion language models. arXiv preprint arXiv:2507.08838. — [arXiv:2507.08838](https://arxiv.org/abs/2507.08838) find
  15. Guanghan Wang, Yair Schiff, Gilad Turok, and Volodymyr Kuleshov. 2025a. d2: Improved techniques for training reasoning diffusion language models. arXiv preprint arXiv:2509.21474. find
  16. Aiwei Liu, Minghua He, Shaoxun Zeng, Sijun Zhang, Linhao Zhang, Chuhan Wu, Wei Jia, Yuan Liu, Xiao Zhou, and Jie Zhou. 2025. Wedlm: Reconciling diffusion language models with standard causal attention for fast inference. arXiv preprint arXiv:2512.22737. — [arXiv:2512.22737](https://arxiv.org/abs/2512.22737) find
  17. Yinjie Wang, Ling Yang, Bowen Li, Ye Tian, Ke Shen, and Mengdi Wang. 2025b. Revolutionizing reinforcement learning framework for diffusion large language models. arXiv preprint arXiv:2509.06949. — [arXiv:2509.06949](https://arxiv.org/abs/2509.06949) find
  18. Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. 2024. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514. — [arXiv:2410.18514](https://arxiv.org/abs/2410.18514) find
  19. Shaoan Xie, Lingjing Kong, Xiangchen Song, Xinshuai Dong, Guangyi Chen, Eric P Xing, and Kun Zhang. 2025a. Step-aware policy optimization for reasoning in diffusion large language models. arXiv preprint arXiv:2510.01544. — [arXiv:2510.01544](https://arxiv.org/abs/2510.01544) find
  20. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. arXiv preprint arXiv:2502.09992.
  21. Jingyang Ou, Jiaqi Han, Minkai Xu, Shaoxuan Xu, Jianwen Xie, Stefano Ermon, Yi Wu, and Chongxuan Li. 2025. Principled rl for diffusion llms emerges from a sequence-level perspective. arXiv preprint arXiv:2512.03759. find
  22. Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. 2024. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736. — [arXiv:2406.03736](https://arxiv.org/abs/2406.03736) find
  23. Kevin Rojas, Jiahe Lin, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, Molei Tao, and Wei Deng. 2025. Improving reasoning for diffusion language models via group diffusion policy optimization. arXiv preprint arXiv:2510.08554. — [arXiv:2510.08554](https://arxiv.org/abs/2510.08554) find
  24. Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, and 1 others. 2025b. Dreamcoder 7b: An open diffusion language model for code. arXiv preprint arXiv:2509.01142. — [arXiv:2509.01142](https://arxiv.org/abs/2509.01142) find
  25. Jingyi Yang, Guanxu Chen, Xuhao Hu, and Jing Shao. 2025a. Taming masked diffusion language models via consistency trajectory reinforcement learning with fewer decoding step. arXiv preprint arXiv:2509.23924. — [arXiv:2509.23924](https://arxiv.org/abs/2509.23924) find
  26. Kai Yang, Xin Xu, Yangkun Chen, Weijie Liu, Jiafei Lyu, Zichuan Lin, Deheng Ye, and Saiyong Yang. 2025b. Entropic: Towards stable long-term training of llms via entropy stabilization with proportional-integral control. arXiv preprint arXiv:2511.15248. — [arXiv:2511.15248](https://arxiv.org/abs/2511.15248) find
  27. Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. 2025c. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505.15809. — [arXiv:2505.15809](https://arxiv.org/abs/2505.15809) find
  28. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487. find
  29. Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Quanquan Gu. 2023. Diffusion language models can perform many tasks with scaling and instruction-finetuning. arXiv preprint arXiv:2308.12219. — [arXiv:2308.12219](https://arxiv.org/abs/2308.12219) find
  30. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
  31. Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. 2024. Entropy-regularized process reward model. arXiv preprint arXiv:2412.11006. find
  32. Lingzhe Zhang, Liancheng Fang, Chiming Duan, Minghua He, Leyi Pan, Pei Xiao, Shiyu Huang, Yunpeng Zhai, Xuming Hu, Philip S Yu, and 1 others. 2025. A survey on parallel text generation: From parallel decoding to diffusion language models. arXiv preprint arXiv:2508.08712. — [arXiv:2508.08712](https://arxiv.org/abs/2508.08712) find
  33. Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. 2025a. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216. — [arXiv:2504.12216](https://arxiv.org/abs/2504.12216) find
  34. Siyan Zhao, Mengchen Liu, Jing Huang, Miao Liu, Chenyu Wang, Bo Liu, Yuandong Tian, Guan Pang, Sean Bell, Aditya Grover, and 1 others. 2025b. Inpainting-guided policy optimization for diffusion large language models. arXiv preprint arXiv:2509.10396. — [arXiv:2509.10396](https://arxiv.org/abs/2509.10396) find
  35. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and 1 others. 2025. Group sequence policy optimization. arXiv preprint arXiv:2507.18071.
  36. Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and 1 others. 2025a. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505.19223.
  37. Fengqi Zhu, Zebin You, Yipeng Xing, Zenan Huang, Lin Liu, Yihong Zhuang, Guoshan Lu, Kangyu Wang, Xudong Wang, Lanning Wei, and 1 others. 2025b. Llada-moe: A sparse moe diffusion language model. arXiv preprint arXiv:2509.24389. — [arXiv:2509.24389](https://arxiv.org/abs/2509.24389) find
  38. Fengqi Zhu, Zebin You, Yipeng Xing, Zenan Huang, Lin Liu, Yihong Zhuang, Guoshan Lu, Kangyu Wang, Xudong Wang, Lanning Wei, and 1 others. 2025c. Llada-moe: A sparse moe diffusion language model. arXiv preprint arXiv:2509.24389. — [arXiv:2509.24389](https://arxiv.org/abs/2509.24389) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2026-d-treerpo-more-reliable-policy/