Elamur Research⌕
☰
home / papers / 2026 / Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards

Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards

Paper · 2026 · Shaoan Xie, Lingjing Kong, Xiangchen Song et al.

Summary

Masked diffusion language models (MdLLMs) such as LLaDA generate text by gradually unmasking a sequence. Reinforcement learning methods such as diffu-GRPO reward only the correctness of the final answer, so they do not supervise the denoising steps themselves. As a result, models often waste intermediate steps on repeated or empty tokens and settle the answer only at the end, which makes their reasoning hard to interpret and unstable.

Process Reward Models built for autoregressive LLMs are a poor fit here. They struggle with partially masked intermediate states, they are open to reward hacking, and they are costly to run during training. The authors therefore propose the Denoising Process Reward (DPR). DPR estimates how much a denoising interval contributes to the outcome by comparing the correctness of rollouts started from different timesteps. It needs no human annotation and no external reward model. An efficient stochastic estimator reuses ordinary training rollouts, so DPR fits into GRPO-style pipelines at little extra cost.

On reasoning benchmarks, DPR improves stability, interpretability and accuracy over LLaDA and diffu-GRPO, and code and checkpoints are released.

Key points

  • DPR is a process-level signal computed from the model's own denoising trajectory, built on top of GRPO and diffu-GRPO.
  • On COUNTDOWN, the full method reaches 51.6, 52.0 and 56.3 accuracy at sequence lengths 128, 256 and 512. A variant without upweighting (Ours-NoUpweight) scores 41.0, 41.4 and 50.4.
  • The efficient estimator runs at 3.42 sec/it while performing comparably to Ours-Cover (6.23 sec/it) and Ours-Random (4.76 sec/it). The authors describe its training speed as close to diffu-GRPO.
  • In an ablation over the number of samples N ∈ {1, 3, 6, 9}, N = 1 gives noisy reward estimates, while N ≥ 3 is stable. The main experiments use N = 3.
  • The method has two acknowledged limitations. It relies on diffu-GRPO's mean-field likelihood approximation. It also uses a simplified estimator (t₂ = T), which may introduce bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates138 works · 6,590 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Shaoan Xie, Lingjing Kong, Xiangchen Song, Xinshuai Dong, Guangyi Chen, Eric P. Xing and Kun Zhang. 2026. Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1978

@inproceedings{xie2026advancing,
  title = {{Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards}},
  author = {Shaoan Xie, Lingjing Kong, Xiangchen Song, Xinshuai Dong, Guangyi Chen, Eric P. Xing and Kun Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1978},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1978}
}

References

37 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Black-Phoenix. 2024. 4x4-sudoku-dataset: 1 million 4×4 sudoku boards. https://github.com/ Black-Phoenix/4x4-Sudoku-Dataset. ↗
  2. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  4. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, and 1 others. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456.
  5. Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, and 1 others. 2024. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891. — [arXiv:2410.17891](https://arxiv.org/abs/2410.17891) find
  6. Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. 2025. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639. — [arXiv:2506.20639](https://arxiv.org/abs/2506.20639) find
  7. Haoyu He, Katrin Renz, Yong Cao, and Andreas Geiger. 2025. Mdpo: Overcoming the training-inference divide of masked diffusion language models. arXiv preprint arXiv:2508.13148. — [arXiv:2508.13148](https://arxiv.org/abs/2508.13148) find
  8. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  9. Feng Hong, Geng Yu, Yushi Ye, Haicheng Huang, Huangjie Zheng, Ya Zhang, Yanfeng Wang, and Jiangchao Yao. 2025. Wide-in, narrow-out: Revok-able decoding for efficient and effective dllms. arXiv preprint arXiv:2507.18578. — [arXiv:2507.18578](https://arxiv.org/abs/2507.18578) find
  10. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  11. Pengxiang Li, Yefan Zhou, Dilxat Muhtar, Lu Yin, Shilin Yan, Li Shen, Yi Liang, Soroush Vosoughi, and Shiwei Liu. 2025. Diffusion language models know the answer before decoding. arXiv preprint arXiv:2508.19982. — [arXiv:2508.19982](https://arxiv.org/abs/2508.19982) find
  12. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  13. Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. 2025. Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction. arXiv preprint arXiv:2508.02558. — [arXiv:2508.02558](https://arxiv.org/abs/2508.02558) find
  14. Xiaohang Tang, Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. 2025. wd1: Weighted policy optimization for reasoning in diffusion language models. arXiv preprint arXiv:2507.08838. — [arXiv:2507.08838](https://arxiv.org/abs/2507.08838) find
  15. Xiaoran Liu, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. 2025a. Longllada: Unlocking long context capabilities in diffusion llms. arXiv preprint arXiv:2506.14429. — [arXiv:2506.14429](https://arxiv.org/abs/2506.14429) find
  16. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275.
  17. Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. 2025b. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint arXiv:2506.06295. — [arXiv:2506.06295](https://arxiv.org/abs/2506.06295) find
  18. Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. 2025a. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307. — [arXiv:2503.00307](https://arxiv.org/abs/2503.00307) find
  19. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, and 1 others. 2024. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592.
  20. Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781. — [arXiv:2505.15781](https://arxiv.org/abs/2505.15781) find
  21. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. arXiv preprint arXiv:2502.09992.
  22. Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. 2025. Tinyzero. https://github.com/Jiayi-Pan/TinyZero. Accessed: 2025-01-24. ↗
  23. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191.
  24. Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. 2024. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184.
  25. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935.
  26. Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, and 1 others. 2025b. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291. — [arXiv:2503.10291](https://arxiv.org/abs/2503.10291) find
  27. Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, and Chunhua Shen. 2025c. Time is a feature: Exploiting temporal dynamics in diffusion language models. arXiv preprint arXiv:2508.09138. — [arXiv:2508.09138](https://arxiv.org/abs/2508.09138) find
  28. Yinjie Wang, Ling Yang, Bowen Li, Ye Tian, Ke Shen, and Mengdi Wang. 2025d. Revolutionizing reinforcement learning framework for diffusion large language models. arXiv preprint arXiv:2509.06949. — [arXiv:2509.06949](https://arxiv.org/abs/2509.06949) find
  29. Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618. — [arXiv:2505.22618](https://arxiv.org/abs/2505.22618) find
  30. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487.
  31. Fei Yu, Anningzhe Gao, and Benyou Wang. 2023. Ovm, outcome-supervised value models for planning in mathematical reasoning. arXiv preprint arXiv:2311.09724.
  32. Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. 2024. Free process rewards without process labels. arXiv preprint arXiv:2412.01981.
  33. Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024a. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735–64772.
  34. Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. 2024b. Entropy-regularized process reward model. arXiv preprint arXiv:2412.11006. find
  35. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.
  36. Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. 2025a. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216. — [arXiv:2504.12216](https://arxiv.org/abs/2504.12216) find
  37. Siyan Zhao, Mengchen Liu, Jing Huang, Miao Liu, Chenyu Wang, Bo Liu, Yuandong Tian, Guan Pang, Sean Bell, Aditya Grover, and 1 others. 2025b. Inpainting-guided policy optimization for diffusion large language models. arXiv preprint arXiv:2509.10396. — [arXiv:2509.10396](https://arxiv.org/abs/2509.10396) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2026-advancing-reasoning-diffusion-language-models/