Elamur Research⌕
☰
home / papers / 2026 / Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models

Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models

Paper · 2026 · Linhao Zhong, Linyu Wu, Bozhen Fang et al.

Summary

Masked diffusion language models (MDLMs) generate text in parallel by gradually unmasking positions. They have two weaknesses. First, once a token is committed, it can never be revised. Second, the model predicts distributions for every position at each step but keeps only a few, so most of that work is thrown away.

EvoToken-DLM addresses both by representing undecided tokens as soft probability mixtures over the vocabulary. Each token passes through four stages: mask, mask-blended soft token, pure soft token, and finally a discrete token. Soft embeddings are built from the top-K predicted candidates, and a ratio α controls how much of the mask embedding is mixed in.

Training uses continuous trajectory supervision. The model unrolls several consecutive refinement steps and receives a loss at each one, so training mirrors how inference actually works. The approach needs no architectural changes and can be adapted from existing MDLMs with light fine-tuning. It works with KV-caching and blockwise diffusion.

Across several backbones and benchmarks, it consistently beats binary-masking baselines with negligible added latency. This makes revisable decoding a practical, general upgrade for diffusion LMs.

Key points

  • The motivating example uses a 30-token prompt and 512 generated tokens. Only 2 of 542 predicted distributions are used per step; the other 540 are discarded.
  • Token states: [MASK] → Soft([MASK] ∪ V) → Soft(V) → [Decode]. The final token is the highest-confidence prediction seen after the token entered the Soft(V) stage.
  • The method was evaluated on three backbones, with results for LLaDA-1.5 in the appendix:
    • LLaDA-Instruct-8B;
    • Dream-Instruct-7B;
    • the blockwise model D2F-LLaDA.
  • Benchmarks mentioned include Countdown and SVAMP. Performance is robust across a wide range of top-K values.
  • Limitation: adapting backbones initialized from autoregressive models is harder and converges more slowly. Code is available at github.com/aim-uofa/EvoTokenDLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Linhao Zhong, Linyu Wu, Bozhen Fang, Tianjian Feng, Chenchen Jing, Wen Wang, Jiaheng Zhang, Hao Chen and Chunhua Shen. 2026. Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.315

@inproceedings{zhong2026beyond,
  title = {{Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models}},
  author = {Linhao Zhong, Linyu Wu, Bozhen Fang, Tianjian Feng, Chenchen Jing, Wen Wang, Jiaheng Zhang, Hao Chen and Chunhua Shen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.315},
  url = {https://doi.org/10.18653/v1/2026.acl-long.315}
}

References

47 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  2. Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. 2023. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11575– 11596. find
  3. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769. find
  4. Michael Hersche, Samuel Moor-Smith, Thomas Hofmann, and Abbas Rahimi. 2025. Soft-masked diffusion language models. arXiv preprint arXiv:2510.17206. — [arXiv:2510.17206](https://arxiv.org/abs/2510.17206) find
  5. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840– 6851. find
  6. Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. 2025. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573. — [arXiv:2503.09573](https://arxiv.org/abs/2503.09573) find
  7. Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Nicklas Majamäki, Navdeep Jaitly, Yi-An Ma, and Lianhui Qin. 2025. Ladir: Latent diffusion enhances llms for text reasoning. arXiv preprint arXiv:2510.08558. find
  8. Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993.
  9. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  10. Wenrui Bao, Zhiben Chen, Dan Xu, and Yuzhang Shang. 2025. Learning to parallel: Accelerating diffusion large language models via learnable parallel decoding. arXiv preprint arXiv:2509.25188. find
  11. Xiaoran Liu, Yuerong Song, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. 2025a. Longllada: Unlocking long context capabilities in diffusion llms. arXiv preprint arXiv:2506.14429. — [arXiv:2506.14429](https://arxiv.org/abs/2506.14429) find
  12. Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, and 1 others. 2025. Llada2. 0: Scaling up diffusion language models to 100b. arXiv preprint arXiv:2512.15745. — [arXiv:2512.15745](https://arxiv.org/abs/2512.15745) find
  13. Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. 2025. dparallel: Learnable parallel decoding for dllms. arXiv preprint arXiv:2509.26488. — [arXiv:2509.26488](https://arxiv.org/abs/2509.26488) find
  14. Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and 1 others. 2025. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. arXiv preprint arXiv:2510.06303. find
  15. Yangzhou Liu, Yue Cao, Hao Li, Gen Luo, Zhe Chen, Weiyun Wang, Xiaobo Liang, Biqing Qi, Lijun Wu, Changyao Tian, and 1 others. 2025b. Sequential diffusion language models. arXiv preprint arXiv:2509.24007. — [arXiv:2509.24007](https://arxiv.org/abs/2509.24007) find
  16. Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. 2025c. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint arXiv:2506.06295. — [arXiv:2506.06295](https://arxiv.org/abs/2506.06295) find
  17. Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025a. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781. — [arXiv:2505.15781](https://arxiv.org/abs/2505.15781) find
  18. Yuxin Ma, Lun Du, Lanning Wei, Kun Chen, Qian Xu, Kangyu Wang, Guofeng Feng, Guoshan Lu, Lin Liu, Xiaojing Qi, and 1 others. 2025b. dinfer: An efficient inference framework for diffusion language models. arXiv preprint arXiv:2510.08666. find
  19. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. 2025. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages…
  20. Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR. find
  21. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. arXiv preprint arXiv:2502.09992.
  22. Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. 2025. Tinyzero. https://github.com/Jiayi-Pan/TinyZero. Accessed: 2025-01-24. ↗
  23. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191.
  24. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952.
  25. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695.
  26. Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. 2024. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184.
  27. Jucheng Shen, Gaurav Sarkar, Yeonju Ro, Sharath Nit-tur Sridhar, Zhangyang Wang, Aditya Akella, and Souvik Kundu. 2025. Improving the throughput of diffusion-based large language models via a training-free confidence-aware calibration. arXiv preprint arXiv:2512.07173.
  28. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
  29. Yang Song and Stefano Ermon. 2019. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32.
  30. Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. 2025. Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction. arXiv preprint arXiv:2508.02558. — [arXiv:2508.02558](https://arxiv.org/abs/2508.02558) find
  31. Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, and Chunhua Shen. 2025a. Time is a feature: Exploiting temporal dynamics in diffusion language models. arXiv preprint arXiv:2508.09138. — [arXiv:2508.09138](https://arxiv.org/abs/2508.09138) find
  32. Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. 2025b. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09192. — [arXiv:2508.09192](https://arxiv.org/abs/2508.09192) find
  33. Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. 2025a. Fast-dllm v2: Efficient block-diffusion llm. arXiv preprint arXiv:2509.26328. — [arXiv:2509.26328](https://arxiv.org/abs/2509.26328) find
  34. Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025b. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618. — [arXiv:2505.22618](https://arxiv.org/abs/2505.22618) find
  35. Linyu Wu, Linhao Zhong, Wenjie Qu, Yuexin Li, Yue Liu, Shengfang Zhai, Chunhua Shen, and Jiaheng Zhang. 2025c. Dmark: Order-agnostic watermarking for diffusion large language models. arXiv preprint arXiv:2510.02902. — [arXiv:2510.02902](https://arxiv.org/abs/2510.02902) find
  36. Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. 2025a. Softcot: Soft chain-of-thought for efficient reasoning with llms. arXiv preprint arXiv:2502.12134.
  37. Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. 2025b. SoftCoT++: Test-time scaling with soft chain-of-thought reasoning. arXiv preprint arXiv:2505.11484. — [arXiv:2505.11484](https://arxiv.org/abs/2505.11484) find
  38. Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. 2025. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505.15809. — [arXiv:2505.15809](https://arxiv.org/abs/2505.15809) find
  39. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487.
  40. Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. 2025. Soft thinking: Unlocking the reasoning potential of llms in continuous concept space. arXiv preprint arXiv:2505.15778.
  41. Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. 2025. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216. — [arXiv:2504.12216](https://arxiv.org/abs/2504.12216) find
  42. Huangjie Zheng, Shansan Gong, Ruixiang Zhang, Tianrong Chen, Jiatao Gu, Mingyuan Zhou, Navdeep Jaitly, and Yizhe Zhang. 2025. Continuously augmented discrete diffusion model for categorical generative modeling. arXiv preprint arXiv:2510.01329. — [arXiv:2510.01329](https://arxiv.org/abs/2510.01329) find
  43. Linhao Zhong, Linyu Wu, Wen Wang, Yuling Xi, Chenchen Jing, Jiaheng Zhang, Hao Chen, and Chunhua Shen. 2026. Efficient self-evaluation for diffusion language models via sequence regeneration. arXiv preprint arXiv:2603.02760.
  44. Cai Zhou, Chenyu Wang, Dinghuai Zhang, Shangyuan Tong, Yifei Wang, Stephen Bates, and Tommi Jaakkola. 2025a. Next semantic scale prediction via hierarchical diffusion language models. arXiv preprint arXiv:2510.08632. — [arXiv:2510.08632](https://arxiv.org/abs/2510.08632) find
  45. Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, and Dinghuai Zhang. 2025b. Coevolutionary continuous discrete diffusion: Make your diffusion language model a latent reasoner. arXiv preprint arXiv:2510.03206. — [arXiv:2510.03206](https://arxiv.org/abs/2510.03206) find
  46. Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and 1 others. 2025. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505.19223.
  47. Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang, and Jianfeng Gao. 2025. Text generation beyond discrete token sampling. arXiv preprint arXiv:2505.14827. — [arXiv:2505.14827](https://arxiv.org/abs/2505.14827) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2026-beyond-hard-masks-progressive-token/