Elamur Research⌕
☰
home / papers / 2026 / PURE: Post-hoc Unlocking and REfinement for Discrete Diffusion Decoding

PURE: Post-hoc Unlocking and REfinement for Discrete Diffusion Decoding

Paper · 2026 · Yangryeol Park, Kunhui Lee, Hanback Choi et al.

Summary

Masked diffusion language models (MDLMs) decode tokens in parallel, but under the usual monotonic unmasking scheme a token cannot be changed once it is committed. Early mistakes therefore persist. Existing remasking methods act while the sequence is still being generated. This paper instead looks at fixing a finished draft afterwards. In an oracle analysis, the authors found that remasking only the wrong tokens usually brings the same error back. They call this contextual lock-in: neighbouring tokens adapt to the error and keep supporting it. They also note that errors tend to cluster in short spans.

Their method, PURE, needs no training. It tracks how token confidence changes during drafting and uses an instability score (∆i) to choose seed positions. It then remasks each seed's local window and a random sample of earlier tokens, and runs a brief extra denoising chain. On LLaDA-8B-Instruct, PURE raises GSM8K accuracy by 12.9 points over the baseline, with only a small number of added steps. This makes post-hoc refinement an inexpensive way to turn extra test-time compute into accuracy.

Key points

  • Oracle study: run on GSM1K, using the 160 problems that LLaDA-8B-Instruct gets wrong (256 tokens, 256 steps) and a 10-step refinement budget.
    • Pointwise correction (w=0, ρ=0) gave the lowest correction rate.
    • Widening the window (w) and adding leftward relaxation (ρ) each helped, and combining them worked best.
  • Two mechanisms:
    • Deterministic window masking remasks the erroneous token plus the w tokens on each side of it.
    • Stochastic leftward relaxation remasks a fraction ρ of preceding context tokens.
  • Entropy probe: structured masking raises entropy at error positions as w and ρ grow. Count-matched random masking stays near zero.
  • Compute–quality trade-off:
    • Even B=10 refinement steps on a T=256 draft give a clear gain on GSM8K.
    • Drafts made with fewer steps recover much of their lost accuracy after refinement.
  • Limitations:
    • The method was mainly tested on reasoning tasks with a fixed output length.
    • It was designed for single-block decoding; a naive blockwise or KV-cached plug-in lost the gains.
    • It is weaker when an error requires changes at distant positions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NaverSeongnam-si · South Korea18 works · 11,678 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yangryeol Park, Kunhui Lee, Hanback Choi, Cheoneum Park, Donghyeon Jeon, Inho Kang and Seung-Hoon Na. 2026. PURE: Post-hoc Unlocking and REfinement for Discrete Diffusion Decoding. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1234

@inproceedings{park2026pure,
  title = {{PURE: Post-hoc Unlocking and REfinement for Discrete Diffusion Decoding}},
  author = {Yangryeol Park, Kunhui Lee, Hanback Choi, Cheoneum Park, Donghyeon Jeon, Inho Kang and Seung-Hoon Na},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1234},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1234}
}

References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems.
  2. Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Xiaocheng Lu, Yuxin Ma, and 12 others. 2025. Llada2.0: Scaling up diffusion language models to 1… — [arXiv:2512.15745](https://arxiv.org/abs/2512.15745) find
  3. Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and Bowen Zhou. 2025. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. Preprint, arXiv:2510.06303. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  5. Yihong Dong, Zhaoyu Ma, Xue Jiang, Zhiyuan Fan, Jiaru Qian, Yongmin Li, Jianha Xiao, Zhi Jin, Rongyu Cao, Binhua Li, Fei Huang, Yongbin Li, and Ge Li. 2025. Saber: An efficient sampling with adaptive acceleration and backtracking enhanced remasking for diffusion language model. Preprint, arXiv:2510.… — [arXiv:2510.18165](https://arxiv.org/abs/2510.18165) find
  6. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Conference on Empirical Methods in Natural Language Processing.
  7. Haoyu He, Katrin Renz, Yong Cao, and Andreas Geiger. 2025. MDPO: Overcoming the training-inference divide of masked diffusion language models. Preprint, arXiv:2508.13148. — [arXiv:2508.13148](https://arxiv.org/abs/2508.13148) find
  8. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  9. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc. find
  10. Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. 2022. Video diffusion models. In Advances in Neural Information Processing Systems, volume 35, pages 8633– 8646. Curran Associates, Inc. find
  11. Feng Hong, Geng Yu, Yushi Ye, Haicheng Huang, Huangjie Zheng, Ya Zhang, Yanfeng Wang, and Jiangchao Yao. 2025. Wide-In, Narrow-Out: Revok-able decoding for efficient and effective DLLM. In NeurIPS 2025 Workshop on Efficient Reasoning. find
  12. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems. find
  13. Zachary Horvitz, Raghav Singhal, Hao Zou, Carles Domingo-Enrich, Zhou Yu, Rajesh Ranganath, and Kathleen McKeown. 2025. No Compute Left Behind: Rethinking reasoning and sampling with masked diffusion models. Preprint, arXiv:2510.19990. — [arXiv:2510.19990](https://arxiv.org/abs/2510.19990) find
  14. Pengcheng Huang, Shuhao Liu, Zhenghao Liu, Yukun Yan, Shuo Wang, Zulong Chen, and Tong Xiao. 2025a. Pc-sampler: Position-aware calibration of decoding bias in masked diffusion models. Preprint, arXiv:2508.13021. — [arXiv:2508.13021](https://arxiv.org/abs/2508.13021) find
  15. Zemin Huang, Yuhang Wang, Zhiyang Chen, and Guo-Jun Qi. 2025b. Don’t Settle Too Early: Self-reflective remasking for diffusion language models. Preprint, arXiv:2509.23653. — [arXiv:2509.23653](https://arxiv.org/abs/2509.23653) find
  16. Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, and Sitan Chen. 2025. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. In Forty-second International Conference on Machine Learning. find
  17. Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. 2025. Mercury: Ultra-fast language models based on diffusion. Preprint, arXiv:2506.17298. find
  18. Hugo Lavenant and Giacomo Zanella. 2025. Error bounds and optimal schedules for masked diffusions with factorized approximations. Preprint, arXiv:2510.25544. — [arXiv:2510.25544](https://arxiv.org/abs/2510.25544) find
  19. Sanghyun Lee, Seungryong Kim, Jongho Park, and Dongmin Park. 2025. Lookahead unmasking elicits accurate decoding in diffusion language models. Preprint, arXiv:2511.05563. — [arXiv:2511.05563](https://arxiv.org/abs/2511.05563) find
  20. Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin. 2025a. Beyond fixed: Training-free variable-length denoising for diffusion large language models. Preprint, arXiv:2508.00819. — [arXiv:2508.00819](https://arxiv.org/abs/2508.00819) find
  21. Pengxiang Li, Yefan Zhou, Dilxat Muhtar, Lu Yin, Shilin Yan, Li Shen, Yi Liang, Soroush Vosoughi, and Shiwei Liu. 2025b. Diffusion language models know the answer before decoding. Preprint, arXiv:2508.19982. — [arXiv:2508.19982](https://arxiv.org/abs/2508.19982) find
  22. Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stark, Yilun Xu, Tommi Jaakkola, and Rafael Gomez-Bombarelli. 2025. Think while You Generate: Discrete diffusion with planned denoising. In The Thirteenth International Conference on Learning Representations. find
  23. Aaron Lou, Chenlin Meng, and Stefano Ermon. 2024. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning. find
  24. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online. Association for Computational Linguistics. find
  25. Nikita Mounier and Parsa Idehpour. 2025a. Review, Remask, Refine: Process-guided block diffusion for text generation. In ICML 2025 Workshop on Methods and Opportunities at Small Scale. find
  26. Nikita Mounier and Parsa Idehpour. 2025b. Review, remask, refine (r3): Process-guided block diffusion for text generation. Preprint, arXiv:2507.08018. — [arXiv:2507.08018](https://arxiv.org/abs/2507.08018) find
  27. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
  28. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
  29. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  30. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695. find
  31. Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. 2024. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  32. Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. 2024. Simplified and generalized masked diffusion for discrete data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  33. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265, Lille, Fr…
  34. Yang Song and Stefano Ermon. 2019. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  35. Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang, Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, and 3 others. 2025. Seed Diffusion: A large-scale diffusion language model w…
  36. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  37. Runchu Tian, Junxia Cui, Xueqiang Xu, Feng Yao, and Jingbo Shang. 2025. Finish first, perfect later: Test-time token-level cross-validation for diffusion large language models. Preprint, arXiv:2510.05090. — [arXiv:2510.05090](https://arxiv.org/abs/2510.05090) find
  38. Guanghan Wang, Yair Schiff, Subham Sahoo, and Volodymyr Kuleshov. 2025a. Remasking discrete diffusion models with inference-time scaling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  39. Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. 2025b. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing. Preprint, arXiv:2508.09192. — [arXiv:2508.09192](https://arxiv.org/abs/2508.09192) find
  40. Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. 2025a. Fast-dllm v2: Efficient block-diffusion llm. Preprint, arXiv:2509.26328. — [arXiv:2509.26328](https://arxiv.org/abs/2509.26328) find
  41. Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2025b. Fast-dLLM: Training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding. Preprint, arXiv:2505.22618. — [arXiv:2505.22618](https://arxiv.org/abs/2505.22618) find
  42. Chenxiao Yang, Cai Zhou, David Wipf, and Zhiyuan Li. 2025. On powerful ways to generate: Autoregression, diffusion, and beyond. Preprint, arXiv:2510.06190. — [arXiv:2510.06190](https://arxiv.org/abs/2510.06190) find
  43. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7B: Diffusion large language models. Preprint, arXiv:2508.15487. find
  44. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Vishnu Raja, Charlotte Zhuang, Dylan Z Slack, Qin Lyu, Sean M. Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. 2024. A careful examination of large language model performance on grade school ari… find
  45. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.
  46. Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. LLaDA 1.5: Variance-reduced preference optimization for large language diffusion models. Preprint, arXiv:2505.19223.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2026-pure-post-hoc-unlocking-refinement/