Pseudo-Likelihood Training for Reasoning Diffusion Language Models
Paper · 2026 · Shiv Shankar
Summary
Policy-gradient RL methods such as PPO and GRPO are the standard way to post-train LLMs for reasoning. They need token-level likelihoods of sampled outputs, which masked diffusion language models (dLLMs) cannot provide cheaply. Existing workarounds either use expensive Monte-Carlo ELBO estimates or a one-step mean-field approximation (diffu-GRPO/d1), and the latter produces biased gradients.
The paper introduces PADRE, which treats KL-regularized RL as inference toward a reward-tilted Boltzmann distribution. It replaces the exact sequence likelihood with a pseudo-likelihood objective: each output token is scored conditioned on all other output tokens and the prompt. The author argues this objective has the same optimum as standard KL-regularized RL. These conditionals are computed efficiently by appending a masked copy of the completion that shares position embeddings with it. Random masking of prompt tokens is added as a regularizer.
The method fine-tunes LLaDA-8B with group-normalized rewards. It outperforms d1, wd1, VRPO and Dream 7B on GSM8K, MATH500 and MATH. This makes RL fine-tuning of reasoning-focused dLLMs more principled and practical.
Key points
- The base model is LLaDA-8B, trained with the code, hyperparameters, composite format-plus-correctness reward and 0-shot prompts from d1 (Zhao et al., 2025).
- On a small test subset, the d1 one-step approximation diverges from the true likelihood with KL of 2.87 (GSM8K) and 3.15 (MATH500). PADRE's conditionals (PKL) show an average KL of roughly 1 or lower.
- The best variant, PKL⁺ (with prompt perturbation), reaches 87.3% on GSM8K, 42.4% on MATH500 (+2.2 over d1-LLaDA) and 44.3% on MATH. Dream 7B scores 42.9% on MATH.
- Two cheaper variants, PKL-U and PKL-T, are less accurate than PKL but still approximate the true likelihood better than d1. Both also beat d1 in accuracy.
- Coding benchmarks (HumanEval, MBPP) were also evaluated; those results appear in the paper's appendix.
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works
Cite
Cite as Shiv Shankar. 2026. Pseudo-Likelihood Training for Reasoning Diffusion Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).
@inproceedings{shankar2026pseudo,
title = {{Pseudo-Likelihood Training for Reasoning Diffusion Language Models}},
author = {Shiv Shankar},
booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2026},
doi = {10.18653/v1/2026.eacl-long.257},
url = {https://doi.org/10.18653/v1/2026.eacl-long.257}
}References
- Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021b. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
- Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR…
- Leemon Baird. 1995. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the Twelfth International Conference on Machine Learning, pages 30–37. find
- Julian Besag. 1975. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society Series D: The Statistician, 24(3):179–195. \#\# Conclusion find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/shankar-2026-pseudo-likelihood-training-reasoning-diffusion/