Elamur Research⌕
☰
home / papers / 2023 / Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

Paper · 2023 · Aaron Lou, Chenlin Meng and Stefano Ermon

Summary

Autoregressive models have long dominated generative modeling of discrete data such as text. Diffusion models excel on images but have lagged on language, partly because score matching, the theory behind continuous diffusion, has no well-performing discrete counterpart. Lou, Meng and Ermon (Stanford; ICML 2024) propose score entropy, a loss built on a Bregman divergence. It trains a network to estimate ratios between probabilities of neighbouring states (the "concrete score") and keeps those estimates positive through a log-barrier effect. A denoising variant makes training scalable, and a diffusion-weighted version yields an ELBO for likelihood training and evaluation. The resulting Score Entropy Discrete Diffusion (SEDD) models outperform earlier language diffusion approaches (D3PM-style mean prediction, ratio matching, concrete score matching, and continuous embedding diffusion). At similar sizes they are competitive with GPT-2. SEDD also produces good samples without temperature or nucleus tricks, lets users trade sampling steps for quality, and supports prompting at arbitrary positions. This makes non-autoregressive text generation a credible alternative to the left-to-right paradigm.

Key points

  • Compared with existing language diffusion models of comparable size, SEDD lowers perplexity by roughly 25–75% and beats GPT-2 on its zero-shot perplexity tasks.
  • Without annealing, SEDD samples reach about 6–8× better generative perplexity than un-annealed GPT-2. Similar quality is obtained with 32× fewer network evaluations.
  • For both standard prompting and infilling, SEDD matches the MAUVE scores of autoregressive models using nucleus sampling, with no specialized training for infilling.
  • With about 100 sampling steps, inference time matches autoregressive decoding (Hugging Face transformers baseline). Because no KV-cache is needed, batch sizes can be roughly 4–6× larger.
  • Code is released at github.com/louaaron/Score-Entropy-Discrete-Diffusion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Aaron Lou, Chenlin Meng and Stefano Ermon. 2023. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. arXiv:2310.16834.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.16834 · arXiv 2310.16834

@misc{lou2023discrete,
  title = {{Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution}},
  author = {Aaron Lou, Chenlin Meng and Stefano Ermon},
  year = {2023},
  doi = {10.48550/arxiv.2310.16834},
  eprint = {2310.16834},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.16834}
}

References

49 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anderson, W. J. Continuous-time Markov chains: An applications-oriented approach. Springer Science & Business Media, 2012. find
  2. Efron, B. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106:1602 – 1614, 2011. URL https://api.semanticscholar.org/CorpusID:23284154. ↗
  3. Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021.
  4. Gillespie, D. T. Approximate accelerated stochastic simulation of chemically reacting systems. Journal of Chemical Physics, 115:1716–1733, 2001. URL https://api.semanticscholar.org/CorpusID:5109777. ↗
  5. Benton, J., Shi, Y., Bortoli, V. D., Deligiannidis, G., and Doucet, A. From denoising diffusions to denoising markov models. ArXiv, abs/2211.03595, 2022. URL https://api.semanticscholar.org/CorpusID:253384277. — [arXiv:2211.03595](https://arxiv.org/abs/2211.03595) ↗
  6. Gokaslan, A. and Cohen, V. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019. ↗
  7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Gong, S., Li, M., Feng, J., Wu, Z., and Kong, L. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
  8. Hyvärinen, A. Some extensions of score matching. Comput. Stat. Data Anal., 51:2499–2512, 2007. URL https://api.semanticscholar.org/CorpusID:2352990. ↗
  9. Graves, A., Srivastava, R. K., Atkinson, T., and Gomez, F. Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023. find
  10. Kelly, F. Reversibility and stochastic networks. 1980. URL https://api.semanticscholar.org/CorpusID:125211322. ↗
  11. Gulrajani, I. and Hashimoto, T. Likelihood-based diffusion language models. In Advances in Neural Information Processing Systems, 2023. find
  12. Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems, 2022.
  13. Han, X., Kumar, S., and Tsvetkov, Y. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17432, 2022. find
  14. Lou, A. and Ermon, S. Reflected diffusion models. In International Conference on Machine Learning. PMLR, 2023. find
  15. Hanson, F. B. Applied Stochastic Processes and Control for Jump-Diffusions: Modeling, Analysis and Computation. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2007. doi: 10.1137/1.9780898718638. URL https://epubs.siam.org/doi/abs/10. 1137/1.9780898718638. — [doi:10.1137/1.9780898718638](https://doi.org/10.1137/1.9780898718638) ↗
  16. Mahabadi, R. K., Tae, J., Ivison, H., Henderson, J., Beltagy, I., Peters, M. E., and Cohan, A. Tess: Text-to-text self-conditioned simplex diffusion. arXiv preprint arXiv:2305.08379, 2023. find
  17. He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. Diffusionbert: Improving generative masked language models with diffusion models. In Annual Meeting of the Association for Computational Linguistics, 2022.
  18. Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021. URL https://api.semanticscholar.org/CorpusID:245704504. ↗
  19. Ho, J. Classifier-free diffusion guidance. ArXiv, abs/2207.12598, 2022. URL https://api.semanticscholar.org/CorpusID: 249145348. ↗
  20. Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. In Advances in Neural Information Processing Systems, 2022. find
  21. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020. find
  22. Øksendal, B. Stochastic differential equations : an introduction with applications. Journal of the American Statistical Association, 82:948, 1987. find
  23. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
  24. Peebles, W. S. and Xie, S. Scalable diffusion models with transformers. In International Conference on Computer Vision, 2023.
  25. Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454–12465, 2021. find
  26. Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816–4828, 2021.
  27. Hutchinson, M. F. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics - Simulation and Computation, 18:1059–1076, 1989. URL https://api.semanticscholar.org/CorpusID: 120969358. ↗
  28. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID: 160025533. ↗
  29. Hyvärinen, A. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res., 6:695– 709, 2005. find
  30. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022. URL https://api.semanticscholar.org/CorpusID:248097655. ↗
  31. Shih, A., Sadigh, D., and Ermon, S. Training and inference on any-order autoregressive models the right way. Advances in Neural Information Processing Systems, 35: 2762–2775, 2022. find
  32. Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, 2017.
  33. Vincent, P. A connection between score matching and denoising autoencoders. Neural Computation, 23:1661– 1674, 2011. find
  34. Wang, A. and Cho, K. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094, 2019.
  35. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
  36. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural lan… — Transformers: State-of-the-Art Natural Language Processing
  37. Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=St1giarCHLP. ↗
  38. Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Neural Information Processing Systems, 2019.
  39. Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced score matching: A scalable approach to density and score estimation. In Conference on Uncertainty in Artificial Intelligence, 2019. find
  40. Ye, J., Zheng, Z., Bao, Y., Qian, L., and Wang, M. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025, 2023. — [arXiv:2302.10025](https://arxiv.org/abs/2302.10025) find
  41. Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. In Neural Information Processing Systems, 2021b. find
  42. Yule, G. U. On a method of investigating periodicities in disturbed series with special reference to wolfer’s sunspot numbers. Statistical Papers of George Udny Yule, pp. 389–420, 1971. find
  43. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021c. URL https://openreview.net/forum? id=PxTIG12RRHS. ↗
  44. Zheng, L., Yuan, J., Yu, L., and Kong, L. A reparameterized discrete diffusion model for text generation. ArXiv, abs/2302.05737, 2023. — [arXiv:2302.05737](https://arxiv.org/abs/2302.05737) find
  45. Ziegler, Z. and Rush, A. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, pp. 7673–7682. PMLR, 2019. find
  46. Strudel, R., Tallec, C., Altché, F., Du, Y., Ganin, Y., Mensch, A., Grathwohl, W. S., Savinov, N., Dieleman, S., Sifre, L., et al. Self-conditioned embedding diffusion for text generation. 2022. find
  47. Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.09864, 2021.
  48. Sun, H., Yu, L., Dai, B., Schuurmans, D., and Dai, H. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representations, 2023. find
  49. Tran, D., Vafa, K., Agrawal, K., Dinh, L., and Poole, B. Discrete flows: Invertible generative models of discrete data. Advances in Neural Information Processing Systems, 32, 2019. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lou-2023-discrete-diffusion-modeling-estimating-ratios/