Elamur Research
☰
home / papers / 2020 / Denoising Diffusion Implicit Models

Denoising Diffusion Implicit Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2010.02502

Summary

Diffusion-based image generators such as DDPM produce samples rivaling GANs, but generation is very slow because the reverse chain must run through every step of a long noising process, typically T = 1000. The authors propose DDIM, which keeps the DDPM training procedure but changes how samples are drawn. They note that the DDPM loss depends only on per-step marginals of the noising process, not on its full joint distribution. This lets them build a family of non-Markovian forward processes, controlled by a variance parameter σ, that share the same surrogate objective. Setting σ to zero gives a deterministic, implicit generator that can skip timesteps and sample along a short sub-sequence. A model trained once as a DDPM can therefore be reused without retraining. On CIFAR10, CelebA and LSUN, DDIM keeps sample quality much better than DDPM when steps are cut sharply, achieving 10× to 50× wall-clock speedups. Its deterministic latent mapping also enables meaningful interpolation and accurate reconstruction, which makes diffusion models far more practical.

Key points

  • Motivating cost: drawing 50k 32×32 images from a DDPM takes about 20 hours on an Nvidia 2080 Ti, while a GAN needs under a minute. At 256×256 resolution, the DDPM time approaches 1000 hours.
  • Theorem 1: for every σ > 0, the variational objective J_σ equals a weighted DDPM loss L_γ up to a constant. The parameter η interpolates between deterministic DDIM (η = 0) and DDPM-like sampling (η = 1).
  • Experimental setup: experiments use CIFAR10 (32×32), CelebA (64×64), LSUN Bedroom and LSUN Church (both 256×256). The models are pretrained DDPM U-Net (Wide ResNet) checkpoints, except CelebA, which the authors trained themselves.
  • Results with 20 sampling steps (FID):
    • LSUN Bedroom: DDIM 8.89 vs DDPM 22.77.
    • LSUN Church: DDIM 12.47 vs DDPM 23.37.
    • For reference, 1000-step DDPM scores 6.36 (Bedroom) and 7.89 (Church).
    • At 100 steps on Church, DDPM is slightly better (8.27 vs 10.58).
  • Consistency: the same initial x_T yields images with similar high-level features regardless of chain length. Spherical linear interpolation (slerp) of x_T values produces smooth semantic interpolations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

360 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 353 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2010.02502 · arXiv 2010.02502

@misc{song2020denoising,
  title = {{Denoising Diffusion Implicit Models}},
  author = {Song, Jiaming and Meng, Chenlin and Ermon, Stefano},
  year = {2020},
  doi = {10.48550/arxiv.2010.02502},
  eprint = {2010.02502},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.02502}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875, January 2017. — [arXiv:1701.07875](https://arxiv.org/abs/1701.07875) find
  2. David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what a gan cannot generate. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4502–4511, 2019. find
  3. Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski. Deep generative stochastic networks trainable by backprop. In International Conference on Machine Learning, pp. 226–234, January 2014. find
  4. Christopher M Bishop. Pattern recognition and machine learning. springer, 2006. find
  5. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, September 2018. — [arXiv:1809.11096](https://arxiv.org/abs/1809.11096) find
  6. John Charles Butcher and Nicolette Goodwin. Numerical methods for ordinary differential equations, volume 2. Wiley Online Library, 2008. find
  7. Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713, September 2020. — [arXiv:2009.00713](https://arxiv.org/abs/2009.00713) find
  8. Ricky T Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint arXiv:1806.07366, June 2018. — [arXiv:1806.07366](https://arxiv.org/abs/1806.07366) find
  9. Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. arXiv preprint arXiv:1605.08803, May 2016. — [arXiv:1605.08803](https://arxiv.org/abs/1605.08803) find
  10. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014. find
  11. Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning a transition operator as a stochastic recurrent net. In Advances in Neural Information Processing Systems, pp. 4392–4402, 2017. find
  12. Will Grathwohl, Ricky T Q Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, October 2018. — [arXiv:1810.01367](https://arxiv.org/abs/1810.01367) find
  13. Wasserstein GAN. arXiv preprint Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5769–5779, 2017. find
  14. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two Time-Scale update rule converge to a local nash equilibrium. arXiv preprint arXiv:1706.08500, June 2017. — [arXiv:1706.08500](https://arxiv.org/abs/1706.08500) find
  15. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, June 2020. find
  16. Aapo Hyvärinen. Estimation of Non-Normalized statistical models by score matching. Journal of Machine Learning Researc h, 6:695–709, 2005. find
  17. Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Rémi Tachet des Combes, and Ioannis Mitliagkas. Adversarial score matching and improved sampling for image generation. September 2020. find
  18. Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker– planck equation. SIAM journal on mathematical analysis, 29(1):1–17, 1998. find
  19. Tero Karras, Samuli Laine, and Timo Aila. A Style-Based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04948, December 2018. — [arXiv:1812.04948](https://arxiv.org/abs/1812.04948) find
  20. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119, 2020. find
  21. Diederik P Kingma and Max Welling. arXiv:1312.6114v10, December 2013. find our card
  22. Daniel Levy, Matthew D Hoffman, and Jascha Sohl-Dickstein. Generalizing hamiltonian monte carlo with neural networks. arXiv preprint arXiv:1711.09268, 2017. — [arXiv:1711.09268](https://arxiv.org/abs/1711.09268) find
  23. Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv:1610.03483, October 2016. — [arXiv:1610.03483](https://arxiv.org/abs/1610.03483) find
  24. Radford M Neal et al. Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011. find
  25. Alejandro F Queiruga, N Benjamin Erichson, Dane Taylor, and Michael W Mahoney. Continuous-in-depth neural networks. arXiv preprint arXiv:2008.02389, 2020. — [arXiv:2008.02389](https://arxiv.org/abs/2008.02389) find
  26. Martin Raphan and Eero P Simoncelli. Least squares estimation without priors or supervision. Neural computation, 23(2):374–420, February 2011. ISSN 0899-7667, 1530-888X. find
  27. Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770, May 2015. find
  28. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014. find our card
  29. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer, 2015. find our card
  30. Tim Salimans, Diederik P Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. arXiv preprint arXiv:1410.6460, October 2014. — [arXiv:1410.6460](https://arxiv.org/abs/1410.6460) find
  31. Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245–254, 1985. find
  32. Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.03585, March 2015. find our card
  33. Auto-Encoding variational bayes. arXiv preprint Jiaming Song, Shengjia Zhao, and Stefano Ermon. A-nice-mc: Adversarial training for mcmc. arXiv preprint arXiv:1706.07561, June 2017. — [arXiv:1706.07561](https://arxiv.org/abs/1706.07561) find
  34. Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv:1907.05600, July 2019. find our card
  35. Yang Song and Stefano Ermon. Improved techniques for training Score-Based generative models. arXiv preprint arXiv:2006.09011, June 2020. — [arXiv:2006.09011](https://arxiv.org/abs/2006.09011) find
  36. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. find our card
  37. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, September 2016a. find our card
  38. Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, January 2016b. — [arXiv:1601.06759](https://arxiv.org/abs/1601.06759) find
  39. Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011. find
  40. Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, May 2016. — [arXiv:1605.07146](https://arxiv.org/abs/1605.07146) find
  41. Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empirical study. In Advances in Neural Information Processing Systems, pp. 10792–10801, 2018. find