Elamur Research⌕
☰
home / papers / 2024 / Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation

Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation

Paper · 2024 · Mingyuan Zhou, Huangjie Zheng, Zhendong Wang et al.

Summary

Diffusion models produce excellent images, but sampling requires many passes through the network. For example, EDM uses 35 NFE on CIFAR-10 32x32 and 511 NFE on ImageNet 64x64. Score identity Distillation (SiD) compresses a pretrained diffusion model into a generator that creates an image in a single forward pass, and it does so without any real training data.

The authors treat the forward diffusion of both real and generated data as semi-implicit distributions. This viewpoint yields three score-related identities. With them, the authors approximate a model-based score-matching objective, a variant of Fisher divergence that would otherwise be intractable. The objective compares the scores of diffused real and fake data across noise levels and is estimated with score networks plus Monte Carlo sampling. The generator learns only from its own synthesized images.

The method was tested by distilling VP-EDM teachers on CIFAR-10, ImageNet 64x64, FFHQ 64x64 and AFHQv2 64x64. SiD reaches state-of-the-art one-step FID on all four datasets and sometimes matches or beats the teacher. FID also drops very quickly during training. The practical benefit is diffusion-level image quality at a fraction of the inference cost.

Key points

  • Positioning: SiD extends the Diffusion-GAN line of work, along with Diff-Instruct and DMD. Those methods match diffused distributions using JSD or KL divergence, whereas SiD matches scores with a Fisher-divergence-style loss. Code (PyTorch) is at github.com/mingyuanzhou/SiD, and the paper appeared at ICML 2024.
  • ImageNet 64x64 progress: With α = 1.0, batch size 1024 and learning rate 5e-6, FID falls from 153.52 at the start to 3.07 after about 50M generated images (around 49K iterations).
  • CIFAR-10 efficiency: On unconditional CIFAR-10, SiD beats Diff-Instruct and DMD after fewer than 20M images. That takes under 10 hours on 16 A100-40GB GPUs, or about 20 hours on 8 V100-16GB GPUs.
  • Per-iteration cost: Each iteration is about 28% slower than Diff-Instruct on CIFAR-10 and ImageNet, and about 36% slower on FFHQ and AFHQ. The extra time comes from backpropagating through both score networks.
  • Convergence trend: Log-FID decreases roughly linearly against the log of the number of processed images, which the authors describe as exponentially fast improvement. The trend eventually levels off.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 328 works

Cite

Cite as Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin and Hai Huang. 2024. Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation. arXiv:2404.04057.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.04057 · arXiv 2404.04057

@misc{zhou2024score,
  title = {{Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation}},
  author = {Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin and Hai Huang},
  year = {2024},
  doi = {10.48550/arxiv.2404.04057},
  eprint = {2404.04057},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.04057}
}

References

80 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bao, F., Li, C., Cao, Y., and Zhu, J. All are worth words: A ViT backbone for score-based diffusion models. arXiv preprint arXiv:2209.12152, 2022. find
  2. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626–6637, 2017. find
  3. Berthelot, D., Autef, A., Lin, J., Yap, D. A., Zhai, S., Hu, S., Zheng, D., Talbott, W., and Gu, E. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.04248, 2023. — [arXiv:2303.04248](https://arxiv.org/abs/2303.04248) find
  4. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020. find
  5. Brock, A., Donahue, J., and Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=B1xsqj09Fm. ↗
  6. Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1–33, 2022. find
  7. Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188–8197, 2020. find
  8. Holmes, C. C. and Walker, S. G. Assigning a value to a power likelihood in a general Bayesian model. Biometrika, 104(2):497–503, 2017. find
  9. Chung, H., Sim, B., Ryu, D., and Ye, J. C. Improving Hong, M., Wai, H.-T., Wang, Z., and Yang, Z. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33(1):147–180, 2023. find
  10. Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
  11. Liu, L., Ren, Y., Lin, Z., and Zhao, Z. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum? id=PlKWVd2yBkY. ↗
  12. Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. ↗
  13. Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022b.
  14. Jabri, A., Fleet, D., and Chen, T. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212.11972, 2022. — [arXiv:2212.11972](https://arxiv.org/abs/2212.11972) find
  15. Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum… ↗
  16. Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401–4410, 2019. find
  17. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020. find
  18. Luhman, E. and Luhman, T. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021. — [arXiv:2101.02388](https://arxiv.org/abs/2101.02388) find
  19. Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=k7FuTOWMOc7. ↗
  20. Luo, C. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022. — [arXiv:2208.11970](https://arxiv.org/abs/2208.11970) find
  21. Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. Latent consistency models: Synthesizing high-resolution images with few-step inference. ArXiv, abs/2310.04378, 2023a.
  22. Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023. find
  23. Luo, S., Tan, Y., Patil, S., Gu, D., von Platen, P., Passos, A., Huang, L., Li, J., and Zhao, H. Lcm-lora: A universal stable-diffusion acceleration module, 2023b. find
  24. Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, Z. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023c. URL https://openreview.net/ forum?id=MLIs5iRq4w. ↗
  25. Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
  26. Kingma, D. P., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2021. — [arXiv:2107.00630](https://arxiv.org/abs/2107.00630) find
  27. Lyu, S. Interpretation and generalization of score matching. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 359–366, 2009. find
  28. Krizhevsky, A. et al. Learning multiple layers of features from tiny images. 2009. find
  29. Lyu, Z., Xu, X., Yang, C., Lin, D., and Dai, B. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205.12524, 2022. — [arXiv:2205.12524](https://arxiv.org/abs/2205.12524) find
  30. Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019. find
  31. Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297–14306, 2023. find
  32. Lawson, J., Tucker, G., Dai, B., and Ranganath, R. Energy-inspired models: Learning with sampler-induced distributions. Advances in Neural Information Processing Systems, 32, 2019. find
  33. Moens, V., Ren, H., Maraval, A., Tutunov, R., Wang, J., and Ammar, H. Efficient semi-implicit variational inference. arXiv preprint arXiv:2101.06070, 2021. — [arXiv:2101.06070](https://arxiv.org/abs/2101.06070) find
  34. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Sys…
  35. Molchanov, D., Kharitonov, V., Sobolev, A., and Vetrov, D. Doubly semi-implicit variational inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2593–2602. PMLR, 2019. find
  36. Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TIdIXIpzhoI. ↗
  37. Nguyen, T. H. and Tran, A. SwiftBrush: One-step text-to-image diffusion model with variational score distillation. arXiv preprint arXiv:2312.05239, 2023. — [arXiv:2312.05239](https://arxiv.org/abs/2312.05239) find
  38. Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162–8171. PMLR, 2021. find
  39. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pp. 2234–2242, 2016.
  40. Pandey, K., Mukherjee, A., Rai, P., and Kumar, A. Dif-fuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.00308, 2022. — [arXiv:2201.00308](https://arxiv.org/abs/2201.00308) find
  41. Sauer, A., Chitta, K., Müller, J., and Geiger, A. Projected gans converge faster. Advances in Neural Information Processing Systems, 34:17480–17492, 2021. find
  42. Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. ArXiv, abs/2311.17042, 2023. — [arXiv:2311.17042](https://arxiv.org/abs/2311.17042) find
  43. Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023.
  44. Shen, H., Xiao, Q., and Chen, T. On penalty-based bilevel gradient descent method. arXiv preprint arXiv:2302.05185, 2023. — [arXiv:2302.05185](https://arxiv.org/abs/2302.05185) find
  45. Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dreamfusion: Text-to-3d using 2d diffusion. ArXiv, abs/2209.14988, 2022. — [arXiv:2209.14988](https://arxiv.org/abs/2209.14988) find
  46. Sobolev, A. and Vetrov, D. P. Importance weighted hierarchical variational inference. Advances in Neural Information Processing Systems, 32, 2019. find
  47. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022.
  48. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
  49. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, pp. 1278–1286, 2014.
  50. Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
  51. Robbins, H. E. An empirical Bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pp. 388–394. Springer, 1992. find
  52. Song, Y. and Dhariwal, P. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189, 2023. find
  53. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.
  54. Song, Y. and Ermon, S. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems, pp. 11918–11930, 2019.
  55. Ronneberger, O., P.Fischer, and Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351 of LNCS, pp. 234–241. Springer, 2015. URL http://lmb.informatik.uni-freiburg.de/Publications/2015/RFB15a. (av… ↗
  56. Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consistency models. arXiv preprint arXiv:2303.01469, 2023. find
  57. Tee, J. T. J., Zhang, K., Kim, C., Gowda, D. N., Yoon, H. S., and Yoo, C. D. Physics informed distillation for diffusion models, 2024. URL https://openreview.net/ forum?id=a24gfxA7jD. ↗
  58. Titsias, M. K. and Ruiz, F. Unbiased implicit variational inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 167–176. PMLR, 2019. find
  59. Yin, M. and Zhou, M. Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5660–5669, 2018. find
  60. Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation, 2023. find
  61. Vahdat, A. and Kautz, J. NVAE: A deep hierarchical variational autoencoder. In Advances in neural information processing systems, 2020. find
  62. Yu, L. and Zhang, C. Semi-implicit variational inference via score matching. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=sd90a2ytrt. ↗
  63. Vincent, P. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7): 1661–1674, 2011. find
  64. Wang, Z., Jiang, Y., Zheng, H., Wang, P., He, P., Wang, Z., Chen, W., and Zhou, M. Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint arXiv:2304.12526, 2023a. — [arXiv:2304.12526](https://arxiv.org/abs/2304.12526) find
  65. Yu, L., Xie, T., Zhu, Y., Yang, T., Zhang, X., and Zhang, C. Hierarchical semi-implicit variational inference with application to diffusion model acceleration. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id=ghIBaprxsV. ↗
  66. Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, J. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, 2023b. find
  67. Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=Loek7hfb46P. ↗
  68. Wang, Z., Zheng, H., He, P., Chen, W., and Zhou, M. Diffusion-GAN: Training GANs with diffusion. In The Eleventh International Conference on Learning Representations, 2023c. URL https://openreview.net/ forum?id=HZf7UbpWHuA. ↗
  69. Zhao, S., Liu, Z., Lin, J., Zhu, J.-Y., and Han, S. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33: 7559–7570, 2020. find
  70. Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681–688. Citeseer, 2011. find
  71. Zheng, H. and Zhou, M. Exploiting chain rule and Bayes’ theorem to compare probability distributions. Advances in Neural Information Processing Systems, 34:14993– 15006, 2021. find
  72. Xiao, Z., Kreis, K., and Vahdat, A. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=JprM0p-q0Co. ↗
  73. Xu, Y., Zhao, Y., Xiao, Z., and Hou, T. UFOGen: You forward once large scale text-to-image generation via diffusion GANs. ArXiv, abs/2311.09257, 2023. — [arXiv:2311.09257](https://arxiv.org/abs/2311.09257) find
  74. Zheng, H., He, P., Chen, W., and Zhou, M. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=HDxgaKk956l. ↗
  75. Xue, S., Yi, M., Luo, W., Zhang, S., Sun, J., Li, Z., and Ma, Z.-M. SA-Solver: Stochastic Adams solver for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2023. find
  76. Zheng, H., Nie, W., Vahdat, A., Azizzadenesheli, K., and Anandkumar, A. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning, pp. 42390–42402. PMLR, 2023b. find
  77. Yang, Y., Martin, R., and Bondell, H. Variational approximations using Fisher divergence. arXiv preprint arXiv:1905.05284, 2019. — [arXiv:1905.05284](https://arxiv.org/abs/1905.05284) find
  78. Zheng, H., Wang, Z., Yuan, J., Ning, G., He, P., You, Q., Yang, H., and Zhou, M. Learning stackable and skippable LEGO bricks for efficient, reconfigurable, and variable-resolution diffusion modeling, 2023c. find
  79. Ye, J., Zhu, D., and Zhu, Q. J. Exact penalization and necessary optimality conditions for generalized bilevel programming problems. SIAM Journal on optimization, 7(2):481–507, 1997. find
  80. Ye, S. and Liu, F. Score mismatching for generative modeling. arXiv preprint arXiv:2309.11043, 2023. — [arXiv:2309.11043](https://arxiv.org/abs/2309.11043) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2024-score-identity-distillation-exponentially-fast/