Elamur Research
☰
home / papers / 2022 / Hierarchical Text-Conditional Image Generation with CLIP Latents

Hierarchical Text-Conditional Image Generation with CLIP Latents

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2204.06125

Summary

The paper addresses text-to-image synthesis by building on CLIP's shared image–text representation space. The OpenAI authors propose unCLIP, a two-stage system. A prior maps a caption to a plausible CLIP image embedding, and a diffusion decoder turns that embedding into a picture. Two diffusion upsamplers then raise the resolution from 64×64 to 1024×1024, and CLIP itself stays frozen throughout. The authors compare two prior designs: an autoregressive Transformer over quantized embedding codes and a Gaussian diffusion prior. The diffusion variant is cheaper to compute and gives better samples. Compared with GLIDE, unCLIP matches it on photorealism and caption fidelity while producing noticeably more varied outputs. Because the decoder roughly inverts the CLIP image encoder, the same model also supports image variations, blending between two images, and zero-shot text-guided edits. Known weaknesses are binding attributes to the right objects, rendering legible text, and fine detail in cluttered scenes. The work matters as the foundation of DALL·E 2 and as a tool for revealing what CLIP embeddings keep or discard.

Key points

  • The architecture factorizes P(x|y) into a prior P(z_i|y) and a decoder P(x|z_i, y). The decoder is a modified GLIDE-style diffusion model that injects CLIP embeddings through the timestep embedding and four extra context tokens.
  • Classifier-free guidance is enabled during training in two ways. The decoder zeroes CLIP embeddings 10% of the time and drops captions 50% of the time, while the prior drops text conditioning 10% of the time.
  • For the AR prior, PCA reduces the CLIP image embeddings from 1,024 to 319 components, and each component is quantized into 1,024 buckets. This cuts the number of predicted tokens by about three times.
  • The upsamplers (64→256 and 256→1024) use only convolutions and are trained on corrupted inputs, with Gaussian blur for the first stage and BSR degradation for the second. They are unconditional ADMNets that run without guidance.
  • Images are encoded as a pair (z_i, x_T) obtained via DDIM inversion. This representation enables variations controlled by η, interpolation between images, and edits along text directions. The first deployment of the model was the DALL·E 2 Preview platform.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

396 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 378 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2204.06125 · arXiv 2204.06125

@misc{ramesh2022hierarchical,
  title = {{Hierarchical Text-Conditional Image Generation with CLIP Latents}},
  author = {Ramesh, Aditya and Dhariwal, Prafulla and Nichol, Alex and Chu, Casey and Chen, Mark},
  year = {2022},
  doi = {10.48550/arxiv.2204.06125},
  eprint = {2204.06125},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2204.06125}
}

References

63 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A Causal Masked Multimodal Model of the Internet. arXiv:2201.07520, 2022. — [arXiv:2201.07520](https://arxiv.org/abs/2201.07520) find
  2. Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models. CoRR, abs/2201.06503, 2022. URL https: //arxiv.org/abs/2201.06503. — [arXiv:2201.06503](https://arxiv.org/abs/2201.06503) find
  3. Florian Bordes, Randall Balestriero, and Pascal Vincent. High Fidelity Visualization of What Your Self-Supervised Representation Knows About. arXiv:2112.09164, 2021. — [arXiv:2112.09164](https://arxiv.org/abs/2112.09164) find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  5. Rewon Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images. arXiv:2011.10650, 2021. — [arXiv:2011.10650](https://arxiv.org/abs/2011.10650) find
  6. Katherine Crowson. AVA Linear Probe. https://twitter.com/RiversHaveWings/status/ 1472346186728173568?s=20&t=T-HRr3Gw5HRGjQaMDtRe3A, 2021. ↗
  7. Katherine Crowson. CLIP guided diffusion HQ 256x256. https://colab.research.google.com/ drive/12a\_Wrfi2\_gwwAuN3VvMTwVMz9TfqctNj, 2021. ↗
  8. Katherine Crowson. CLIP Guided Diffusion 512x512, Secondary Model Method. https://twitter. com/RiversHaveWings/status/1462859669454536711, 2021. find
  9. Katherine Crowson. v-diffusion. https://github.com/crowsonkb/v-diffusion-pytorch, 2021. ↗
  10. Karan Desai and Justin Johnson. VirTex: Learning Visual Representations from Textual Annotations. arXiv:2006.06666, 2020. — [arXiv:2006.06666](https://arxiv.org/abs/2006.06666) find
  11. Prafulla Dhariwal and Alex Nichol. arXiv:2105.05233, 2021. Diffusion Models Beat GANs on Image Synthesis. — [arXiv:2105.05233](https://arxiv.org/abs/2105.05233) find
  12. Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CogView: Mastering Text-to-Image Generation via Transformers. arXiv:2105.13290, 2021. find
  13. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.1192… find
  14. Patrick Esser, Robin Rombach, and Björn Ommer. Taming Transformers for High-Resolution Image Synthesis. arXiv:2012.09841, 2020. — [arXiv:2012.09841](https://arxiv.org/abs/2012.09841) find
  15. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimization for Efficiently Improving Generalization. arXiv:2010.01412, 2020. — [arXiv:2010.01412](https://arxiv.org/abs/2010.01412) find
  16. Andreas Fürst, Elisabeth Rumetshofer, Viet Thuong Tran, Hubert Ramsauer, Fei Tang, Johannes Lehner, D P Kreil, Michael K Kopp, Günter Klambauer, Angela Bitto-Nemling, and Sepp Hochreiter. CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP, 2022. URL https://openreview. net/forum?id=qw674L… find
  17. Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors. arXiv:2203.13131, 2022. — [arXiv:2203.13131](https://arxiv.org/abs/2203.13131) find
  18. Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. arXiv:2108.00946, 2021. — [arXiv:2108.00946](https://arxiv.org/abs/2108.00946) find
  19. Federico A. Galatolo, Mario G. C. A. Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search. arXiv:2102.01645, 2021. — [arXiv:2102.01645](https://arxiv.org/abs/2102.01645) find
  20. Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal Neurons in Artificial Neural Networks. Distill, 2021. doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons. — [doi:10.23915/distill.00030](https://doi.org/10.23915/distill.00030) ↗
  21. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. arXiv:1406.2661, 2014. find
  22. Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector Quantized Diffusion Model for Text-to-Image Synthesis. arXiv:2111.14822, 2021. — [arXiv:2111.14822](https://arxiv.org/abs/2111.14822) find
  23. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. find
  24. Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https://openreview.net/ forum?id=qw8AKxfYbI. ↗ our card
  25. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. arXiv:2006.11239, 2020. find
  26. Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. arXiv:2106.15282, 2021. — [arXiv:2106.15282](https://arxiv.org/abs/2106.15282) find
  27. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980, 2014. find
  28. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. arXiv:1405.0312, 2014. find our card
  29. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv:1711.05101, 2017. find
  30. Pamela Mishkin, Lama Ahmad, Miles Brundage, Gretchen Krueger, and Girish Sastry. DALL·E 2 Preview - Risks and Limitations. 2022. URL https://github.com/openai/dalle-2-preview/ blob/main/system-card.md. ↗
  31. Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. SLIP: Self-supervision meets Language-Image Pre-training. arXiv:2112.12750, 2021. — [arXiv:2112.12750](https://arxiv.org/abs/2112.12750) find
  32. Ryan Murdock. The Big 1351038053033406468, 2021. Sleep. https://twitter.com/advadnoun/status/ ↗
  33. Naila Murray, Luca Marchesotti, and Florent Perronnin. AVA: A large-scale database for aesthetic visual analysis. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2408–2415, 2012. doi: 10.1109/CVPR.2012.6247954. — [doi:10.1109/cvpr.2012.6247954](https://doi.org/10.1109/cvpr.2012.6247954) find
  34. Alex Nichol and Prafulla Dhariwal. arXiv:2102.09672, 2021. Improved Denoising Diffusion Probabilistic Models. — [arXiv:2102.09672](https://arxiv.org/abs/2102.09672) find
  35. Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. arXiv:2112.10741, 2021. — [arXiv:2112.10741](https://arxiv.org/abs/2112.10741) find
  36. Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. arXiv:2103.17249, 2021. — [arXiv:2103.17249](https://arxiv.org/abs/2103.17249) find
  37. Karl Pearson. LIII. On lines and planes of closest fit to systems of points in space, November 1901. URL https://doi.org/10.1080/14786440109462720. — [doi:10.1080/14786440109462720](https://doi.org/10.1080/14786440109462720) ↗
  38. Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. arXiv:2111.15640, 2021. — [arXiv:2111.15640](https://arxiv.org/abs/2111.15640) find
  39. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020, 2021. find our card
  40. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. arXiv:2102.12092, 2021. find our card
  41. Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidelity Images with VQ-VAE-2. arXiv:1906.00446, 2019. — [arXiv:1906.00446](https://arxiv.org/abs/1906.00446) find
  42. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752, 2021. find
  43. Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image Super-Resolution via Iterative Refinement. arXiv:arXiv:2104.07636, 2021. — [arXiv:2104.07636](https://arxiv.org/abs/2104.07636) find
  44. Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning Visual Representations with Caption Annotations. arXiv:2008.01392, 2020. — [arXiv:2008.01392](https://arxiv.org/abs/2008.01392) find
  45. Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How Much Can CLIP Benefit Vision-and-Language Tasks? arXiv:2107.06383, 2021. — [arXiv:2107.06383](https://arxiv.org/abs/2107.06383) find
  46. Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. arXiv:1503.03585, 2015. find our card
  47. Jiaming Song, Chenlin Meng, and Stefano Ermon. arXiv:2010.02502, 2020. Denoising Diffusion Implicit Models. find our card
  48. Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models. arXiv:2006.09011, 2020. — [arXiv:2006.09011](https://arxiv.org/abs/2006.09011) find
  49. Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao. DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis. arXiv:2008.05865, 2020. — [arXiv:2008.05865](https://arxiv.org/abs/2008.05865) find
  50. Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. arXiv:2007.03898, 2020. — [arXiv:2007.03898](https://arxiv.org/abs/2007.03898) find
  51. Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based Generative Modeling in Latent Space. In Neural Information Processing Systems (NeurIPS), 2021. find
  52. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. arXiv:1711.00937, 2017. — [arXiv:1711.00937](https://arxiv.org/abs/1711.00937) find
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762, 2017. find
  54. Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP. arXiv:2203.00386, 2022. — [arXiv:2203.00386](https://arxiv.org/abs/2203.00386) find
  55. Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. GAN Inversion: A Survey. arXiv:2101.05278, 2021. — [arXiv:2101.05278](https://arxiv.org/abs/2101.05278) find
  56. Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. arXiv:1711.10485, 2017. — [arXiv:1711.10485](https://arxiv.org/abs/1711.10485) find
  57. Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving Text-to-Image Synthesis Using Contrastive Learning. arXiv:2107.02423, 2021. — [arXiv:2107.02423](https://arxiv.org/abs/2107.02423) find
  58. Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-Modal Contrastive Learning for Text-to-Image Generation. arXiv:2101.04702, 2021. — [arXiv:2101.04702](https://arxiv.org/abs/2101.04702) find
  59. Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a Practical Degradation Model for Deep Blind Image Super-Resolution. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021. doi: 10.1109/iccv48922.2021.00475. URL http://dx.doi.org/10.1109/ ICCV48922.2021.00475. — [doi:10.1109/iccv48922.2021.00475](https://doi.org/10.1109/iccv48922.2021.00475) find
  60. Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive Learning of Medical Visual Representations from Paired Images and Text. arXiv:2010.00747, 2020. — [arXiv:2010.00747](https://arxiv.org/abs/2010.00747) find
  61. Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. LAFITE: Towards Language-Free Training for Text-to-Image Generation. arXiv:2111.13792, 2021. — [arXiv:2111.13792](https://arxiv.org/abs/2111.13792) find
  62. Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative Visual Manipulation on the Natural Image Manifold. arXiv:1609.03552, 2016. — [arXiv:1609.03552](https://arxiv.org/abs/1609.03552) find
  63. Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis. arXiv:1904.01310, 2019. — [arXiv:1904.01310](https://arxiv.org/abs/1904.01310) find