Elamur Research⌕
☰
home / papers / 2022 / CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Paper · 2022 · Wenyi Hong, Ming Ding, Wendi Zheng et al.

Summary

Autoregressive transformers work well for text-to-image generation, but extending them to video is difficult. Training from scratch is very expensive, and paired text-video data is scarce and often only loosely matched. Fixed-length clipping also separates captions from the actions they describe. CogVideo addresses these problems with a 9.4B-parameter transformer that starts from the pretrained text-to-image model CogView2 instead of random weights. It uses dual-channel attention to add temporal modeling on top of the inherited image knowledge.

The authors also introduce multi-frame-rate hierarchical training. A frame-rate token is prepended to the text, and frames are sampled at the slowest rate that still yields five frames, so each training sample covers a whole action. One model generates key frames in order. A second model, sharing the same CogLM structure with a different attention mask, recursively fills in the frames between them. Swin-style attention, extended to autoregressive generation, reduces compute and memory costs.

CogVideo produces 480×480 clips and beats publicly available systems in both automatic and human evaluation. It is presented as the first open-source large-scale pretrained general-domain text-to-video model.

Key points

  • Scale and output: CogVideo has 9.4B parameters and was trained on 5.4M text-video pairs. It generates 4-second, 32-frame clips at 480×480, and its prompts are in Chinese.
  • Data scarcity: The authors cite VATEX, the largest annotated text-video dataset, at only 41,250 videos. Retrieval-based sources such as Howto100M are only weakly relevant to their captions.
  • Tokenization: A VQVAE tokenizer turns each frame into 20×20 = 400 tokens, and each training sample contains 5 frames.
  • Hierarchy ablation: On a 5,000-sample Kinetics-600 test subset (FVD, lower is better), CogVideo scores 108.27. One-stage sliding-window generation scores 137.13 with N_overlap = 1 and 120.82 with N_overlap = 2.
  • Initialization ablation: On the same subset, a model initialized from CogView2 but without video pretraining scores 124.92, and a randomly initialized model scores 166.13. CogView2 weights also make the training loss fall faster. Code and models are released at github.com/THUDM/CogVideo.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,344 works

Cite

Cite as Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu and Jie Tang. 2022. CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers. arXiv:2205.15868.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.15868 · arXiv 2205.15868

@misc{hong2022cogvideo,
  title = {{CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers}},
  author = {Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu and Jie Tang},
  year = {2022},
  doi = {10.48550/arxiv.2205.15868},
  eprint = {2205.15868},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.15868}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  2. J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. find
  3. J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018. — [arXiv:1808.01340](https://arxiv.org/abs/1808.01340) find
  4. A. Clark, J. Donahue, and K. Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571, 2019. — [arXiv:1907.06571](https://arxiv.org/abs/1907.06571) find
  5. M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
  6. M. Ding, W. Zheng, W. Hong, and J. Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204.14217, 2022. — [arXiv:2204.14217](https://arxiv.org/abs/2204.14217) find
  7. P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:2012.09841, 2020. — [arXiv:2012.09841](https://arxiv.org/abs/2012.09841) find
  8. C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016. find
  9. S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638, 2022. — [arXiv:2204.03638](https://arxiv.org/abs/2204.03638) find
  10. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014. find
  11. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022. — [arXiv:2204.03458](https://arxiv.org/abs/2204.03458) find
  12. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014. find
  13. J. Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7083–7093, 2019. find
  14. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  15. P. Luc, A. Clark, S. Dieleman, D. d. L. Casas, Y. Doron, A. Cassirer, and K. Simonyan. Transformation-based adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.04035, 2020. — [arXiv:2003.04035](https://arxiv.org/abs/2003.04035) find
  16. A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630–2640, 2019. find
  17. R. Rakhimov, D. Volkhonskiy, A. Artemov, D. Zorin, and E. Burnaev. Latent video transformer. arXiv preprint arXiv:2006.10704, 2020. — [arXiv:2006.10704](https://arxiv.org/abs/2006.10704) find
  18. A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.
  19. M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830–2839, 2017. find
  20. M. Saito, S. Saito, M. Koyama, and S. Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128(10):2586–2606, 2020. find
  21. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2234–2242, 2016.
  22. K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) find
  23. I. Sutskever, J. Martens, and G. Hinton. Generating text with recurrent neural networks. In ICML’11, page 1017–1024, 2011. find
  24. Y. Tian, J. Ren, M. Chai, K. Olszewski, X. Peng, D. N. Metaxas, and S. Tulyakov. A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104.15069, 2021. — [arXiv:2104.15069](https://arxiv.org/abs/2104.15069) find
  25. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015. find
  26. S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535, 2018. find
  27. T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. — [arXiv:1812.01717](https://arxiv.org/abs/1812.01717) find
  28. A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309–6318, 2017. find
  29. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017.
  30. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016. find
  31. X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581–4591, 2019. find
  32. Y. Wang, M. Long, J. Wang, Z. Gao, and P. S. Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. Advances in neural information processing systems, 30, 2017. find
  33. D. Weissenborn, O. Täckström, and J. Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv:1906.02634, 2019. — [arXiv:1906.02634](https://arxiv.org/abs/1906.02634) find
  34. C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. — [arXiv:2104.14806](https://arxiv.org/abs/2104.14806) find
  35. C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan. N\\" uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint arXiv:2111.12417, 2021. — [arXiv:2111.12417](https://arxiv.org/abs/2111.12417) find
  36. W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021. — [arXiv:2104.10157](https://arxiv.org/abs/2104.10157) find
  37. S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint arXiv:2202.10571, 2022. — [arXiv:2202.10571](https://arxiv.org/abs/2202.10571) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hong-2022-cogvideo-large-scale-pretraining/