Elamur Research⌕
☰
home / papers / 2026 / Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

Paper · 2026 · Yucheng Zhou and Jianbing Shen

Summary

Autoregressive video generators predict long sequences of discrete VQ-VAE tokens, which makes training slow and costly. The authors first test a simple shortcut: training on fewer frames and extending videos step by step at inference. This trains faster, but inference becomes slower and errors compound, so frames lose coherence. They show this both theoretically and through PSNR and optical-flow measurements.

Their fix, Local Optimization (Local Opt.), computes the loss only on tokens inside randomly placed, overlapping windows of frame blocks, while earlier tokens act as context. Windows are sampled more often at the start of the video, because poor early frames degrade the later ones. A second component, Representation Continuity (ReCo), is motivated by Lipschitz continuity. It adds a loss term that discourages sudden jumps in hidden representations between steps.

Experiments cover class-conditional benchmarks (FFS, SKY, UCF, Taichi) and text-to-video generation (trained on Vimeo, evaluated zero-shot on MSR-VTT). The approach matches or beats the baselines at about half the training cost. Cheaper training of this kind is relevant to building unified multimodal LLMs.

Key points

  • Fewer-Frames baseline is a poor shortcut. It trains roughly 2.5× faster, but FVD collapses compared with the Baseline:

    • FFS: 229.32 vs 73.65
    • SKY: 292.41 vs 89.09

  • Training speed. Local Opt. with ReCo trains about 2× faster than the Baseline, keeps baseline-level inference time, and matches the Baseline in PSNR-based consistency.

  • Class-conditional results (FVD, lower is better). The 770M ReCo⋆ beats Baseline⋆ on all four datasets:

    DatasetReCo⋆Baseline⋆
    FFS42.546.1
    SKY58.862.7
    UCF251.4254.5
    Taichi98.3105.5

  • Compatibility with a stronger tokenizer. Combined with the LARP tokenizer, ReCo♠ reaches FVD 46.2 on FFS and 56.1 on UCF, improving on LARP itself (62.6 and 57.0).

  • Text-to-video (Loong 7B, MSR-VTT zero-shot).

    • At equal training cost, ReCo raises CLIP Score from 0.2903 to 0.3056 and lowers FVD from 274 to 212.
    • ReCo* gets 0.2911 CLIP and 267 FVD using only 50% of the training cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MacauMacao · China43 works · 2,368 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Yucheng Zhou and Jianbing Shen. 2026. Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1501

@inproceedings{zhou2026accelerating,
  title = {{Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity}},
  author = {Yucheng Zhou and Jianbing Shen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1501},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1501}
}

References

48 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. 2023. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, abs/2311.15127. — [arXiv:2311.15127](https://arxiv.org/abs/2311.15127) find
  2. Guanjie Chen, Xinyu Zhao, Yucheng Zhou, Tianlong Chen, and Yu Cheng. 2024. Accelerating vision diffusion transformers with skip branches. CoRR, abs/2411.17616. — [arXiv:2411.17616](https://arxiv.org/abs/2411.17616) find
  3. Guanjie Chen, Xinyu Zhao, Yucheng Zhou, Xiaoye Qu, Tianlong Chen, and Yu Cheng. 2025. Towards stabilized and efficient diffusion transformers through long-skip-connections with spectral constraints. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17708–17718. find
  4. Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. 2024. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14169. — [arXiv:2412.14169](https://arxiv.org/abs/2412.14169) find
  5. Gunnar Farnebäck. 2003. Two-frame motion estimation based on polynomial expansion. In Image Analysis, 13th Scandinavian Conference, SCIA 2003, Halmstad, Sweden, June 29 - July 2, 2003, Proceedings, volume 2749 of Lecture Notes in Computer Science, pages 363–370. Springer. find
  6. Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. 2022. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221. — [arXiv:2211.13221](https://arxiv.org/abs/2211.13221) find
  7. Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2023. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  8. Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. 2024a. Autoregressive image generation without vector quantization. CoRR, abs/2406.11838. find
  9. Wenhao Li, Yichao Cao, Xiu Su, Xi Lin, Shan You, Mingkai Zheng, Yi Chen, and Chang Xu. 2024b. Training-free long video generation with chain of diffusion model experts. arXiv preprint arXiv:2408.13423. — [arXiv:2408.13423](https://arxiv.org/abs/2408.13423) find
  10. Yizhuo Li, Yuying Ge, Yixiao Ge, Ping Luo, and Ying Shan. 2024c. Dicode: Diffusion-compressed deep tokens for autoregressive video generation with language models. CoRR, abs/2412.04446. find
  11. Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. 2024. Latte: Latent diffusion transformer for video generation. CoRR, abs/2401.03048. — [arXiv:2401.03048](https://arxiv.org/abs/2401.03048) find
  12. Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, and 69 others. 2024. Movie gen: A cast of media founda… — [arXiv:2410.13720](https://arxiv.org/abs/2410.13720) find
  13. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  14. Shuhuai Ren, Shuming Ma, Xu Sun, and Furu Wei. 2025. Next block prediction: Video generation via semi-autoregressive modeling. arXiv preprint arXiv:2502.07737. — [arXiv:2502.07737](https://arxiv.org/abs/2502.07737) find
  15. Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2018. Faceforensics: A large-scale video dataset for forgery detection in human faces. CoRR, abs/1803.09179. — [arXiv:1803.09179](https://arxiv.org/abs/1803.09179) find
  16. Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. First order motion model for image animation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Van… find
  17. Han Song, Yucheng Zhou, Jianbing Shen, and Yu Cheng. 2026. From broad exploration to stable synthesis: Entropy-guided optimization for autoregressive image generation. In The Fourteenth International Conference on Learning Representations. find
  18. K Soomro. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) find
  19. Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. 2024a. Autoregressive model beats diffusion: Llama for scalable image generation. CoRR, abs/2406.06525. — [arXiv:2406.06525](https://arxiv.org/abs/2406.06525) find
  20. Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng, Li Dong, Shaohan Huang, Jianyong Wang, and Furu Wei. 2024b. Multimodal latent language modeling with next-token diffusion. CoRR, abs/2412.08635. — [arXiv:2412.08635](https://arxiv.org/abs/2412.08635) find
  21. Chameleon Team. 2024. Chameleon: modal early-fusion foundation models. abs/2405.09818.
  22. Mixed-CoRR, Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. 2018. Mocogan: Decomposing motion and content for video generation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1526–1535. Computer Vision Found… find
  23. Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. 2018. Towards accurate generative models of video: A new metric & challenges. CoRR, abs/1812.01717. — [arXiv:1812.01717](https://arxiv.org/abs/1812.01717) find
  24. Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. 2023. Phenaki: Variable length video generation from open domain textual descriptions. In The Eleventh International Conference on Learning… find
  25. Chenglin Wang, Yucheng Zhou, Shawn Chen, Tao Wang, and Kai Zhang. 2026. Ladr: Locality-aware dynamic rescue for efficient text-to-image generation with diffusion large language models. arXiv preprint arXiv:2603.13450.
  26. Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. 2025. LARP: Tokenizing videos with a learned autoregressive generative prior. In The Thirteenth International Conference on Learning Representations. find
  27. Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. 2023a. Modelscope text-to-video technical report. CoRR, abs/2308.06571. — [arXiv:2308.06571](https://arxiv.org/abs/2308.06571) find
  28. Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. 2024a. Omnitokenizer: A joint image-video tokenizer for visual generation. CoRR, abs/2406.09399. — [arXiv:2406.09399](https://arxiv.org/abs/2406.09399) find
  29. Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. 2023b. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. CoRR, abs/2305.10874. — [arXiv:2305.10874](https://arxiv.org/abs/2305.10874) find
  30. Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. 2024c. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757. — [arXiv:2410.02757](https://arxiv.org/abs/2410.02757) find
  31. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600–612. find
  32. Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. 2018. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2364–2373. Co… find
  33. Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. 2021. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157. — [arXiv:2104.10157](https://arxiv.org/abs/2104.10157) find
  34. Hongji Yang, Wencheng Han, Yucheng Zhou, and Jianbing Shen. 2025a. Dc-controlnet: Decoupling inter-and intra-element conditions in image generation with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19065–19074. find
  35. Hongji Yang, Yucheng Zhou, Wencheng Han, and Jianbing Shen. 2025b. Self-rewarding large vision-language models for optimizing prompts in text-to-image generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7332– 7349. find
  36. Hongji Yang, Yucheng Zhou, Wencheng Han, Runzhou Tao, Zhongying Qiu, Jianfei Yang, and Jianbing Shen. 2025c. Hicogen: Hierarchical compositional text-to-image generation in diffusion models via reinforcement learning. arXiv preprint arXiv:2511.19965. — [arXiv:2511.19965](https://arxiv.org/abs/2511.19965) find
  37. Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. 2023. Video probabilistic diffusion models in projected latent space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 18456–18466. IEEE. find
  38. Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. 2023c. Vide-olcm: Video latent consistency model. CoRR, abs/2312.09109. — [arXiv:2312.09109](https://arxiv.org/abs/2312.09109) find
  39. Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. 2022. Generating videos with dynamics-aware implicit generative adversarial networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open-Review.n… find
  40. Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, and 6 others. 2024b. Emu3: Next-token prediction is all you need. CoRR,… — [arXiv:2409.18869](https://arxiv.org/abs/2409.18869) find
  41. Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. 2024. Instructvideo: Instructing video diffusion models with human feedback. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA,… find
  42. David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. 2023. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. CoRR, abs/2309.15818. find
  43. Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. 2024. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404. — [arXiv:2412.20404](https://arxiv.org/abs/2412.20404) find
  44. Yucheng Zhou, Xiubo Geng, Tao Shen, Chongyang Tao, Guodong Long, Jian-Guang Lou, and Jianbing Shen. 2023. Thread of thought unraveling chaotic contexts. arXiv preprint arXiv:2311.08734. — [arXiv:2311.08734](https://arxiv.org/abs/2311.08734) find
  45. Yucheng Zhou, Hao Li, and Jianbing Shen. 2026. Condition errors refinement in autoregressive image generation with diffusion loss. In The Fourteenth International Conference on Learning Representations. find
  46. Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. 2024a. Visual in-context learning for large vision-language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15890–15902. find
  47. Yucheng Zhou, Jianbing Shen, and Yu Cheng. 2025. Weak to strong generalization for large language models with multi-capabilities. In The Thirteenth International Conference on Learning Representations. find
  48. Yucheng Zhou, Jihai Zhang, Guanjie Chen, Jianbing Shen, and Yu Cheng. 2024b. Less is more: Vision representation compression for efficient video generation with large language models. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-accelerating-training-autoregressive-video-generation/