Elamur Research⌕
☰
home / papers / 2025 / Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models

Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models

Paper · 2025 · Xuyang Liu, Yiyu Wang, Junpeng Ma et al.

Summary

Video large language models must process very long sequences of visual tokens, because every sampled frame adds hundreds of them. This makes inference slow and memory-hungry. The authors review existing training-free token pruning methods and find two problems. First, these methods compress all frames equally, even though some frames carry far more distinctive content than others. Second, many depend on a ViT [CLS] token, which SigLIP-based encoders lack, or on explicit attention maps, which rules out efficient kernels such as FlashAttention. From this analysis they derive three design principles: model adaptability, frame uniqueness and operator compatibility. Their method, VidCom², is a plug-and-play compressor that works in two stages. First, Frame Compression Adjustment scores how unique each frame is relative to the whole video and gives more distinctive frames a larger token budget. Second, Adaptive Token Compression keeps the tokens that differ most from both their own frame and the video as a whole. The method keeps nearly full accuracy at a fraction of the token count and runs alongside efficient attention operators, so it is practical for long videos.

Key points

  • Evaluated on LLaVA-OV (LLaVA-OneVision) and LLaVA-Video, using benchmarks including MVBench, MLVU and VideoMME.
  • With 25% of the visual tokens, VidCom² retains 99.6% of LLaVA-OV's original performance and cuts LLM generation latency by 70.8%.
  • At a 15% token budget, it beats the next-best method by 3.9% on LLaVA-OV and 2.2% on LLaVA-Video. DyCoke, by comparison, cannot go below 25% retention.
  • Ablations show that:
    • averaging token uniqueness per frame works better than taking the maximum;
    • a global window (32 frames) works best for long videos, so it is the default.
  • Adding the Frame Compression Adjustment stage to intra-LLM methods such as FastV and SparseVLM also improves their results. Models at 72B scale were not tested; code is available at github.com/xuyang-liu16/VidCom2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Sichuan UniversityChengdu · China14 works · 1,293 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Xuyang Liu, Yiyu Wang, Junpeng Ma and Linfeng Zhang. 2025. Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.98

@inproceedings{liu2025video,
  title = {{Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models}},
  author = {Xuyang Liu, Yiyu Wang, Junpeng Ma and Linfeng Zhang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.98},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.98}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token merging: Your ViT but faster. In Proceedings of the International Conference on Learning Representations. find
  2. Junjie Chen, Xuyang Liu, Zichen Wen, Yiyu Wang, Siteng Huang, and Honggang Chen. 2025. Variation-aware vision token dropping for faster large vision-language models. arXiv preprint arXiv:2509.01552. — [arXiv:2509.01552](https://arxiv.org/abs/2509.01552) find
  3. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024a. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Proceedings of the European Conference on Computer Vision.
  4. Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, and 1 others. 2024b. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188. find
  5. Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In Proceedings of the International Conference on Learning Representations.
  6. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Proceedings of the Advances in Neural Information Processing Systems. find
  7. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. In Proceedings of the Advances in Neural Information Processing Systems, volume 36, pages 46212– 46244. find
  8. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075.
  9. Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, and 1 others. 2023. Perception test: A diagnostic benchmark for multimodal video models. In Proceedings of the Advances in Neural Information Pro… find
  10. Yuhang Han, Xuyang Liu, Pengxiang Ding, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. 2024. Rethinking token reduction in mllms: Towards a unified paradigm for training-free acceleration. arXiv preprint arXiv:2411.17686. — [arXiv:2411.17686](https://arxiv.org/abs/2411.17686) find
  11. Hui Sun, Shiyin Lu, Huanyu Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Ming Li. 2025. Mdp3: A training-free approach for listwise frame selection in video-llms. arXiv preprint arXiv:2501.02885. — [arXiv:2501.02885](https://arxiv.org/abs/2501.02885) find
  12. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
  13. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. 2025. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. find
  14. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024b. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206.
  15. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing vision-language model’s perception of the w…
  16. Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. 2024c. Token-Packer: Efficient visual projector for multimodal LLM. arXiv preprint arXiv:2407.02392. — [arXiv:2407.02392](https://arxiv.org/abs/2407.02392) find
  17. Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, and 1 others. 2025. Internvideo2. 5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386. — [arXiv:2501.12386](https://arxiv.org/abs/2501.12386) find
  18. Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. 2024. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803. — [arXiv:2411.10803](https://arxiv.org/abs/2411.10803) find
  19. Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, and Honggang Chen. 2025a. Global compression commander: Plug-and-play inference acceleration for high-resolution large vision-language models. arXiv preprint arXiv:2501.05179. — [arXiv:2501.05179](https://arxiv.org/abs/2501.05179) find
  20. Xuyang Liu, Zichen Wen, Shaobo Wang, Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, and 1 others. 2025b. Shifting ai efficiency from model-centric to data-centric compression. arXiv preprint arXiv:2505.19147. — [arXiv:2505.19147](https://arxiv.org/abs/2505.19147) find
  21. Junpeng Ma, Qizhe Zhang, Ming Lu, Zhibin Wang, Qiang Zhou, Jun Song, and Shanghang Zhang. 2025. Mmg-vid: Maximizing marginal gains at segment-level and token-level for efficient video llms. arXiv preprint arXiv:2508.21044. — [arXiv:2508.21044](https://arxiv.org/abs/2508.21044) find
  22. Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. 2025a. Token pruning in multimodal large language models: Are we solving the right problem? In Findings of the Association for Computational Linguistics: ACL 2025, pages 15537–15549. find
  23. Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. 2025b. Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502.11494.
  24. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. In Proceedings of the Advances in Neural Information Processing Systems, volume 37, pages 28828–28857. find
  25. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2025. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision… find
  26. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. find
  27. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11941–11952.
  28. Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 543–553.
  29. Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and 1 others. 2024a. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772.
  30. Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. 2024b. [CLS] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818. — [arXiv:2412.01818](https://arxiv.org/abs/2412.01818) find
  31. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. 2025. SparseVLM: Visual token sparsification for efficient vision-language model inference. In Proceedings of the International Conference on… find
  32. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024c. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. find
  33. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264. find
  34. In the appendix, we provide more benchmark details in Section A, model details in Section B, more baseline details in Section C, sensitivity analysis in Section D, algorithm details in Section E, and more visualization of frame uniqueness quantified by VidCom 2 in Section F. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-video-compression-commander-plug-play/