Elamur Research⌕
☰
home / papers / 2025 / SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning

SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning

Paper · 2025 · Yicheng Ji, Jun Zhang, Heming Xia et al.

Summary

Video LLMs turn sampled frames into very long sequences of visual tokens. This makes decoding slow and memory-bound. Pruning tokens at prefill saves compute but discards visual detail, which can lower answer quality. SpecVLM keeps output lossless by applying speculative decoding instead. A small Vid-LLM drafts tokens, and the large target model verifies them in parallel. The authors observe that with long videos, the draft model's slowdown comes mostly from its growing KV cache rather than its parameter count. They also find that its drafting accuracy barely changes when many video tokens are removed. SpecVLM therefore prunes only the draft model's video context, in two steps. First, it keeps tokens that the verifier's attention marks as informative. Second, it thins the remaining low-attention tokens evenly across space. The method needs no training. Tested on four video understanding benchmarks, it removes up to 90% of the draft's video tokens and gives up to 2.68× faster decoding with unchanged output. This makes it a practical route to faster long-video inference.

Key points

  • SpecVLM is presented as the first training-free speculative decoding framework built specifically for video LLMs. Code is released at github.com/zju-jiyicheng/SpecVLM.
  • A preliminary study on VideoDetailCaption with LLaVA-OneVision showed how drafting reacts to random pruning:
    • Up to about 50% pruning, average accept length stays stable and sometimes improves.
    • Above that level, accept length drops noticeably.
    • Removing all video tokens hurts the most.
  • The pruning works in two stages. Stage I keeps highly attended tokens using Top-P retention guided by verifier attention. Stage II reduces the low-attention remainder in a spatially uniform way.
  • Evaluation covered VideoDetailCaption, MVBench, MVLU and LongVideoBench. Pruning over 90% of the draft's video tokens still preserved close to 90% of speculation accuracy.
  • Reported lossless speedups reach 2.68× for LLaVA-OneVision-72B and 2.11× for Qwen2.5-VL-32B. Gains are largest in memory-bound long-video settings.
  • An off-the-shelf Vid-LLM serves as the draft model without fine-tuning, which the authors note limits the maximum achievable acceleration.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China57 works · 32,843 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yicheng Ji, Jun Zhang, Heming Xia, Jinpeng Chen, Lidan Shou, Gang Chen and Huan Li. 2025. SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.366

@inproceedings{ji2025specvlm,
  title = {{SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning}},
  author = {Yicheng Ji, Jun Zhang, Heming Xia, Jinpeng Chen, Lidan Shou, Gang Chen and Huan Li},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.366},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.366}
}

References

53 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Imtiaz Ahmed, Sadman Islam, Partha Protim Datta, Imran Kabir, Naseef Ur Rahman Chowdhury, and Ahshanul Haque. 2025. Qwen 2.5: A comprehensive review of the leading resource-efficient llm with potentioal to surpass all competitors. Authorea Preprints. find
  2. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. 4 https://huggingface.co/datasets/MLVU/MVLU (CC-BY-NC-SA-4.0 license) 5 https://huggingface.co/datasets/OpenGVLab/ MVBench (CC-BY-4.0 license) 6 http://github.com/longvideobench/ LongVideoBench (CC-BY-NC… ↗
  3. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, pages 5209–5235. PMLR.
  4. Jian Chen, Vashisth Tiwari, Ranajoy Sadhukhan, Zhuoming Chen, Jinyuan Shi, Ian En-Hsu Yen, and Beidi Chen. 2025. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. The Thirteenth International Conference on Learning Representations. find
  5. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024a. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer.
  6. Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, and 1 others. 2024b. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188. find
  7. Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, and 1 others. 2024. Glide with a cape: a low-hassle method to accelerate speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, pages 11704–11… find
  8. Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098–89124. find
  9. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. 2025a. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. Preprint, arXiv:2407.11550. — [arXiv:2407.11550](https://arxiv.org/abs/2407.11550) find
  10. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. 2025b. Identify critical kv cache in llm inference from an output perturbation perspective. Preprint, arXiv:2502.03805. — [arXiv:2502.03805](https://arxiv.org/abs/2502.03805) find
  11. Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. 2024. On speculative decoding for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8285–8289. find
  12. Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. 2023. Longcoder: A long-range pre-trained language model for code completion. In International Conference on Machine Learning, pages 12098–12107. PMLR. find
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  14. Xiaohu Huang, Hao Zhou, and Kai Han. 2025. Prunevid: Visual token pruning for efficient video large language models. In ACL. find
  15. Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. 2023. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36:39236–39256. find
  16. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR. find
  17. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
  18. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024b. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206.
  19. Mingxiao Li, Na Su, Fang Qu, Zhizhou Zhong, Ziyang Chen, Yuan Li, Zhaopeng Tu, and Xiaolong Li. 2025a. Vista: Enhancing vision-text alignment in mllms via cross-modal mutual information maximization. Preprint, arXiv:2505.10917. — [arXiv:2505.10917](https://arxiv.org/abs/2505.10917) find
  20. Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and 1 others. 2025b. Mapsparse: Accelerating pre-filling for long-context visual language models via modality-aware permutation sparse attention. In ICLR 2025 Work… find
  21. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024c. Eagle-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432.
  22. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024d. Eagle: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, pages 28935–28948. PMLR.
  23. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971–5984.
  24. Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. 2024. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803. — [arXiv:2411.10803](https://arxiv.org/abs/2411.10803) find
  25. Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. 2025. Video compression commander: Plug-and-play inference acceleration for video large language models. arXiv preprint arXiv:2505.14454.
  26. LMMs-Lab. 2024. Video detail caption. Accessed: 2024-11. find
  27. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, and 1 others. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In ASPLOS, pages 932–949. find
  28. Professor David Patterson. 2005. Latency lags bandwidth. In Proceedings of the 2005 International Conference on Computer Design, pages 3–6. find
  29. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388. — [arXiv:2403.15388](https://arxiv.org/abs/2403.15388) find
  30. Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang, Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. 2025. Growing a twig to accelerate large vision-language models. arXiv preprint arXiv:2503.14075. — [arXiv:2503.14075](https://arxiv.org/abs/2503.14075) find
  31. Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, and Guiguang Ding. 2025. Fastvid: Dynamic density pruning for fast video large language models. arXiv preprint arXiv:2503.11187. — [arXiv:2503.11187](https://arxiv.org/abs/2503.11187) find
  32. Mingbo Song, Heming Xia, Jun Zhang, Chak Tou Leong, Qiancheng Xu, Wenjie Li, and Sujian Li. 2025. KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization. Preprint, arXiv:2505.16162.
  33. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. COLM. find
  34. Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. 2025. Adaptive keyframe sampling for long video understanding. In CVPR. find
  35. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. 2025. Dycoke: Dynamic compression of tokens for fast video large language models. In CVPR. find
  36. Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, and Panpan Xu. 2025. VL-cache: Sparsity and modality-aware KV cache compression for vision-language model inference acceleration. In The Thirteenth International Conference on Learning Representations. find
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  38. Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. 2025. Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502.11494.
  39. Jun Zhang, Jue Wang, Huan Li, Zhongle Xie, Ke Chen, and Lidan Shou. 2025c. CHASe: Client Heterogeneity-Aware Data Selection for Effective Federated Active Learning . IEEE Transactions on Knowledge & Data Engineering, 37(06):3088–3102. find
  40. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857. find
  41. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and 1 others. 2024b. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417. find
  42. Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  43. Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, and Yang You. 2025. A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms. CVPR. find
  44. Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. 2025. SWIFT: On-the-fly self-speculative decoding for LLM inference acceleration. In The Thirteenth International Conference on Learning Representations. find
  45. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2025a. Mlvu: A comprehensive benchmark for multi-task long video understanding. CVPR. find
  46. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. find
  47. Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, and Lidan Shou. 2025b. FloE: On-the-Fly MoE Inference on Memory-constrained GPU. Forty-second International Conference on Machine Learning. find
  48. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2025. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. CVPR. find
  49. Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An. 2025a. Longspec: Long-context speculative decoding with efficient drafting and verification. arXiv preprint arXiv:2502.17421. — [arXiv:2502.17421](https://arxiv.org/abs/2502.17421) find
  50. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025b. Visionzip: Longer is better but not necessary in vision language models. CVPR. find
  51. Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2024a. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page…
  52. Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Qin Xie, Guiming Xie, and Xuejian Gong. 2025a. HMI: hierarchical knowledge management for efficient multi-tenant inference in pretrained language models. The VLDB Journal, 34(4):43. find
  53. Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Yang You, Guiming Xie, Xuejian Gong, and Kunlong Zhou. 2025b. Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models. The Thirteenth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ji-2025-specvlm-enhancing-speculative-decoding-video/