Elamur Research⌕
☰
home / papers / 2026 / Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs

Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs

Paper · 2026 · Libo Zhang, Zhaoning Zhang, Hongwanyang et al.

Summary

Speculative decoding speeds up vision-language models without loss, but it breaks down for Video LLMs. Long videos produce tens of thousands of visual tokens, and a small draft model cannot handle them well. Its KV cache grows, its short context window is exceeded, and its attention gets spread thin. The authors find that extra visual tokens can actually reduce draft accuracy. They also observe that the target model's deep text hidden states already contain the key visual meaning, which they call "visual semantic internalization."

Their framework, Sparrow, builds on this finding:

  • HSR-VATA: the draft model reuses the target model's text hidden states and attends only to text-anchored windows, so it never processes raw visual tokens.
  • IVSB: during training, the draft model receives the target's intermediate-layer visual states, which carry richer semantics, instead of raw visual features.
  • Multi-token prediction: this reduces the mismatch between training and inference.

With 25k visual tokens, Sparrow reaches an average 2.82× decoding speedup and stays robust as sequences grow, which makes speculative decoding practical for long-video tasks.

Key points

  • Earlier multimodal drafters degrade on long video: from 0.4k to 25k tokens, MSD's average accepted length drops by about 63%, and ViSpec's drops by about 30%.
  • For Qwen2.5-VL-7B, video inputs can reach roughly 25k visual tokens, which is over 99% of the sequence.
  • At 0.5k tokens, Sparrow achieves a peak 3.30× speedup with accepted length 4.50 (ViSpec 3.22×, MSD 2.96×). At 25k tokens, it keeps accepted length 4.35, while MSD slows to 0.42×.
  • In the ablation, removing VATA drops accepted length at 25k tokens to 1.21; the full model reaches 4.37.
  • The main limitation is that prefill cost still grows with video length and caps end-to-end speedup. Code is available at github.com/ddInference/Sparrow.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of Defense TechnologyChangsha · China10 works · 931 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Libo Zhang, Zhaoning Zhang, Hongwanyang, Peng Qiao and Dongsheng Li. 2026. Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.450

@inproceedings{zhang2026sparrow,
  title = {{Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs}},
  author = {Libo Zhang, Zhaoning Zhang, Hongwanyang, Peng Qiao and Dongsheng Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.450},
  url = {https://doi.org/10.18653/v1/2026.acl-long.450}
}

References

55 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Imtiaz Ahmed, Sadman Islam, Partha Protim Datta, Imran Kabir, Naseef Ur Rahman Chowdhury, and Ahshanul Haque. 2025. Qwen 2.5: A comprehensive review of the leading resource-efficient llm with potentioal to surpass all competitors. Authorea Preprints. find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  3. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. find
  4. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer.
  5. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
  6. Yingqi Fan, Anhao Zhao, Jinlan Fu, Junlong Tong, Hui Su, Yijie Pan, Wei Zhang, and Xiaoyu Shen. 2025. Visipruner: Decoding discontinuous cross-modal dynamics for efficient multimodal llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18896–18913. find
  7. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2025a. Mme: A comprehensive evaluation benchmark for multimodal large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Dat…
  8. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2025b. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR.
  9. Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057.
  10. Mugilan Ganesan, Shane Segal, Ankur Aggarwal, Nish Sinnadurai, Sean Lie, and Vithursan Thangarasa. 2025. Massv: Multimodal adaptation and self-data distillation for speculative decoding of vision-language models. arXiv preprint arXiv:2505.10526. find
  11. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Yakun Sophia Shao. 2025. Speed: Speculative pipelined execution for efficient decoding. In Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques, pages 19–32. Springer. find
  12. Yunhai Hu, Zining Liu, Zhenyuan Dong, Tianfan Peng, Bradley McDanel, and Sai Qian Zhang. 2025a. Speculative decoding and beyond: An in-depth survey of techniques. arXiv preprint arXiv:2502.19732. — [arXiv:2502.19732](https://arxiv.org/abs/2502.19732) find
  13. Yunhai Hu, Tianhua Xia, Zining Liu, Rahul Raman, Xingyu Liu, Bo Bao, Eric Sather, Vithursan Thangarasa, and Sai Qian Zhang. 2025b. Dream: Drafting with refined target features and entropy-adaptive cross-attention fusion for multimodal speculative decoding. arXiv preprint arXiv:2505.19201. — [arXiv:2505.19201](https://arxiv.org/abs/2505.19201) find
  14. Mingxiao Huo, Jiayi Zhang, Hewei Wang, Jinfeng Xu, Zheyu Chen, Huilin Tai, and Yijun Chen. 2025. Spec-llava: Accelerating vision-language models with dynamic tree-based speculative decoding. arXiv preprint arXiv:2509.11961. — [arXiv:2509.11961](https://arxiv.org/abs/2509.11961) find
  15. Yicheng Ji, Jun Zhang, Jinpeng Chen, Cong Wang, Lidan Shou, Gang Chen, and Huan Li. 2026. See the forest for the trees: Loosely speculative decoding via visual-semantic guidance for efficient inference of video llms. arXiv preprint arXiv:2604.05650. find
  16. Yicheng Ji, Jun Zhang, Heming Xia, Jinpeng Chen, Lidan Shou, Gang Chen, and Huan Li. 2025. Specvlm: Enhancing speculative decoding of video llms via verifier-guided token pruning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7216–7230.
  17. Jialiang Kang, Han Shu, Wenshuo Li, Yingjie Zhai, and Xinghao Chen. 2025. Vispec: Accelerating vision-language models with vision-aware speculative decoding. arXiv preprint arXiv:2509.15235. — [arXiv:2509.15235](https://arxiv.org/abs/2509.15235) find
  18. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In European conference on computer vision, pages 235–251. Springer.
  19. Minjae Lee, Wonjun Kang, Minghao Yan, Christian Classen, Hyung Il Koo, and Kangwook Lee. 2024. In-batch ensemble drafting: Toward fast and robust speculative decoding for multimodal language models. OpenReview.net. find
  20. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR. find
  21. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
  22. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024b. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895. find
  23. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024c. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206.
  24. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024d. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.
  25. Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. 2024b. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint arXiv:2404.18911. — [arXiv:2404.18911](https://arxiv.org/abs/2404.18911) find
  26. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024c. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306.
  27. Jiahao Liu, Qifan Wang, Jingang Wang, and Xunliang Cai. 2024d. Speculative decoding via early-exiting for faster llm inference with thompson sampling control mechanism. arXiv preprint arXiv:2406.03853. find
  28. Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. 2024e. Parallel speculative decoding with adaptive draft length. arXiv e-prints, pages arXiv–2408. find
  29. Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. 2024f. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803. — [arXiv:2411.10803](https://arxiv.org/abs/2411.10803) find
  30. LMMs-Lab. 2024a. Video detail caption. Accessed: 2024-11. find
  31. LMMs-Lab. 2024b. Videochatgpt. Accessed: 2024-04. find
  32. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. find
  33. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12585– 12602.
  34. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv:2306.05424.
  35. Luxi Lin, Zhihang Lin, Zhanpeng Zeng, and Rongrong Ji. 2025. Speculative decoding reimagined for multimodal large language models. arXiv preprint arXiv:2505.14260. — [arXiv:2505.14260](https://arxiv.org/abs/2505.14260) find
  36. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pages 2263– 2279.
  37. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. find
  38. Giovanni Monea, Armand Joulin, and Edouard Grave. 2023. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13581. — [arXiv:2311.13581](https://arxiv.org/abs/2311.13581) find
  39. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912. find
  40. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. 2025. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18992–19001. find
  41. Jikai Wang, Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, and Min Zhang. 2025a. Opt-tree: Speculative decoding with adaptive draft tree structure. Transactions of the Association for Computational Linguistics, 13:188–199.
  42. Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, and 1 others. 2025b. Internvideo2. 5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386. — [arXiv:2501.12386](https://arxiv.org/abs/2501.12386) find
  43. Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, and 1 others. 2024. Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint arXiv:2407.08223. — [arXiv:2407.08223](https://arxiv.org/abs/2407.08223) find
  44. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857. find
  45. Zhinan Xie, Peisong Wang, Shuang Qiu, and Jian Cheng. 2025. Hivis: Hiding visual tokens from the drafter for speculative decoding in vision-language models. arXiv preprint arXiv:2509.23928. cient large vision-language models. arXiv preprint arXiv:2505.22654. — [arXiv:2509.23928](https://arxiv.org/abs/2509.23928) find
  46. Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858.
  47. Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, and Huan Li. 2026. Efficient inference for large vision-language models: Bottlenecks, techniques, and prospects. arXiv preprint arXiv:2604.05546. find
  48. Libo Zhang, Zhaoning Zhang, Rui Li, Zhiliang Tian, Songzhu Mei, Dongsheng Li, and 1 others. 2025b. Dovetail: A cpu/gpu heterogeneous speculative decoding for llm inference. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17393–17406. find
  49. Xuan Zhang, Cunxiao Du, Sicheng Yu, Jiawei Wu, Fengzhuo Zhang, Wei Gao, and Qian Liu. 2025c. Sparse-to-dense: A free lunch for lossless acceleration of video understanding in llms. arXiv preprint arXiv:2505.19155. find
  50. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. find
  51. Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. 2025d. Cross-modal information flow in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19781–19791. find
  52. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2024. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247. — [arXiv:2410.17247](https://arxiv.org/abs/2410.17247) find
  53. Hao Yin, Guangzong Si, and Zilei Wang. 2025. Lifting the veil on visual information flow in mllms: Unlocking pathways to faster inference. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9382–9391. find
  54. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.
  55. Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang, Yaqi Xie, Katia Sycara, Haitao Mi, and Dong Yu. 2025a. Vscan: Rethinking visual token reduction for effi- find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-sparrow-text-anchored-window-attention/