Elamur Research⌕
☰
home / papers / 2026 / From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration

From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration

Paper · 2026 · Jiaqi Shi, Xulong Zhang, Yuechan Li et al.

Summary

When multimodal LLMs handle high-resolution images, most of the prefill compute goes to visual tokens. Existing speedup techniques, such as visual token pruning and layer skipping, are mostly tuned on Vicuna/Mistral-based models like LLaVA. The authors show that these methods lose 5.7–22.4% of relative performance on a Qwen2.5 backbone. Using LLaVA-NeXT as a control, they trace this loss to how the language backbone processes visual information, not to the vision front-end.

They probe layer-wise hidden states with truncated matrix entropy and find a shared three-stage pattern: modality alignment, global aggregation and visual saturation. From this, they split redundancy into two kinds:

  • Intrinsic Visual Redundancy (IVR): inherited from dense ViT patches and present in every model.
  • Secondary Saturation Redundancy (SSR): appears in deep layers and differs by architecture. Vicuna/Mistral show idle layers, while Qwen shows extreme token sparsity.

Their framework, HalfV, removes IVR with one shared pruning step and then handles SSR adaptively for each backbone, with no extra training or controllers. On Qwen2.5-VL it keeps 96.8% of performance at a 4.1× FLOPs speedup and outperforms prior baselines.

Key points

  • Models evaluated include LLaVA-1.5v (7B/13B), LLaVA-NeXT-7B and Qwen2.5-VL-7B. Benchmarks include POPE, MME, MMBench, SQA, GQA and AI2D. Baselines include FastV, DART, HoloV and ShortV. Code is at github.com/civilizwa/HalfV.
  • Efficiency on POPE:
    • On LLaVA-NeXT-7B, HalfV lowers overall runtime by 38.9% and latency by 46.8%, while TTFT drops 48.5% and accuracy improves by 0.6%.
    • On Qwen2.5-VL-7B, it cuts total time by 48.6% with a 2.3% accuracy drop, which is 1.9% better than DART.
  • Overhead is low:
    • The IVR module takes 7.3% of TTFT on LLaVA-NeXT-7B and 5.8% on Qwen2.5-VL-7B.
    • The SSR module costs under 1% on both.
  • The best point to start SSR reduction is the Stage II→III transition. Starting earlier or later both hurt accuracy.
  • Limitations: HalfV needs access to hidden states, so it cannot be used with closed models such as GPT or Gemini. It was only tested on models with 4B–13B parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksHardware and Architecture · 81 worksEngineering · 133 worksElectrical and Electronic Engineering · 18 works

Cite

Cite as Jiaqi Shi, Xulong Zhang, Yuechan Li, Xiaoyang Qu and Jianzong Wang. 2026. From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.787

@inproceedings{shi2026inheritance,
  title = {{From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration}},
  author = {Jiaqi Shi, Xulong Zhang, Yuechan Li, Xiaoyang Qu and Jianzong Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.787},
  url = {https://doi.org/10.18653/v1/2026.acl-long.787}
}

References

44 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. 2025. Divprune: Diversity-based visual token pruning for large multimodal models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pages 9392– 9401. Computer Visi… find
  2. Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S. Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. 2025. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intellige… find
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. CoRR, abs/2502.13923…
  4. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23rd Annual ACM Symposium on User Interface Softw… find
  5. Richard Cangelosi and Alain Goriely. 2007. Component retention in principal component analysis with application to cdna microarray data. Biology direct, 2:2. find
  6. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024a. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-O…
  7. Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. 2024b. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024… find
  8. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. CoRR, abs/2312.14238.
  9. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans… find
  10. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th In…
  11. Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. 2025. Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. Preprint, arXiv:2412.13180. — [arXiv:2412.13180](https://arxiv.org/abs/2412.13180) find
  12. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. 2025. Mme: A comprehensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394.
  13. Teofilo F. Gonzalez. 1985. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293–306. find
  14. Yixiao He, Haifeng Sun, Pengfei Ren, Jingyu Wang, Huazheng Wang, Qi Qi, Zirui Zhuang, and Jing Wang. 2025. Evaluating and mitigating object hallucination in large vision-language models: Can they still see removed objects? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter… find
  15. Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700–6709. Computer Vision Foundation / IEE…
  16. Nick Jiang, Anish Kachinthaya, Suzanne Petryk, and Yossi Gandelsman. 2025. Interpreting and editing vision-language representations to mitigate hallucinations. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  17. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in…
  18. S. Kullback and R. A. Leibler. 1951. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. find
  19. Kaiyuan Li, Xiaoyue Chen, Chen Gao, Yong Li, and Xinlei Chen. 2025. Balanced token pruning: Accelerating vision language models beyond local optimization. CoRR, abs/2505.22038. — [arXiv:2505.22038](https://arxiv.org/abs/2505.22038) find
  20. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 202…
  21. Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. 2025. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 -March 4, 2025, Philadelphia, PA, USA, pages 53… find
  22. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26286–26296. IEEE.
  23. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llava-next: Improved reasoning, ocr, and world knowledge. find
  24. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
  25. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024c. Mmbench: Is your multi-modal model an all-around player? In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-Octo…
  26. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems 35: Annual Conference on… find
  27. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. CoRR, abs/2403.15388. — [arXiv:2403.15388](https://arxiv.org/abs/2403.15388) find
  28. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8317– 8326. Computer Visio… find
  29. Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  30. Jintao Tong, Wenwei Jin, Pengda Qin, Anqi Li, Yixiong Zou, Yuhong Li, Yuhua Li, and Ruixuan Li. 2025. Flowcut: Rethinking redundancy via information flow for efficient vision-language models. CoRR, abs/2505.19536. — [arXiv:2505.19536](https://arxiv.org/abs/2505.19536) find
  31. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605. find
  32. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  33. Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. 2025. Stop looking for important tokens in multimodal language models: Duplication matters more. CoRR, abs/2502.11494. token pruning for multi-modal large language models. In AAAI-25, Sponsor…
  34. Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, and Le Sun. 2025. Shortv: Efficient multimodal large language models by freezing visual tokens in ineffective layers. CoRR, abs/2504.00502. — [arXiv:2504.00502](https://arxiv.org/abs/2504.00502) find
  35. Qiong Wu, Wenhao Lin, Weihao Ye, Yiyi Zhou, Xiaoshuai Sun, and Rongrong Ji. 2024. Accelerating multimodal large language models via dynamic visual-token exit and the empirical findings. CoRR, abs/2411.19628. — [arXiv:2411.19628](https://arxiv.org/abs/2411.19628) find
  36. Yifan Zhang, Zhiquan Tan, Jingqin Yang, Weiran Huang, and Yang Yuan. 2024a. Matrix information theory for self-supervised learning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. find
  37. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. 2024. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. CoRR, abs/2410.17247. — [arXiv:2410.17247](https://arxiv.org/abs/2410.17247) find
  38. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A. Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. 2024b. Sparsevlm: Visual token sparsification for efficient vision-language model inference. CoRR, abs/2410.04417. find
  39. Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Min Yang, Lingpeng Kong, and Ngai Wong. 2025. UNComp: Can matrix entropy uncover sparsity? — a compressor design from an uncertainty-aware perspective. In Proceedings of the 2025 Con… find
  40. Xin Zou, Di Lu, Yizhou Wang, Yibo Yan, Yuanhuiyi Lyu, Xu Zheng, Linfeng Zhang, and Xuming Hu. 2025. Don’t just chase "highlighted tokens" in mllms: Revisiting visual holistic context retention. CoRR, abs/2510.02912. — [arXiv:2510.02912](https://arxiv.org/abs/2510.02912) find
  41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  42. Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Pon-nuswamy Sadayappan, Xia Hu, and Bo Yuan. 2025b. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In IEEE/CVF Conference on Computer Vi… find
  43. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025c. Visionzip: Longer is better but not necessary in vision language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pages 19792–1… find
  44. Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. 2025. Fit and prune: Fast and training-free visual find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-inheritance-saturation-disentangling-evolution-visual/