Elamur Research⌕
☰
home / papers / 2026 / Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale

Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale

Paper · 2026 · Jianhong Tu, Nicholas Crispino, Kyle Montgomery et al.

Summary

Visual text compression renders long text as images so a vision-language model can read it with fewer context tokens than the raw text would need. It is unclear how reliably VLMs handle such dense, text-only images. FICO is a benchmark that measures this. It starts from LongBench v2 documents of 8k–64k tokens, renders them with ReportLab, and creates seven variants along two axes. The density axis changes font size and DPI. The fidelity axis applies binarization, JPEG compression, and upscaling blur. Models are tested on three tasks, from low-level to high-level: OCR (scored by character error rate), multi-key NIAH retrieval, and long-context VQA. The authors evaluated 13 general-purpose VLMs and 3 OCR-specialized models. Accuracy falls sharply when text gets denser or resolution drops. Skill on one task predicts little about the others. VQA holds up relatively well, which suggests exact characters are lost before overall meaning. The results help practitioners choose rendering settings that trade token savings against reliability.

Key points

  • Dataset size:
    • OCR: 76 documents, 5,312 single-image instances.
    • NIAH: 185 documents, 3,885 instances.
    • VQA: 185 instances.
    • NIAH and VQA together use 19,672 rendered images.
    • Default rendering is NotoSans 11pt at 96 DPI on A4 pages.
  • Architecture trade-off: Dynamic native-resolution models such as Qwen2.5-VL score highest on OCR and NIAH with high-resolution input. They degrade badly at low DPI or with dense text. Tile-based and fixed-resolution models peak lower but stay more stable.
  • Weak cross-task transfer: Kimi-VL-A3B and Glyph perform well on NIAH and VQA but have high OCR error.
  • Reasoning variants: "Thinking" versions raise OCR error for all three models tested (e.g., Qwen3-VL-8B by about 10 points). They consistently improve text-only VQA.
  • Compute cost: For Qwen3-VL-8B at 64K context, prefill cost is about 1,018 TFLOPS (Original rendering) and 255 TFLOPS (LowDPI). The text-only baseline costs about 2,097 TFLOPS. HighDPI costs far more, around 8,787 TFLOPS. Code and data are at github.com/wang-research-lab/fico-bench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🎓University of California, Santa CruzSanta Cruz · United States36 works · 3,515 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 worksEfficient models & compression · 242 works

Cite

Cite as Jianhong Tu, Nicholas Crispino, Kyle Montgomery, Chenguang Wang and Dawn Song. 2026. Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1758

@inproceedings{tu2026fico,
  title = {{Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale}},
  author = {Jianhong Tu, Nicholas Crispino, Kyle Montgomery, Chenguang Wang and Dawn Song},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1758},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1758}
}

References

51 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025a. Qwen2.5-vl technical report. CoRR, abs/2502.1392…
  2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longbench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… find
  3. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025b. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Associ…
  4. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. find
  5. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXI, Lecture Notes in Computer Science, pages 19–35. Springer.
  6. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. CoRR, abs/2503.09567.
  7. Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, Yushi Bai, Jie Tang, Hongning Wang, and Minlie Huang. 2025. Glyph: Scaling context windows via visual-text compression. CoRR, abs/2510.17800.
  8. Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Xueqing Wang, Changda Zhou, Hongen Liu, Manhui Lin, Yue Zhang, Yubo Zhang, Handong Zheng, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. 2025. Paddleocr-vl: Boosting multilingual document parsing via a 0.9b ultra-co… — [arXiv:2510.14528](https://arxiv.org/abs/2510.14528) find
  9. Daxiang Dong, Mingming Zheng, Dong Xu, Bairong Zhuang, Wenyu Zhang, Chunhua Luo, Haoran Wang, Zijian Zhao, Jie Li, Yuxuan Li, Hanjun Zhong, Mengyue Liu, Jieting Chen, Shupeng Li, Lun Tian, Yaping Feng, Xin Li, Donggang Jiang, Yong Chen, and 16 others. 2025. Qianfan-vl: Domain-enhanced universal visi… — [arXiv:2509.18189](https://arxiv.org/abs/2509.18189) find
  10. Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, and 73 others. 2025. Kimi-vl technical report. CoRR, abs/2504.07491.
  11. Jun Gao, Qian Qiao, Tianxiang Wu, Zili Wang, Ziqiang Cao, and Wenjie Li. 2025. AIM: let any multimodal large language models embrace efficient in-context learning. In Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intellige… find
  12. Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  13. Google DeepMind. 2025. Gemini 3 Flash model card. Model card, Google DeepMind. find
  14. Sun. 2024. MMLONGBENCH-DOC: benchmarking long-context document understanding with visualizations. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. find
  15. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752. find
  16. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, and 58 others. 2025. Glm-4.1v-thinking: Towards versatile multimodal reasoning wit…
  17. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: what’s the real context size of your long-context language models? CoRR, abs/2404.06654.
  18. DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. Block-recurrent transformers. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2… find
  19. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Llmlingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 1…
  20. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe…
  21. Gregory Kamradt. 2023. Needle in a haystack - pressure testing llms. find
  22. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine L… find
  23. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. Docvqa: A dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pages 2199–2208. IEEE.
  24. Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, Zhenjiang Jin, Guang Liang, Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Yuanhong Zheng, Dongsheng Ma, and 42 others. 2025. Mineru2.5: A decoupled vision-language m…
  25. OpenAI. 2025. Update to GPT-5 system card: GPT-5.2. System card, OpenAI. PDF: https://cdn.openai.com/pdf/ 3a4153c8-c748-4b71-8e31-aecbde944f8d/oai_ 5_2_system-card.pdf. ↗
  26. Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. 2025. Omnidocbench: Benchmarking diverse PDF document parsing wi… find
  27. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. 2023. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, U… find
  28. Xinkuan Qiu, Meina Kan, Yongbin Zhou, and Shiguang Shan. 2025. Benchmarking multimodal large language models against image corruptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9014–9023. find
  29. Stefano Rando, Luca Romani, Alessio Sampieri, Yuta Kyuragi, Luca Franco, Fabio Galasso, Tatsunori Hashimoto, and John Yang. 2025. Longcodebench: Evaluating coding llms at 1m context windows. CoRR, abs/2505.07897. — [arXiv:2505.07897](https://arxiv.org/abs/2505.07897) find
  30. Vladimir I. Levenshtein. 1965. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady, 10:707–710. find
  31. ReportLab Inc. 2024. Reportlab toolkit. find
  32. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for… — [arXiv:2403.15388](https://arxiv.org/abs/2403.15388) find
  33. Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. 2025. Deepseekmath-v2: Towards self-verifiable mathematical reasoning. CoRR, abs/2511.22570.
  34. Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. 2025. Scaling long-horizon LLM agent via context-folding. CoRR, abs/2510.11967. — [arXiv:2510.11967](https://arxiv.org/abs/2510.11967) find
  35. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2023. Efficient transformers: A survey. ACM Comput. Surv., 55(6):109:1–109:28. find
  36. Gemma Team. 2025a. Gemma 3 technical report. CoRR, abs/2503.19786. find
  37. Qwen Team. 2025b. Qwen3-vl technical report. CoRR, abs/2511.21631.
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
  39. Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. 2025a. Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models. In Proceedings of the 2025 Conference of the Nations of the Americas…
  40. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, and 56 others. 2025b. Internvl3.5: Advancing open-source multimodal…
  41. Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, and Wenhai Wang. 2024. Needle in A multimodal haystack. In Advances in Neural Information Processing Systems 38: Annual Co… find
  42. Ling Xing, Alex Jinpeng Wang, Rui Yan, Hongyu Qu, Zechao Li, and Jinhui Tang. 2025a. See the text: From tokenization to visual reading. CoRR, abs/2510.18840. — [arXiv:2510.18840](https://arxiv.org/abs/2510.18840) find
  43. Ling Xing, Alex Jinpeng Wang, Rui Yan, and Jinhui Tang. 2025b. Vision-centric token compression in large language model. CoRR, abs/2502.00791. — [arXiv:2502.00791](https://arxiv.org/abs/2502.00791) find
  44. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. 2025. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  45. Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. 2025. Memagent: Reshaping long-context LLM with multi-conv rl-based memory agent. CoRR, abs/2507.02259. — [arXiv:2507.02259](https://arxiv.org/abs/2507.02259) find
  46. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025a. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceeding…
  47. Qianhao Yuan, Jie Lou, Zichao Li, Jiawei Chen, Yaojie Lu, Hongyu Lin, Le Sun, Debing Zhang, and Xianpei Han. 2025b. Memsearcher: Training llms to reason, search and manage memory via end-to-end reinforcement learning. CoRR, abs/2511.02805. find
  48. Jiawei Zhang, Tianyu Pang, Chao Du, Yi Ren, Bo Li, and Min Lin. 2024. Benchmarking large multimodal models against common corruptions. CoRR, abs/2401.11943. — [arXiv:2401.11943](https://arxiv.org/abs/2401.11943) find
  49. Hongbo Zhao, Meng Wang, Fei Zhu, Wenzhuo Liu, Bolin Ni, Fanhu Zeng, Gaofeng Meng, and Zhaoxiang Zhang. 2025. Vtcbench: Can vision-language models understand long context with vision-text compression? CoRR, abs/2512.15649. — [arXiv:2512.15649](https://arxiv.org/abs/2512.15649) find
  50. Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, and Dong Yu. 2024. DOCBENCH: A benchmark for evaluating llm-based document reading systems. CoRR, abs/2407.10701. find
  51. Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. Deepseek-ocr: Contexts optical compression. CoRR, abs/2510.18234.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tu-2026-fico-evaluating-vision-language-models/