Vision language models are blind: Failing to translate detailed visual features into words

Paper · 2024 · Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri et al.

Summary

Modern vision-language models (VLMs) earn strong scores on chart, diagram and document benchmarks. Many of those benchmarks, however, can be partly solved from text alone or from memorized web content. This paper asks whether VLMs can actually see fine spatial detail. The authors built BlindTest, a new suite of 7 synthetic tasks based on eye exams and made from simple 2D shapes, such as lines, circles, squares, letters and grids. Each task is trivial for a person and needs almost no world knowledge. Examples include counting line crossings, judging whether two circles overlap, naming a circled letter, counting nested or overlapping shapes, counting grid rows and columns, and tracing paths on a toy subway map. The tests vary image sizes, line widths and prompt wording. Four leading commercial VLMs did poorly overall, and slow-thinking models and prompting tricks did not help. Accuracy rose sharply when shapes were spaced further apart. Linear probes showed that the vision encoders do hold the needed information, so the failure lies in turning visual features into language. This exposes a perception gap that standard benchmarks miss.

Key points

Models and overall accuracy: GPT-4o, Gemini-1.5 Pro, Claude-3 Sonnet and Claude-3.5 Sonnet averaged 58.07% on BlindTest. Sonnet-3.5 led at 77.84%, against an expected human score of 100%.
Per-task results: Counting intersections of two piecewise-linear plots reached only about 41–76%. Two-circle overlap was about 75–93%. Counting paths between subway stations was about 31–58%.
Other models and prompting: Slow-thinking models (Gemini 2.0 Flash-Thinking, QVQ-Preview) scored 71.59%, versus 72.75% for standard models. Eight open-source models (LLaVA-OneV, Phi-3.5, InternVL-2; 0.5B–72B) did worse than the closed ones. 2-shot, chain-of-thought and meta-prompting gave no gains.
Effect of spacing: Adding space between letters or shapes pushed some models close to 100%. For example, LLaVA-OneV-72B gained +20 points on the circled-letter task, and Sonnet-3 gained +63 points on counting overlapping circles.
Linear probing: Probes on the CLIP and SigLIP encoder features of LLaVA-OneV-S and Phi-3.5 solved the line-intersection and two-circle tasks at ≥99.47% accuracy. This held both before and after the projection layer, while the full VLMs scored only about 33–84%. Code and data are available at VLMsAreBlind.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields, topics and concepts

📚 Shelves 3Computer Science · 19,503 works Artificial Intelligence · 18,805 works Computer Vision and Pattern Recognition · 3,621 works

🧭 Topics 2Language models & pretraining · 4,145 works Vision & language · 827 works

Cite

Cite as Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri and Anh Totti Nguyen. 2024. Vision language models are blind: Failing to translate detailed visual features into words. arXiv:2407.06581.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.06581 · arXiv 2407.06581

@misc{rahmanzadehgervi2024vision,
  title = {{Vision language models are blind: Failing to translate detailed visual features into words}},
  author = {Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri and Anh Totti Nguyen},
  year = {2024},
  doi = {10.48550/arxiv.2407.06581},
  eprint = {2407.06581},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.06581}
}

References

6 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdin, M., Jacobs, S.A., Awan, A.A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al.: Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 (2024) 2. Anthropic: Introducing claude 3.5 sonnet \\ a…
Pearson Education (2014) 15. Inc., C.Z.V.: Vision screening. — OpenAI o1 System Card https://www.zeiss.com/vision-care/us/eye-health-and-care/vision-screening.html, (Accessed on 07/03/2024) 16. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al.: Op… ↗
VLMs fail to translate detailed visual features into words 19. yunxin li, Hu, B., Shi, H., Wang, W., Wang, L., Zhang, M.: Visiongraph: Leveraging large multimodal models for graph theory problems in visual context. In: Forty-first International Conference on Machine Learning (2024), https://openrevi…
Rahmanzadehgervi, Bolton, Taesiri, Nguyen. 34. Mirza, M.J., Karlinsky, L., Lin, W., Doveh, S., Micorek, J., Kozinski, M., Kuhene, H., Possegger, H.: Meta-prompting for automating zero-shot visual recognition with llms. arXiv preprint arXiv:2403.11755 (2024) 35. Mirza, M.J., Karlinsky, L., Lin, W., D…
PMLR (2024) 56. Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: Activitynet-qa: a dataset for understanding complex web videos via question answering. In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial… — MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Rahmanzadehgervi, Bolton, Taesiri, Nguyen. Appendix for: Vision language models are blind: Failing to translate detailed visual features into words find

Updated 3 November 2026

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rahmanzadehgervi-2024-vision-language-models-blind-failing/