Elamur Research⌕
☰
home / papers / 2026 / The Halo Effect and Language Takeover: Spatiotemporal Attention Decay Explains Vision-Language Model Failures in Simple Visual Counting

The Halo Effect and Language Takeover: Spatiotemporal Attention Decay Explains Vision-Language Model Failures in Simple Visual Counting

Paper · 2026 · Haochen Zhao and Sujian Li

Summary

The paper asks why vision-language models that handle complex multimodal reasoning still miscount a dozen dots. Rather than blaming resolution or patch tokenization, the authors track how models attend to the image while they generate a two-digit answer. They built a synthetic probe set of 256 clean grayscale images, each containing 10–16 non-overlapping dark circles. They then ran Qwen3-VL (2B, 4B, 8B) and Gemma3 (4B, 12B) with greedy decoding. At each decoding step, they took the final-layer cross-modal attention from the generated token to the image patches and scored it with two new metrics: a Gini-based Visual Sparsity and a normalized Visual Entropy.

They report two coupled failure patterns, which they group as Spatiotemporal Attention Decay:

  • Halo Effect: attention gathers around the outer boundary of the dot cluster instead of on individual dots.
  • Language Takeover: visual focus weakens after the first digit, so the units digit comes more from language priors than from the image.

The work matters because it gives a step-by-step mechanistic explanation of counting hallucinations. It also suggests that fixes should keep visual grounding alive throughout decoding.

Key points

  • Dataset: 256 images at 1024×1024 with circles of radius 10 px. Counts are drawn uniformly from [10, 16], which forces a two-token answer. Rejection sampling keeps the circles from occluding each other.
  • Accuracy: Qwen3-VL-2B and Gemma3-4B score under 6%. Qwen3-VL-4B and 8B level off near 50%. Most models under-count, while Qwen3-VL-4B over-counts slightly.
  • Temporal decay: in every model tested, attention sparsity is lower for the units digit than for the tens digit. Weaker models also show a sharp rise in entropy at the second step.
  • Exception: Gemma3-12B keeps the sharpest attention, with the highest sparsity and lowest entropy, yet reaches only 27.73% accuracy. The authors conclude that focused attention is necessary but not sufficient for correct counting.
  • Limitations: the evidence is correlational, with no causal tests such as activation steering or attention masking. Closed-source models were not evaluated. Only two-digit counts with direct-answer prompts were studied, and Chain-of-Thought prompting was not tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Haochen Zhao and Sujian Li. 2026. The Halo Effect and Language Takeover: Spatiotemporal Attention Decay Explains Vision-Language Model Failures in Simple Visual Counting. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.40

@inproceedings{zhao2026halo,
  title = {{The Halo Effect and Language Takeover: Spatiotemporal Attention Decay Explains Vision-Language Model Failures in Simple Visual Counting}},
  author = {Haochen Zhao and Sujian Li},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.40},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.40}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  2. Freya Behrens, Luca Biggio, and Lenka Zdeborová. 2025. Counting in small transformers: The delicate interplay between attention and feed-forward layers. Preprint, arXiv:2407.11542. — [arXiv:2407.11542](https://arxiv.org/abs/2407.11542) find
  3. Mikhail S. Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V. Sapunov. 2021. Memory transformer. Preprint, arXiv:2006.11527. — [arXiv:2006.11527](https://arxiv.org/abs/2006.11527) find
  4. Jinjin Cao, Zhiyang Chen, Zijun Wang, Liyuan Ma, Weijian Luo, and Guojun Qi. 2025. When images speak louder: Mitigating language bias-induced hallucinations in vlms through cross-modal guidance. Preprint, arXiv:2510.10466. — [arXiv:2510.10466](https://arxiv.org/abs/2510.10466) find
  5. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. Preprint, arXiv:2309.16588. — [arXiv:2309.16588](https://arxiv.org/abs/2309.16588) find
  6. Omri Kaduri, Shai Bagon, and Tali Dekel. 2024. What’s in the image? a deep-dive into the vision of vision language models. Preprint, arXiv:2411.17491. — [arXiv:2411.17491](https://arxiv.org/abs/2411.17491) find
  7. Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. 2025. See what you are told: Visual attention sink in large multimodal models. Preprint, arXiv:2503.03321. — [arXiv:2503.03321](https://arxiv.org/abs/2503.03321) find
  8. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. Preprint, arXiv:1908.08593.
  9. Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Fuxiao Liu, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. 2025. Videohallu: Evaluating and mitigating multi-modal hallucinations on synthetic video understanding. Preprint, arXiv:2505.01481. — [arXiv:2505.01481](https://arxiv.org/abs/2505.01481) find
  10. Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. 2023. Teaching clip to count to ten. Preprint, arXiv:2302.12066. — [arXiv:2302.12066](https://arxiv.org/abs/2302.12066) find
  11. Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu, and Donald E. Brown. 2025. Can vision-language models count? a synthetic benchmark and analysis of attention-based interventions. Preprint, arXiv:2511.17722. — [arXiv:2511.17722](https://arxiv.org/abs/2511.17722) find
  12. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  13. Michael Toker, Ido Galil, Hadas Orgad, Rinon Gal, Yoad Tewel, Gal Chechik, and Yonatan Belinkov. 2025. Padding tone: A mechanistic analysis of padding tokens in t2i models. Preprint, arXiv:2501.06751. find
  14. Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. Preprint, arXiv:1906.04284.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-halo-effect-language-takeover-spatiotemporal/