Elamur Research⌕
☰
home / papers / 2026 / Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation

Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation

Paper · 2026 · Zhenghua Wang, Yixin Wu, Feiran Zhang et al.

Summary

Vision-language models often hallucinate because they lean on the language priors of their LLM backbone instead of the image. The authors examine attention in LLaVA-v1.5 on POPE queries and find two things. Only a few heads attend meaningfully to image tokens. In hallucinated outputs, these vision-attending heads are under-activated, while heads that ignore the image dominate.

To fix this, they propose HWP, a fine-tuning strategy with two parts:

  • Head-wise perturbation: each head's output is scaled by a random factor (1+ε), with ε drawn from a uniform distribution. This acts as a smoother alternative to dropout and pushes more heads to use visual signals.
  • Visual-guided loss: the loss covers only "visual tokens," meaning text tokens whose logits change strongly when the real image is replaced with random noise.

Across several architectures, model sizes and hallucination benchmarks, HWP reduces hallucinations without hurting general reasoning. This makes it a practical training-time alternative to contrastive decoding or handcrafted attention reweighting.

Key points

  • On POPE, HWP improves accuracy by 2.1 points and F1 by 1.6 points on average across three settings over the base model.
  • In ablations on LLaVA-v1.5-13B (COCO-Caption), CHAIR_s falls from 42.8 to 18.2 and CHAIR_i from 10.6 to 6.2 with the full method. LoRA SFT alone reaches only 28.6 and 8.3.
  • Among noise types, uniform noise works best (CHAIR_s 23.0), ahead of Gaussian (23.4), Rademacher (24.0) and dropout (25.6). Performance peaks at a noise intensity of a = 0.10.
  • Training on visual tokens helps up to roughly the 50th-quantile selection threshold; filtering more tokens than that reduces MME scores.
  • Scores on MMMU and MMMU-Pro stay close to the baselines. The main cost is extra training latency from the visual-token selection step.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Zhenghua Wang, Yixin Wu, Feiran Zhang, Qi Qian, Changze Lv, Xuanjing Huang and Xiaoqing Zheng. 2026. Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1016

@inproceedings{wang2026mitigating,
  title = {{Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation}},
  author = {Zhenghua Wang, Yixin Wu, Feiran Zhang, Qi Qian, Changze Lv, Xuanjing Huang and Xiaoqing Zheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1016},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1016}
}

References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, Bingjie Wang, and Chenliang Xu. 2025. Unveiling visual perception in language models: An attention head analysis approach. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4135–4144. find
  2. Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. 2025a. Perturbollava: Reducing multimodal hallucinations with perturbative visual training. arXiv preprint arXiv:2503.06486. — [arXiv:2503.06486](https://arxiv.org/abs/2503.06486) find
  3. Kaiyuan Chen, Shuangyu Xie, Zehan Ma, Pannag R Sanketi, and Ken Goldberg. 2025b. Robo2vlm: Visual question answering from large-scale in-the-wild robot manipulation datasets. arXiv preprint arXiv:2505.15517. — [arXiv:2505.15517](https://arxiv.org/abs/2505.15517) find
  4. Jingyuan Deng and Yujiu Yang. 2025. Maskcd: Mitigating lvlm hallucinations by image head masked contrastive decoding. arXiv preprint arXiv:2510.02790. find
  5. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.
  6. Aarti Ghatkesar, Uddeshya Upadhyay, and Ganesh Venkatesh. 2025. Looking beyond language priors: Enhancing visual comprehension and attention in multimodal models. arXiv preprint arXiv:2505.05626. — [arXiv:2505.05626](https://arxiv.org/abs/2505.05626) find
  7. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  8. Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. 2025. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03321. — [arXiv:2503.03321](https://arxiv.org/abs/2503.03321) find
  9. Jinyeong Kim, Seil Kang, Jiwoo Park, Junhyeok Kim, and Seong Jae Hwang. 2025. Interpreting attention heads for image-to-text information flow in large vision-language models. arXiv preprint arXiv:2509.17588. — [arXiv:2509.17588](https://arxiv.org/abs/2509.17588) find
  10. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075.
  11. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, and 1 others. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 12…
  12. Yi-Lun Lee, Yi-Hsuan Tsai, and Wei-Chen Chiu. 2024. Delve into visual contrastive decoding for hallucination mitigation of large vision-language models. arXiv preprint arXiv:2412.06775. — [arXiv:2412.06775](https://arxiv.org/abs/2412.06775) find
  13. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13…
  14. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.
  15. Yinan Li and Fang Liu. 2016. Whiteout: Gaussian adaptive noise regularization in deep neural networks. arXiv preprint arXiv:1612.01490. — [arXiv:1612.01490](https://arxiv.org/abs/1612.01490) find
  16. Xiaoyu Liang, Jiayuan Yu, Lianrui Mu, Jiedong Zhuang, Jiaqi Hu, Yuchen Yang, Jiangnan Ye, Lu Lu, Jian Chen, and Haoji Hu. 2024. Mitigating hallucination in visual-language models via re-balancing contrastive decoding. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 482… find
  17. Min Lin, Xiwen Liang, Bingqian Lin, Liu Jingzhi, Zijian Jiao, Kehan Li, Yuhan Ma, Yuecheng Liu, Shen Zhao, Yuzheng Zhuang, and 1 others. 2025. Echovla: Robotic vision-language-action model with synergistic declarative memory for mobile manipulation. arXiv preprint arXiv:2511.18112. — [arXiv:2511.18112](https://arxiv.org/abs/2511.18112) find
  18. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  20. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  21. Jiahui Wang, Zuyan Liu, Yongming Rao, and Jiwen Lu. 2025. Sparsemm: Head sparsity emerges from visual concept responses in mllms. arXiv preprint arXiv:2506.05344. — [arXiv:2506.05344](https://arxiv.org/abs/2506.05344) find
  22. Yang Liu, Xinshuai Song, Kaixuan Jiang, Weixing Chen, Jingzhou Luo, Guanbin Li, and Liang Lin. 2024. Meia: Multimodal embodied perception and interaction in unknown environments. arXiv preprint arXiv:2402.00290. — [arXiv:2402.00290](https://arxiv.org/abs/2402.00290) find
  23. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  24. Eric Nalisnick, Anima Anandkumar, and Padhraic Smyth. 2015. A scale mixture perspective of multiplicative noise in neural networks. arXiv preprint arXiv:1506.03208. — [arXiv:1506.03208](https://arxiv.org/abs/1506.03208) find
  25. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045.
  26. Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö Arık, and Tomas Pfister. 2024. Mitigating object hallucination in mllms via data-augmented phrase-level alignment. arXiv preprint arXiv:2405.18654. — [arXiv:2405.18654](https://arxiv.org/abs/2405.18654) find
  27. Sreetama Sarkar, Yue Che, Alex Gavin, Peter Anthony Beerel, and Souvik Kundu. 2025. Mitigating hallucinations in vision-language models through image-guided head suppression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12492–12511. find
  28. Xu Shen, Xinmei Tian, Tongliang Liu, Fang Xu, and Dacheng Tao. 2017. Continuous dropout. IEEE transactions on neural networks and learning systems, 29(9):3926–3937. find
  29. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  30. Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, and 1 others. 2025. Seeing far and clearly: Mitigating hallucinations in mllms with attention causal decoding. In Proceedings of the Computer Vision and Pattern Recognition C… find
  31. Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. Deepseek-ocr: Contexts optical compression. arXiv preprint arXiv:2510.18234.
  32. Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. 2025. Don’t miss the forest for the trees: Attentional vision calibration for large vision language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1927–1951. find
  33. Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, and 1 others. 2024a. Octopus: Embodied vision-language programmer from environmental feedback. In European conference on computer vision, pages 20–38. Springer. find
  34. Yi Yang, Jiaxuan Sun, Siqi Kou, Yihan Wang, and Zhijie Deng. 2025. Lohovla: A unified vision-language-action model for long-horizon embodied tasks. arXiv preprint arXiv:2506.00411. — [arXiv:2506.00411](https://arxiv.org/abs/2506.00411) find
  35. Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lu-song Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. 2024b. Embodied multi-modal agent trained by an llm from a parallel textworld. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26275–26285. find
  36. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vi…
  37. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-mitigating-hallucinations-vlms-enhancing-visual/