Elamur Research⌕
☰
home / papers / 2026 / Attribution-Guided Multi-Object Hallucination and Bias Detection in Vision-Language Models

Attribution-Guided Multi-Object Hallucination and Bias Detection in Vision-Language Models

Paper · 2026 · Sirat Samyoun, Yingtai Xiao and Jian Du

Summary

Vision-language models often name objects that are not in the image (hallucination). They also over-repeat labels, attributes or shared word roots because language priors outweigh visual evidence (linguistic bias). Both problems worsen in cluttered multi-object scenes. There, similarity scores and averaged attention give fake and real objects nearly the same values. The authors propose SHAPLENS, a training-free framework that works across model architectures:

  • Hallucination: a LogitLens-inspired analysis picks the most informative layers, which turn out to be mid-to-late ones. Kernel SHAP then estimates how much each patch inside an object's box supports the predicted label. Scores are fused across layers and compared with adaptive thresholds.
  • Bias: token confidence from the real image is contrasted with a language-only signal from a blank image.

On ADE and COCO with four VLMs, SHAPLENS beats the strongest baselines by 8–12% in accuracy and 10–14% in F1 for hallucination detection. On a curated HQH benchmark it gains up to 6% for bias detection. The method separates hallucination from bias as distinct phenomena and stays stable under image degradation, which makes it useful for reliability auditing.

Key points

  • Evaluated models are LLaVA-7B, LLaVA-13B, Qwen2-VL-2B and LLaMA-3.2-Vision-11B, compared against OPERA, CLIP-based and contextual-embedding baselines.
  • The layer study used 100 ADE images, and the layer patterns were verified on COCO. Attribution accuracy for LLaVA-7B peaks around layers 19–21, and the other models show similar mid-layer trends.
  • Three bias categories are defined: label repetition (e.g., "goat" generated far beyond the true count), attribute bias (excess colour or shape words) and lexical-root propagation (e.g., many "airport …" compounds).
  • Attribute bias is the hardest category, with F1 of roughly 0.26–0.42 for all methods. Label bias is detected most reliably, with F1 near 0.9.
  • Robustness was tested under occlusion (10–30%), Gaussian noise, box shifts (10–50%) and half or quarter partial views. Hallucination metrics change by at most about 0.0003, and classification decisions do not flip.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Sirat Samyoun, Yingtai Xiao and Jian Du. 2026. Attribution-Guided Multi-Object Hallucination and Bias Detection in Vision-Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.210

@inproceedings{samyoun2026attribution,
  title = {{Attribution-Guided Multi-Object Hallucination and Bias Detection in Vision-Language Models}},
  author = {Sirat Samyoun, Yingtai Xiao and Jian Du},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.210},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.210}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oran Barkan, Idan Malkiel, Eitan Barkan, and Noam Koenigstein. 2023. Visual explanations via iterated integrated attributions. In IEEE/CVF International Conference on Computer Vision. find
  2. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112.
  3. Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch-Elor. 2023. Mitigating open-vocabulary caption hallucinations. arXiv preprint arXiv:2312.03631. find
  4. Anthony Brohan, Yevgen Chebotar, and Chelsea Finn. 2023. Robotic manipulation through spatial hallucination mitigation. arXiv preprint arXiv:2303.03534. — [arXiv:2303.03534](https://arxiv.org/abs/2303.03534) find
  5. Liwei Che, Tony Qingze Liu, Jing Jia, Weiyi Qin, Ruixiang Tang, and Vladimir Pavlovic. 2025. Eazy: Eliminating hallucinations in lvlms by zeroing out hallucinatory image tokens. arXiv preprint arXiv:2503.07772. — [arXiv:2503.07772](https://arxiv.org/abs/2503.07772) find
  6. Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David Fouhey, and Joyce Chai. 2024a. Multi-object hallucination in vision language models. Advances in Neural Information Processing Systems, 37:44393–44418. find
  7. Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024b. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425. — [arXiv:2403.00425](https://arxiv.org/abs/2403.00425) find
  8. Ailin Deng and Wei Li. 2024. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. In arXiv:2402.15300. — [arXiv:2402.15300](https://arxiv.org/abs/2402.15300) find
  9. Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. arXiv preprint arXiv:2410.04514. find
  10. Alec Helbling. 2025. Conceptattention: Diffusion transformers learn highly interpretable features. In arXiv:2502.04320. — [arXiv:2502.04320](https://arxiv.org/abs/2502.04320) find
  11. Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vis… find
  12. Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag… find
  13. Donghoon Lee and Yale Song. 2022. Pixelshap: Shapley value-based attribution for dense object detection. In CVPR, pages 11234–11243. find
  14. Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. 2023. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07362. find
  15. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13…
  16. Junyan Lin, Haoran Chen, Yue Fan, Yingqi Fan, Xin Jin, Hui Su, Jinlan Fu, and Xiaoyu Shen. 2025. Multi-layer visual feature fusion in multimodal llms: Methods, analysis, and best practices. arXiv preprint arXiv:2503.06063. — [arXiv:2503.06063](https://arxiv.org/abs/2503.06063) find
  17. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pag…
  18. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  19. Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. 2024. Investigating and mitigating object hallucinations in pretrained vision-language (clip) models. arXiv preprint arXiv:2410.03176. find
  20. Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim. 2023. Langnav: Language as a perceptual representation for navigation. arXiv preprint arXiv:2310.07889. find
  21. Anirudh Phukan, Harshit Kumar Morj, Apoorv Saxena, Koustava Goswami, et al. 2024. Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in vlms. arXiv preprint arXiv:2411.19187. find
  22. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. Pm…
  23. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In ACL, pages 4035– 4045.
  24. Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE International Conference on Computer Vision.
  25. Lloyd S Shapley. 1953. A value for n-person games. Contributions to the Theory of Games, 2(28):307– 317. find
  26. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. International Conference on Machine Learning. find
  27. Sid Tandon and Yang Liu. 2025. Fw-shapley: Real-time estimation of weighted shapley values. In arXiv:2503.06602. — [arXiv:2503.06602](https://arxiv.org/abs/2503.06602) find
  28. Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In NeurIPS, pages 4765–4774.
  29. Ojas Thawkar, Charles D. Lehman, and Andre Araujo. 2023. Multimodal hallucination control in diagnostic imaging. Nature Medicine, 29:1120–1129. find
  30. Christopher Musco and R Teal Witter. 2024. Provably accurate shapley value estimation via leverage score sampling. arXiv preprint arXiv:2410.01917. — [arXiv:2410.01917](https://arxiv.org/abs/2410.01917) find
  31. Shangyu Xing, Fei Zhao, Zhen Wu, Tuo An, Weihao Chen, Chunhui Li, Jianbing Zhang, and Xinyu Dai. 2024. Efuf: Efficient fine-grained unlearning framework for mitigating hallucinations in multimodal large language models. arXiv preprint arXiv:2402.09801. find
  32. Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. 2024. Towards interpreting visual information processing in vision-language models. arXiv preprint arXiv:2410.07149. — [arXiv:2410.07149](https://arxiv.org/abs/2410.07149) find
  33. Yan Neu and Tao Ji. 2025. Detection and assessment of systematic hallucinations of vlms. arXiv preprint arXiv:2503.23573. — [arXiv:2503.23573](https://arxiv.org/abs/2503.23573) find
  34. Bei Yan, Jie Zhang, Zheng Yuan, Shiguang Shan, and Xilin Chen. 2024. Evaluating the quality of hallucination benchmarks for large vision-language models. arXiv preprint arXiv:2406.17115. — [arXiv:2406.17115](https://arxiv.org/abs/2406.17115) find
  35. Nostalgebraist. 2020. Interpreting gpt: The logit lens. find
  36. AI Alignment Forum, August 2020. find
  37. Ziao Yang, Han Yue, Jian Chen, and Hongfu Liu. 2024. On the inflation of knn-shapley value. arXiv preprint arXiv:2405.17489. — [arXiv:2405.17489](https://arxiv.org/abs/2405.17489) find
  38. Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. 2024. Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in lvlms. In European Conference on Computer Vision, pages 395–413. Springer. find
  39. Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2024. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105. find
  40. Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839. — [arXiv:2311.16839](https://arxiv.org/abs/2311.16839) find
  41. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samyoun-2026-attribution-guided-multi-object-hallucination/