Elamur Research
☰
home / papers / 2023 / Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2311.16922

Summary

Large vision-language models (LVLMs) often describe objects that are not in the image. The authors trace this object hallucination to two sources: statistical bias from co-occurrence patterns in pretraining data, and the language priors of the underlying LLM decoder. They show that making the image less clear strengthens both tendencies. In one example, a noised picture of a black banana pushes the model toward "yellow" or "green."

Building on this, they propose Visual Contrastive Decoding (VCD). It needs no training and no external models. At each decoding step, the model runs on the original image and on a copy distorted with diffusion-style Gaussian noise. The next-token logits from the distorted view are then subtracted from the original ones, which down-weights tokens the model would favour without real visual evidence.

VCD was tested on LLaVA-1.5, InstructBLIP and Qwen-VL. It reduced hallucinations on POPE and MME and also improved general perception scores on MME and LLaVA-Bench. Because it is cheap and works across model families, it is a practical alternative to fine-tuning, RLHF or post-hoc revisers.

Key points

  • Decoding rule: VCD scores tokens as (1+α)·logit(y|x,v) − α·logit(y|x,v′), where v′ is the noised image built over T diffusion-style noise steps. Ablations of T, α and β are in the supplementary materials.
  • Benchmark gains: improvements reach +7.4 F1 on POPE and +18% on MME, across LLaVA-1.5, InstructBLIP and Qwen-VL.
  • GPT-4V-aided evaluation on LLaVA-Bench open-ended captioning (scale of 10):
    • Qwen-VL accuracy rose from 4.76 to 6.69.
    • LLaVA-1.5 accuracy rose from 3.23 to 4.15.
    • Detailedness also improved for all three models.
  • Qualitative cases: VCD removes co-occurrence hallucinations, such as a "dining table" and "fork" invented next to a chair.
  • Limitations and code: the method uses only plain Gaussian noise and covers image-text tasks, not video. Code is available at github.com/DAMO-NLP-SG/VCD.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

130 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.16922 · arXiv 2311.16922

@misc{leng2023mitigating,
  title = {{Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding}},
  author = {Leng, Sicong and Zhang, Hang and Chen, Guanzheng and Li, Xin and Lu, Shijian and Miao, Chunyan and Bing, Lidong},
  year = {2023},
  doi = {10.48550/arxiv.2311.16922},
  eprint = {2311.16922},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.16922}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9690–9698, 2020. 1, 3 find
  2. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. arXiv preprint arXiv:1606.07356, 2016. 1, 3 find
  3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022. 2 find our card
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 2 find our card
  5. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1, 2, 5 find our card
  6. Ali Furkan Biten, Lluı́s Gómez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1381–1390, 2022. 2 find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 2 find
  8. Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. arXiv preprint arXiv:2310.01957, 2023. 1 — [arXiv:2310.01957](https://arxiv.org/abs/2310.01957) find
  9. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022. 2 find our card
  10. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2 find
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 2 find
  12. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, [27] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on compute… find