Elamur Research⌕
☰
home / papers / 2023 / Detecting and Preventing Hallucinations in Large Vision Language Models

Detecting and Preventing Hallucinations in Large Vision Language Models

Paper · 2023 · Anisha Gunjal, Jihan Yin and Erhan Bas

Summary

Instruction-tuned vision-language models (LVLMs) often invent content when asked for detailed image descriptions. The authors estimate that roughly 30% of text produced by InstructBLIP, then a leading model, is hallucinated: it mentions absent objects, misdescribes attributes, or gets relationships wrong. To address this, they built M-HalDetect, a human-annotated dataset. It contains InstructBLIP descriptions of COCO images, with each sub-sentence span labelled as accurate, inaccurate, or subjective analysis. They used the dataset in two ways. First, they trained sentence-level and sub-sentence-level reward models that detect hallucinations and rank sampled outputs through best-of-n rejection sampling. Second, they introduced Fine-grained Direct Preference Optimization (FDPO), a DPO variant that learns from span-level labels on single responses rather than from ranked pairs. Human evaluation showed that both methods substantially reduce hallucinations. The reward model also helped other LVLMs and tracked human accuracy judgments closely. This makes it a practical, low-cost automatic evaluator for multimodal factuality.

Key points

  • M-HalDetect contains about 16k image–description pairs: 4,000 COCO val2014 images with four sampled InstructBLIP responses each. It is split into 3,200 training images and 800 development images. Labels cover the classes Accurate, Inaccurate, and Analysis.
  • FDPO cut InstructBLIP's hallucination rate by about 41%. The best variant ignored the Analysis spans instead of treating them as negatives. Penalizing analysis tended to lengthen outputs and could add errors.
  • Best-of-64 rejection sampling with the ternary reward model reduced InstructBLIP hallucinations by about 55%. The cost is inference that is n times slower, and gains diminish as n grows.
  • Transfer to other models: best-of-16 sampling lowered hallucinations in LLaVA by 15% and in mPLUG-OWL by 57%.
  • Limitations: the authors ran only one round of feedback collection. They observed signs of overfitting, with descriptions becoming somewhat more generic and less precise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Anisha Gunjal, Jihan Yin and Erhan Bas. 2023. Detecting and Preventing Hallucinations in Large Vision Language Models. arXiv:2308.06394.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.06394 · arXiv 2308.06394

@misc{gunjal2023detecting,
  title = {{Detecting and Preventing Hallucinations in Large Vision Language Models}},
  author = {Anisha Gunjal, Jihan Yin and Erhan Bas},
  year = {2023},
  doi = {10.48550/arxiv.2308.06394},
  eprint = {2308.06394},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.06394}
}

References

30 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716–23736.
  3. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das-Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  4. Bang, Y.; Cahyawijaya, S.; Lee, N.; Dai, W.; Su, D.; Wilie, B.; Lovenia, H.; Ji, Z.; Yu, T.; Chung, W.; Do, Q. V.; Xu, Y.; and Fung, P. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. arXiv:2302.04023.
  5. Bird, S.; Klein, E.; and Loper, E. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”. find
  6. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess… — Language Models are Few-Shot Learners
  7. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. find
  8. Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500.
  9. Engstrom, L.; Ilyas, A.; Santurkar, S.; Tsipras, D.; Janoos, F.; Rudolph, L.; and Madry, A. 2020. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. CoRR, abs/2005.12729. find
  10. Ganesan, K. 2018. ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks. arXiv:1803.01937. — [arXiv:1803.01937](https://arxiv.org/abs/1803.01937) find
  11. Ji, Z.; Lee, N.; Frieske, R.; Yu, T.; Su, D.; Xu, Y.; Ishii, E.; Bang, Y. J.; Madotto, A.; and Fung, P. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12): 1–38.
  12. Li, C. 2023. Large Multimodal Models: Notes on CVPR 2023 Tutorial. arXiv preprint arXiv:2306.14895. — [arXiv:2306.14895](https://arxiv.org/abs/2306.14895) find
  13. Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, W. X.; and Wen, J.-R. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.
  14. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Let’s Verify Step by Step. arXiv preprint arXiv:2305.20050. Lin, T.; Maire, M.; Belongie, S. J.; Bourdev, L. D.; Girshick, R. B.; Hays, J.; Perona, P.; Ramanan, D.; Dol…
  15. Liu, F.; Lin, K.; Li, L.; Wang, J.; Yacoob, Y.; and Wang, L. 2023a. Aligning Large Multi-Modal Model with Robust Instruction Tuning. arXiv preprint arXiv:2306.14565.
  16. Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023b. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  17. Nakano, R.; Hilton, J.; Balaji, S.; Wu, J.; Ouyang, L.; Kim, C.; Hesse, C.; Jain, S.; Kosaraju, V.; Saunders, W.; Jiang, X.; Cobbe, K.; Eloundou, T.; Krueger, G.; Button, K.; Knight, M.; Chess, B.; and Schulman, J. 2021. WebGPT: Browser-assisted question-answering with human feedback. CoRR, abs/2112…
  18. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744.
  19. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–318. Association for Computational Linguistics.
  20. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  21. ScaleAI. 2023. Scale AI Rapid Portal. https://scale.com/ docs/how-rapid-works. Accessed: 2023-08-01. ↗
  22. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. CoRR, abs/1707.06347. find
  23. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. F. 2020. Learning to summarize from human feedback. CoRR, abs/2009.01325. find
  24. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  25. Wang, P.; Li, L.; Chen, L.; Zhu, D.; Lin, B.; Cao, Y.; Liu, Q.; Liu, T.; and Sui, Z. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. find
  26. Wei, J.; Bosma, M.; Zhao, V. Y.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2021. Fine-tuned Language Models Are Zero-Shot Learners. CoRR, abs/2109.01652.
  27. Wu, Z.; Hu, Y.; Shi, W.; Dziri, N.; Suhr, A.; Ammanabrolu, P.; Smith, N. A.; Ostendorf, M.; and Hajishirzi, H. 2023. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. arXiv preprint arXiv:2306.01693.
  28. Ye, Q.; Xu, H.; Xu, G.; Ye, J.; Yan, M.; Zhou, Y.; Wang, J.; Hu, A.; Shi, P.; Shi, Y.; et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178.
  29. Zheng, R.; Dou, S.; Gao, S.; Hua, Y.; Shen, W.; Wang, B.; Liu, Y.; Jin, S.; Liu, Q.; Zhou, Y.; Xiong, L.; Chen, L.; Xi, Z.; Xu, N.; Lai, W.; Zhu, M.; Chang, C.; Yin, Z.; Weng, R.; Cheng, W.; Huang, H.; Sun, T.; Yan, H.; Gui, T.; Zhang, Q.; Qiu, X.; and Huang, X. 2023. Secrets of RLHF in Large Langua…
  30. Ziegler, D. M.; Stiennon, N.; Wu, J.; Brown, T. B.; Radford, A.; Amodei, D.; Christiano, P. F.; and Irving, G. 2019. Fine-Tuning Language Models from Human Preferences. CoRR, abs/1909.08593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gunjal-2023-detecting-preventing-hallucinations-large-vision/