Elamur Research⌕
☰
home / papers / 2026 / VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation

VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation

Paper · 2026 · Seongheon Park, Changdae Oh, Hyeong Kyu Choi et al.

Summary

Large vision-language models (LVLMs) often hallucinate. Self-evaluation methods built for text-only LLMs, such as entropy, verbalized confidence, Semantic Entropy and EigenScore, mostly capture linguistic plausibility. They can therefore stay confident when an image contradicts what the language prior expects. VAUQ is a training-free, label-free alternative. It checks whether a model's confidence is actually supported by the picture. Its central quantity is the Image-Information Score (IS), which compares the model's uncertainty about its answer with and without the visual input. An unsupervised masking step uses the model's visual attention to hide the most-attended patches. This penalizes answers that remain confident once the key visual evidence is gone. The final hallucination score is a weighted combination of predictive entropy and this core-masked IS. The method was tested on LLaVA, Qwen2.5-VL and InternVL3.5 across benchmarks including ViLP, MMVet, VisualCoT and CVBench. VAUQ beat existing LLM- and LVLM-based self-evaluators throughout, and its gains were largest on counterfactual images. It needs no external judge, extra training or repeated sampling, so it is cheap to deploy for selective prediction.

Key points

  • Pilot study: On ViLP (factual vs. counterfactual images sharing one prompt) with LLaVA-1.5-7B, standard methods degrade sharply on counterfactual images. Entropy falls by 40.9% and EigenScore by 26.0%.
  • Counterfactual gains: VAUQ improves self-evaluation AUROC by +13.3% over state-of-the-art baselines in counterfactual settings.
  • Efficiency: Compared with the sampling-based VL-Uncertainty on LLaVA-1.5-7B, VAUQ cuts per-sample inference time by 94.6% and raises AUROC by 21.4%.
  • Hyperparameters: The entropy/IS weight α works best between 0.5 and 1.5, with vision-heavy tasks like CVBench favoring larger values. The best masking ratio K is 30% on CVBench and 40% on MMVet.
  • Transfer and code: Settings tuned on MMVet reach 72.3% AUROC on CVBench, versus 73.2% when tuned on CVBench itself. Code is available at github.com/deeplearning-wisc/vauq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓University of Wisconsin–MadisonMadison · United States54 works · 49,089 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Seongheon Park, Changdae Oh, Hyeong Kyu Choi, Sean Du and Sharon Li. 2026. VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1321

@inproceedings{park2026vauq,
  title = {{VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation}},
  author = {Seongheon Park, Changdae Oh, Hyeong Kyu Choi, Sean Du and Sharon Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1321},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1321}
}

References

63 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Ping Chen, Xiaoqin Zhang, and Shijian Lu. 2025. Mitigating object hallucinations in large vision-language models with assembly of global and local attention. In CVPR. find
  2. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it’s lying. In EMNLP Findings.
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  4. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In ICLR. find
  5. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. Inside: Llms’ internal states retain the power of hallucination detection. In ICLR.
  6. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In NeurIPS.
  7. Xuefeng Du, Chaowei Xiao, and Yixuan Li. 2024. Haloscope: Harnessing unlabeled llm generations for hallucination detection. In NeurIPS. find
  8. Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. 2025. Truthprint: Mitigating large vision-language models object hallucination via latent truthful-guided pre-intervention. In ICCV. find
  9. Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, and 1 others. 2024. Agent ai: Surveying the horizons of multimodal interaction. arXiv preprint arXiv:2401.03568. — [arXiv:2401.03568](https://arxiv.org/abs/2401.03568) find
  10. Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In CVPR. find
  11. Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. 2022. Large-scale unsupervised semantic segmentation. find
  12. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In EMNLP. find
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR. 26542 9
  15. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, and 1 others. 2025. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints. find
  16. Xiaomin Li, Zhou Yu, Ziji Zhang, Yingying Zhuang, Swair Shah, Narayanan Sadagopan, and Anurag Beniwal. 2025. Semantic volume: Quantifying and detecting both external and internal uncertainty in llms. arXiv preprint arXiv:2502.21239. — [arXiv:2502.21239](https://arxiv.org/abs/2502.21239) find
  17. Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. 2025. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. In CVPR. find
  18. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. In TMLR. find
  19. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  20. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. In TMLR. find
  21. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2024a. Mitigating hallucination in large multi-modal models via robust instruction tuning. In ICLR.
  22. Omri Kaduri, Shai Bagon, and Tali Dekel. 2025. What’s in the image? a deep-dive into the vision of vision language models. In CVPR. find
  23. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024b. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253.
  24. Zaid Khan and Yun Fu. 2024. Consistency and uncertainty: Identifying unreliable responses from black-box vision-language models for selective visual question answering. In CVPR. find
  25. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurIPS.
  26. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, and 1 others. 2024. Openvla: An open-source vision-language-action model. In CoRL. find
  27. Kimi Team. 2025. Kimi-vl technical report. arXiv preprint arXiv:2504.07491.
  28. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR.
  29. Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. Vlind-bench: Measuring language priors in large vision-language models. In NACCL Findings. find
  30. Seongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Seo. 2024. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. In ACL Findings. find
  31. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In CVPR.
  32. Qing Li, Jiahui Geng, Chenyang Lyu, Derui Zhu, Maxim Panov, and Fakhri Karray. 2024. Reference-free hallucination detection for large vision-language models. In EMNLP Findings. find
  33. Sheng Liu, Haotian Ye, and James Zou. 2025. Reducing hallucinations in large vision-language models via latent space steering. In ICLR. find
  34. Shi Liu, Kecheng Zheng, and Wei Chen. 2024c. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In ECCV. find
  35. Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. 2019. Improving referring expression grounding with cross-modal attention-guided erasing. In CVPR. find
  36. Lin Long, Changdae Oh, Seongheon Park, and Sharon Li. 2025. Understanding language prior of lvlms by contrasting chain-of-embedding. arXiv preprint arXiv:2509.23050. — [arXiv:2509.23050](https://arxiv.org/abs/2509.23050) find
  37. Yifan Lu, Ziqi Zhang, Chunfeng Yuan, Jun Gao, Con-gxuan Zhang, Xiaojuan Qi, Bing Li, and Weiming Hu. 2025. Mitigating hallucinations in large vision-language models by self-injecting hallucinations. In EMNLP Findings. find
  38. Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. 2025. Probing visual language priors in vlms. In ICML. find
  39. Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In ICLR. find
  40. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In EMNLP. find
  41. OpenAI. 2025. Gpt-5: Large multimodal model by openai. https://openai.com/research/gpt-5.26543 10 ↗
  42. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2025. Llms know more than they show: On the intrinsic representation of llm hallucinations. In ICLR.
  43. Seongheon Park, Xuefeng Du, Min-Hsuan Yeh, Haobo Wang, and Yixuan Li. 2025. Steer llm latents for hallucination detection. In ICML. find
  44. Seongheon Park and Yixuan Li. 2025. Glsim: Detecting object hallucinations in lvlms via global-local similarity. In NeurIPS. find
  45. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS.
  46. Anirudh Phukan, Harshit Kumar Morj, Apoorv Saxena, Koustava Goswami, and 1 others. 2025. Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in vlms. In NACCL. find
  47. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In ICML.
  48. Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2022. Out-of-distribution detection and selective generation for conditional language models. In ICLR.
  49. Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2020. Grad-cam: visual explanations from deep networks via gradient-based localization. In IJCV. find
  50. Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024. Visual cot: Advancing multimodal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In NeurIPS. find
  51. Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek. 2022. Adversarial masking for self-supervised learning. In ICML. find
  52. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2024. Aligning large multimodal models with factually augmented RLHF. In ACL Findings. fully open, vision-centric exploration of multi…
  53. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025a. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  54. Yiming Wang, Pei Zhang, Baosong Yang, Derek F Wong, and Rui Wang. 2025b. Latent space chain-of-embedding enables output-free llm self-evaluation. In ICLR. find
  55. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817.
  56. Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. 2025. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. In CVPR. find
  57. Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. 2023. Mitigating spurious correlations in multi-modal models during fine-tuning. In ICML. find
  58. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML.
  59. Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. 2025. Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation. In CVPR. find
  60. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS.
  61. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In ICCV. find
  62. Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2024. Analyzing and mitigating object hallucination in large vision-language models. In ICLR.
  63. Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, and 1 others. 2024. Cambrian-1: A 26544 11 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2026-vauq-vision-aware-uncertainty-quantification/