Elamur Research⌕
☰
home / papers / 2026 / CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation

CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation

Paper · 2026 · Ashish Mishra, Tarun Kumar, Arpit Shah et al.

Summary

Vision–language models often mention objects that are not in the image. Existing remedies tend either to make outputs vaguer or to require costly retraining. The Hewlett Packard Labs authors propose CEBC, a post-generation layer that works with any VLM and needs no training. It first produces a small pool of candidates (greedy, beam and best-of-K samples). An external detector (DETR, with OWL-ViT v2 as backup) then scores every object a candidate mentions, and a mention is allowed only if its score passes a threshold τ. Instead of hand-tuning τ, the authors derive it from a conformal quantile of detector scores on hallucinated mentions in a small held-out set, which makes hallucination risk explicitly controllable. Candidates are ranked by risk first and then by a quality score that discourages length drift. If no candidate is clean, unsupported objects are deleted with minimal rewording. Across several VLMs on MS-COCO, GQA and POPE, CEBC lowers CHAIR and POPE hallucination measures while keeping or improving CIDEr, BLEU and CLIPScore. This gives grounded outputs without any fine-tuning.

Key points

  • CEBC has three variants: CEBC_QUAL (the main one, using quality-aware selection plus minimal editing), CEBC_BAL (aimed at balanced accuracy in VQA) and CEBC_RISK (a conservative, precision-oriented version).
  • The method was evaluated on LLaVA-1.5-7B, InstructBLIP, Qwen2-VL-7B-Instruct and Idefics2-8B:
    • CHAIR was measured on 500 MS-COCO Karpathy test images, with τ calibrated on 120 images (δ = 0.05).
    • POPE used 1,000 questions each from MS-COCO and GQA, with thresholds calibrated on 250 held-out questions.
  • In an open-vocabulary test on OpenCHAIR (500 images, OWLv2 as the evidence model, Qwen2.5-1.5B-Instruct as judge), the score dropped from 0.4173 to 0.3789, a relative improvement of about 9.2%.
  • Runtime on LLaVA-1.5-7B rises from 0.526 to 0.830 s per image at K=6, roughly 1.58× greedy decoding. Most of the extra cost comes from generating candidates; the detector pass takes only about 0.053 s.
  • The authors list several limitations:
    • Performance depends on how well the detector covers the objects in the image.
    • On MS-COCO, calibration needed ground-truth object presence.
    • Only object-level errors are handled, not wrong attributes, relations or counts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksText generation · 202 works

Cite

Cite as Ashish Mishra, Tarun Kumar, Arpit Shah, Suparna Bhattacharya and Martin Foltin. 2026. CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2142

@inproceedings{mishra2026cebc,
  title = {{CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation}},
  author = {Ashish Mishra, Tarun Kumar, Arpit Shah, Suparna Bhattacharya and Martin Foltin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2142},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2142}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch-Elor. 2024. Mitigating open-vocabulary caption hallucinations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22680–22698. find
  2. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems.
  3. Ailin Deng, Zhirui Chen, and Bryan Hooi. 2024. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models. find
  4. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, and 1 others. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature machine intelligence, 5(3):220–235.
  5. Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. Preprint, arXiv:2403.14003. — [arXiv:2403.14003](https://arxiv.org/abs/2403.14003) find
  6. Junjie Fei, Teng Wang, Jinrui Zhang, Zhenyu He, Chengjie Wang, and Feng Zheng. 2023. Transferable decoding with visual entities for zero-shot image captioning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3136–3146. find
  7. Joshua Feinglass and Yezhou Yang. 2024. Trope: Training-free object-part enhancement for seamlessly improving fine-grained zero-shot image captioning. Association for Computational Linguistics. find
  8. Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, and Jinqiao Wang. 2025. Cracking the code of hallucination in lvlms with vision-aware head divergence. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volum…
  9. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2025. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  10. Feiyang Huang, Yang Cao, and Jingyue Zhong. 2025. Opcap: Object-aware prompting captioning. In Proceedings of the 7th ACM International Conference on Multimedia in Asia, pages 1–7. find
  11. Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. Preprint, arXiv:2311.17911. find
  12. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Preprint, arXiv:2405.02246.
  13. Yi-Lun Lee, Yi-Hsuan Tsai, and Wei-Chen Chiu. 2024. Delve into visual contrastive decoding for hallucination mitigation of large vision-language models. arXiv preprint arXiv:2412.06775. — [arXiv:2412.06775](https://arxiv.org/abs/2412.06775) find
  14. Tsung-En Lin, Kuan-Yi Lee, and Hung-Yi Lee. 2025. Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models. arXiv preprint arXiv:2510.12851. — [arXiv:2510.12851](https://arxiv.org/abs/2510.12851) find
  15. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024a. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253. find
  16. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024b. Improved baselines with visual instruction tuning. Preprint, arXiv:2310.03744.
  17. Shi Liu, Kecheng Zheng, and Wei Chen. 2024c. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pages 125–140. find
  18. Shi Liu, Kecheng Zheng, and Wei Chen. 2024d. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. Preprint, arXiv:2407.21771. — [arXiv:2407.21771](https://arxiv.org/abs/2407.21771) find
  19. Kazuki Matsuda, Yuiga Wada, and Komei Sugiura. 2024. Deneb: A hallucination-robust automatic evaluation metric for image captioning. In Proceedings of the Asian Conference on Computer Vision, pages 3570–3586. find
  20. Yeji Park, Deokyeong Lee, Junsuk Choe, and Buru Chang. 2025. Convis: Contrastive decoding with hallucination visualization for mitigating hallucinations in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6434–6442. find
  21. Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, and Trevor Darrell. 2024. Aloha: A new measure for hallucination in captioning models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find
  22. Vishnu Prabhakaran, Purav Aggarwal, Vinay Kumar Verma, Gokul Swamy, and Anoop Saladi. 2025. VADE: Visual attention guided hallucination detection and elimination. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14949–14965, Vienna, Austria. Association for Computational… find
  23. Liam van der Poel, Ryan Cotterell, and Clara Meister. 2022. Mutual information alleviates hallucinations in abstractive summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5956–5965, Abu Dhabi, United Arab Emirates. Association for Computat… find
  24. Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. 2024a. Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites. In International Conference on Multimedia Modeling, pages 32–45. Springer. find
  25. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the…
  26. Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. 2025. Mitigating object hallucination in large vision-language models via image-grounded guidance. In Forty-second International Conference on Machine Learning. find
  27. Ge Zheng, Jiaye Qian, Jiajin Tang, and Sibei Yang. 2025. Why lvlms are more prone to hallucinations in longer responses: The role of context. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4101–4113. find
  28. Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, Xiatian Zhu, Lei Deng, Hongbin Liu, and Zhen Lei. 2025. Bayesian test-time adaptation for vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29999– 30009. find
  29. Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2024. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Representations. find
  30. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045.
  31. Sreetama Sarkar, Yue Che, Alex Gavin, Peter Anthony Beerel, and Souvik Kundu. 2025. Mitigating hallucinations in vision-language models through image-guided head suppression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12492–12511. find
  32. Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. 2019. Conformal prediction under covariate shift. Advances in neural information processing systems, 32. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mishra-2026-cebc-conformal-evidence-bounded-control/