Elamur Research⌕
☰
home / papers / 2026 / INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs

INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs

Paper · 2026 · Yangjunqi, Yuecong Min, Jie Zhang et al.

Summary

Video-LLMs often hallucinate. Their answers may contradict what the video shows (faithfulness) or established world knowledge (factuality). Earlier benchmarks mostly test the first type, and only on clean inputs. INFACT fills this gap with 9,800 multiple-choice QA items drawn from public video-QA sets, instructional resources and synthetic clips. A two-sided taxonomy organizes the items: three levels of visual complexity for faithfulness, and domain, procedural and physical knowledge for factuality. Ambiguous items and items answerable without the video were filtered out, followed by human verification.

Beyond a clean Base mode, the benchmark perturbs the video while keeping the question fixed, using visual degradation, evidence corruption, and temporal intervention (shuffling or reversing frames on order-sensitive items). Two metrics capture these effects. Resist Rate (RR) checks whether correct answers survive perturbation. Temporal Sensitivity Score (TSS) checks whether answers change when temporal order is destroyed.

Across 14 models, clean accuracy tracks robustness overall but does not fully predict it. Misleading captions hurt more than visual noise, and many open-source models ignore frame order on factuality questions. The benchmark therefore exposes reliance on shortcuts that clean-input leaderboards hide.

Key points

  • Dataset and taxonomy: INFACT pairs 9,800 questions with 9,800 videos from both real and synthetic sources.
    • Faithfulness is divided into three levels: objects and their properties, movements and the actions being performed, and how things relate across space and time.
    • Factuality covers Domain (Know-WHAT), Procedural (Know-HOW) and Physical (Know-WHY) knowledge.
  • Models and setup: 14 models were tested zero-shot with 16 uniformly sampled frames. Two are proprietary (GPT-5.1, Gemini3-flash). Twelve are open-source, including the Qwen2.5-VL/Qwen3-VL, InternVL3/3.5, PLLaVA, Tarsier-34B and NVILA-8B families.
  • Clean vs. induced performance: Base accuracy correlates strongly with induced-mode reliability (Pearson r=0.978, Spearman ρ=0.969), yet model rankings shift under perturbation.
  • Caption injection vs. adversarial noise: Injected captions are more damaging than adversarial noise.
    • VideoLLaMA2-7B falls from RR 0.751 to 0.564 on faithfulness.
    • Tarsier-34B falls from 0.909 to 0.625 on factuality.
  • Temporal inertia: Several open-source models score TSS=0 on factuality, while Qwen3VL-8B reaches 0.337 and Gemini3-flash 0.719. In a 500-video ablation, adding frames beyond 16 did not consistently improve reliability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🔬Institute of Computing TechnologyBeijing · China57 works · 3,782 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Yangjunqi, Yuecong Min, Jie Zhang, Shiguang Shan and Xilin Chen. 2026. INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2062

@inproceedings{yangjunqi2026infact,
  title = {{INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs}},
  author = {Yangjunqi, Yuecong Min, Jie Zhang, Shiguang Shan and Xilin Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2062},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2062}
}

References

49 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kyungho Bae, Jinhyung Kim, Sihaeng Lee, Soonyoung Lee, Gunhee Lee, and Jinwoo Choi. 2025. Mash-vlm: Mitigating action-scene hallucination in video-llms through disentangled spatial-temporal representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),… find
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. arXiv prepr…
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. arXiv preprint arXiv:2…
  4. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. 2024. Sharegpt4video: Improving video understanding and generation with better captions. Preprint, arXiv:2406.04325. — [arXiv:2406.04325](https://arxiv.org/abs/2406.04325) find
  5. Zhiyuan Chen, Yuecong Min, Jie Zhang, Bei Yan, Jiahao Wang, Xiaozhen Wang, and Shiguang Shan. 2025. A survey of multimodal hallucination evaluation and detection. International Journal of Computer Vision. find
  6. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476.
  7. Wey Yeh Choong, Yangyang Guo, and Mohan Kankanhalli. 2024. Vidhal: Benchmarking temporal hallucinations in vision llms. arXiv preprint arXiv:2411.16771. — [arXiv:2411.16771](https://arxiv.org/abs/2411.16771) find
  8. Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees G. M. Snoek, and Yuki M. Asano. 2025. Lost in time: A new temporal benchmark for videollms. Preprint, arXiv:2410.07752. — [arXiv:2410.07752](https://arxiv.org/abs/2410.07752) find
  9. Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018. Boosting adversarial attacks with momentum. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9185–9193. find
  10. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, and 2 others. 2025. Video-mme: The first-ever comprehensive evaluati…
  11. Gemini. 2025. Gemini-3-flash. https://blog.google/products/gemini/gemini-3-flash/. ↗
  12. Google DeepMind. 2025. Veo. https://deepmind.google/models/veo/. Accessed: 2026-03-10. ↗
  13. Deepak Gupta, Kush Attal, and Dina Demner-Fushman. 2023. A dataset for medical instructional video classification and question answering. Scientific Data, 10(1):158. find
  14. Ming Kong, Xianzhou Zeng, Luyuan Chen, Yadong Li, Bo Yan, and Qiang Zhu. 2025. Mhbench: Demystifying motion hallucination in videollms. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4):4401–4409. find
  15. Parth Parag Kulkarni, Gaurav Kumar Nayak, and Mubarak Shah. 2024. Cityguessr: City-level video geo-localization on a global scale. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXIII, page 293–311, Berlin, Heidelberg. Springer… find
  16. Kyuho Lee, Euntae Kim, Jinwoo Choi, and Buru Chang. 2025. Noah: Benchmarking narrative prior driven hallucination and omission in video large language models. Preprint, arXiv:2511.06475. — [arXiv:2511.06475](https://arxiv.org/abs/2511.06475) find
  17. Chaoyu Li, Eun Woo Im, and Pooyan Fazli. 2025a. Vid-halluc: Evaluating temporal hallucinations in multimodal large language models for video understanding. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13723– 13733. find
  18. Zheng Liu, Mengjie Liu, Jingzhou Chen, Jingwei Xu, Bin Cui, Conghui He, and Wentao Zhang. 2025b. Fusion: Fully integration of vision-language representations for deep cross-modal understanding. Preprint, arXiv:2504.09925. — [arXiv:2504.09925](https://arxiv.org/abs/2504.09925) find
  19. Chenglin Li, Feng Han, Yikun Wang, Ruilin Li, Shuai Dong, Haowen Hou, Haitao Li, Qianglong Chen, Feng Tao, Jingqi Tong, Yin Zhang, and Jiaqi Wang. 2025b. Videopro: Adaptive program reasoning for long video understanding. Preprint, arXiv:2509.17743.
  20. Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, and 8 others. 2024d. Nvila: Efficient frontier visual lang… find
  21. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. 2024. Mvbench: A comprehensive multi-modal video understanding benchmark. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22195–22206.
  22. Guangtao Lyu, Xinyi Cheng, Chenghao Xu, Qi Liu, Muli Yang, Fen Fang, Huilin Chen, Jiexi Yan, Xu Yang, and Cheng Deng. 2025. Revealing perception and generation dynamics in lvlms: Mitigating hallucinations via validated dominance correction. Preprint, arXiv:2512.18813. — [arXiv:2512.18813](https://arxiv.org/abs/2512.18813) find
  23. Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Fuxiao Liu, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. 2025c. Videohallu: Evaluating and mitigating multi-modal hallucinations on synthetic video understanding. arXiv preprint arXiv:2505.01481. — [arXiv:2505.01481](https://arxiv.org/abs/2505.01481) find
  24. Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, Juekai Lin, Xiaoran Shang, Conghui He, Wentao Zhang, and Lijun Wu. 2026. Mmfinereason: Closing the multimodal reasoning gap via open data-centric methods. Preprint, arXiv:2601.21821. — [arXiv:2601.21821](https://arxiv.org/abs/2601.21821) find
  25. Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024a. A survey on hallucination in large vision-language models. Preprint, arXiv:2402.00253.
  26. Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. 2025a. Phd: A chatgpt-prompted visual hallucination evaluation dataset. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19857–19866. find
  27. Shiyu Liu, Xinyi Wen, Zhibin Lan, Ante Wang, and Jinsong Su. 2026. Countering the over-reliance trap: Mitigating object hallucination for lvlms via a self-validation framework. Preprint, arXiv:2601.22451. — [arXiv:2601.22451](https://arxiv.org/abs/2601.22451) find
  28. Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024b. Tempcompass: Do video llms really understand videos? In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 8… find
  29. Zheng Liu, Hao Liang, Bozhou Li, Wentao Xiong, Chong Chen, Conghui He, Wentao Zhang, and Bin Cui. 2024c. Synthvlm: Towards high-quality and efficient synthesis of image-caption datasets for vision-language models. Preprint, arXiv:2407.20756. — [arXiv:2407.20756](https://arxiv.org/abs/2407.20756) find
  30. Guangtao Lyu, Qi Liu, Chenghao Xu, Jiexi Yan, Muli Yang, Xueting Li, Fen Fang, and Cheng Deng. 2026. Revealing and enhancing core visual regions: Harnessing internal attention dynamics for hallucination mitigation in lvlms. Preprint, arXiv:2602.15556.
  31. OpenAI. 2024. Sora: Creating video from text. https://openai.com/index/sora/. Accessed: 2026-03-10. ↗
  32. OpenAI. 2025. Gpt-5.1: A smarter, more conversational chatgpt. https://openai.com/index/gpt-5-1/. ↗
  33. Ruchit Rawal, Reza Shirkavand, Heng Huang, Gowthami Somepalli, and Tom Goldstein. 2025. Argus: Hallucination and omission evaluation in video-llms. Preprint, arXiv:2506.07371. — [arXiv:2506.07371](https://arxiv.org/abs/2506.07371) find
  34. Bhuiyan Sanjid Shafique, Ashmal Vayani, Muhammad Maaz, Hanoona Abdul Rasheed, Dinura Dissanayake, Mohammed Irfan Kurpath, Yahya Hmaiti, Go Inoue, Jean Lahoud, Md. Safirur Rashid, Shadid Intisar Quasem, Maheen Fatima, Franco Vidal, Mykola Maslych, Ketan Pravin More, Sanoojan Baliah, Hasindri Watawana… — [arXiv:2506.07032](https://arxiv.org/abs/2506.07032) find
  35. Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. 2025. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. In International Conference on Representation Learning, volume 2025, pages 7593–7734. find
  36. Kim Sung-Bin, Oh Hyun-Bin, Lee Jung-Mok, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. 2025. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models. In International Conference on Representation Learning, volume 2025, pages 24244–24271. find
  37. Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. Coin: A large-scale dataset for comprehensive instructional video analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  38. Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems. and analyzing sycophancy in video-llms. Preprint, arXiv:2506.07180. find
  39. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479.
  40. Wan AI. 2025. Wan 2.5: Native audio like veo3 + 1080p video generation. https://wan25.ai/. Accessed: 2026-03-10. ↗
  41. Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. 2024a. Tarsier: Recipes for training and evaluating large video description models. Preprint, arXiv:2407.00634. — [arXiv:2407.00634](https://arxiv.org/abs/2407.00634) find
  42. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  43. Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. 2024b. Videohal-lucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint arXiv:2406.16338. — [arXiv:2406.16338](https://arxiv.org/abs/2406.16338) find
  44. Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994. — [arXiv:2404.16994](https://arxiv.org/abs/2404.16994) find
  45. Bei Yan, Zhiyuan Chen, Yuecong Min, Jie Zhang, Jiahao Wang, Xiaozhen Wang, and Shiguang Shan. 2025. Shale: A scalable benchmark for fine-grained hallucination evaluation in lvlms. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 13442–13449. find
  46. Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Na Zhao, Zhiyu Tan, Hao Li, Xingjun Ma, and Jingjing Chen. 2024. Eventhallusion: Diagnosing event hallucinations in video llms. arXiv preprint arXiv:2409.16597. — [arXiv:2409.16597](https://arxiv.org/abs/2409.16597) find
  47. Jie Zhang, Zhongqi Wang, Mengqi Lei, Zheng Yuan, Bei Yan, Shiguang Shan, and Xilin Chen. 2025. Dysca: A dynamic and scalable benchmark for evaluating perception ability of lvlms. In International Conference on Learning Representations (ICLR). find
  48. Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. 2026. Stable-rag: Mitigating retrieval-permutation-induced hallucinations in retrieval-augmented generation. Preprint, arXiv:2601.02993.
  49. Wenrui Zhou, Mohamed Hendy, Shu Yang, Qingsong Yang, Zikun Guo, Yuyu Luo, Lijie Hu, and Di Wang. 2025. Flattery in motion: Benchmarking Open-Source Video-LLMs VideoLLaMA2-7B Qwen2.5VL-7B Qwen3VL-8B ShareGPT4Video-8B NVILA-8B InternVL3-8B InternVL3.5-8B PLLaVA-13B Qwen2.5VL-32B Qwen3VL-32B PLLaVA-34B… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yangjunqi-2026-infact-diagnostic-benchmark-induced-faithfulness/