Elamur Research⌕
☰
home / papers / 2026 / Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs

Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs

Paper · 2026 · Wenrui Zhou, Mohamed Hendy, Shu Yang et al.

Summary

Video-LLMs often give up on what they actually see in a video when a user pushes a wrong answer. This behaviour is called sycophancy. It has been studied for text LLMs and, to a lesser degree, for image models, but no systematic test existed for video. The authors built ViSE, a benchmark of multiple-choice questions on clips from MSVD, MSRVTT and NExT-QA. They chose the clips by measuring how easily a baseline model (Qwen2.5-VL 7B) could be talked out of a correct answer (MSS) and how rarely it accepted valid corrections (CRS). The benchmark covers seven prompt-based sycophancy types, including different strengths of bias, "Are You Sure?" challenges, explicit endorsement or rejection, and mimicry. It also annotates visual task categories. The authors evaluated nine variants of six Video-LLMs. They then tested two mitigation methods that need no training: key-frame selection and representation steering of hidden activations. Both cut sycophancy noticeably, and steering worked best. Simple prompting or voting did not help. The work matters because it provides diagnostics and practical defences for building video assistants that stay grounded in the evidence.

Key points

  • ViSE has 367 videos and 6,367 MCQs. A subset of 141 longer videos carries 1,158 questions labelled with 8 visual task categories (temporal, descriptive, causal and others).
  • Sycophancy is measured with the Misleading Susceptibility Score (MSS). When the video selection was repeated with InternVL 2.5, 87.8% of the chosen videos overlapped, which suggests limited selection bias.
  • On Qwen2.5-VL-7B, the average MSS changed as follows from a baseline of 44.92:
    • majority vote: 44.71;
    • contradiction-checking prompt: 52.84, which is worse;
    • key-frame selection: 30.39;
    • representation steering: 19.52.
  • Key-frame selection reduces the sycophancy rate by up to 22.01%. Steering brings MSS close to zero in several categories for LLaVA-OneVision.
  • Representation steering needs white-box access, so it only works for open-weights models. The benchmark also leaves out hour-long videos.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Wenrui Zhou, Mohamed Hendy, Shu Yang, Qingsong Yang, Zikun Guo, Yuyu Luo, Lijie Hu and Di Wang. 2026. Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.369

@inproceedings{zhou2026flattery,
  title = {{Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs}},
  author = {Wenrui Zhou, Mohamed Hendy, Shu Yang, Qingsong Yang, Zikun Guo, Yuyu Luo, Lijie Hu and Di Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.369},
  url = {https://doi.org/10.18653/v1/2026.acl-long.369}
}

References

59 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bang An, Chengzhi Zhang, Zaiqiao Meng, Jie Zhao, Jie Fu, and Helen Meng. 2024. Chaos with keywords: Exposing large language models’ sycophancy to misleading keywords and evaluating defense strategies. arXiv preprint arXiv:2402.03463. find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923.
  3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In FAccT’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, New York, NY, USA. Associat… find
  4. Jing Bi, Junjia Guo, Susan Liang, Guangyu Sun, Luchuan Song, Yunlong Tang, Jinxi He, Jiarui Wu, Ali Vosoughi, Chen Chen, and Chenliang Xu. 2025. VERIFY: A benchmark of visual explanation and reasoning for investigating multimodal reasoning fidelity. arXiv preprint arXiv:2503.11557. — [arXiv:2503.11557](https://arxiv.org/abs/2503.11557) find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. 2024. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint a… — [arXiv:2410.10818](https://arxiv.org/abs/2410.10818) find
  7. Meng Cao, Tianyu Wu, Ziqi Li, Yixin Zhang, Zhipin Liu, Yuxiang Wang, Jiaqi Zhang, Yupan Liu, Kun Li, Dongmei Zhang, and Nan Duan. 2025. Video SimpleQA: Towards factuality evaluation in large video language models. arXiv preprint arXiv:2503.18923. — [arXiv:2503.18923](https://arxiv.org/abs/2503.18923) find
  8. Wei Chen, Zhen Huang, Liang Xie, Binbin Lin, Houqiang Li, Le Lu, Xinmei Tian, Deng Cai, Yonggang Zhang, Wenxiao Wang, and 1 others. 2024a. From yes-men to truth-tellers: addressing sycophancy in large language models with pinpoint tuning. arXiv preprint arXiv:2409.01658. — [arXiv:2409.01658](https://arxiv.org/abs/2409.01658) find
  9. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024b. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.
  10. Andrew Fanous, Youssef Cinqotrois, Muhammad El-Nokrashy, Mohamed El-Ghannam, Mohamed Abdalla, and Fakhri Karray. 2025. Syceval: Evaluating llm sycophancy. arXiv preprint arXiv:2502.08177. — [arXiv:2502.08177](https://arxiv.org/abs/2502.08177) find
  11. Xilin Gong, Shu Yang, Zehua Cao, Lynne Billard, and Di Wang. 2026. Faithful-patchscopes: Understanding and mitigating model bias in hidden representations explanation of large language models. arXiv preprint arXiv:2602.00300. — [arXiv:2602.00300](https://arxiv.org/abs/2602.00300) find
  12. Zikun Guo, Xinyue Xu, Pei Xiang, Shu Yang, Xin Han, Di Wang, and Lijie Hu. 2025. Benchmarking and mitigate psychological sycophancy in medical vision-language models. arXiv e-prints, pages arXiv–2509. find
  13. Jingyu Hu, Shu Yang, Xilin Gong, Hongming Wang, Weiru Liu, and Di Wang. 2025. Monica: Real-time monitoring and calibration of chain-of-thought sycophancy in large reasoning models. arXiv preprint arXiv:2511.06419. — [arXiv:2511.06419](https://arxiv.org/abs/2511.06419) find
  14. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  15. Mohamed Fazli Imam, Chenyang Lyu, and Alham Fikri Aji. 2025. Can multimodal LLMs do visual temporal understanding and reasoning? the answer is no! arXiv preprint arXiv:2501.10674. — [arXiv:2501.10674](https://arxiv.org/abs/2501.10674) find
  16. Xinyan Jiang, Wenjing Yu, Di Wang, and Lijie Hu. 2026. Global evolutionary steering: Refining activation steering control via cross-layer consistency. arXiv preprint arXiv:2603.12298. — [arXiv:2603.12298](https://arxiv.org/abs/2603.12298) find
  17. Xinyan Jiang, Lin Zhang, Jiayi Zhang, Qingsong Yang, Guimin Hu, Di Wang, and Lijie Hu. 2025. Msrs: Adaptive multi-subspace representation steering for attribute alignment in large language models. arXiv preprint arXiv:2508.10599. — [arXiv:2508.10599](https://arxiv.org/abs/2508.10599) find
  18. Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Jameel Hassan Abdul Samadh, Muzammal Naseer, Federico Tombari, Fahad Shahbaz Khan, and Salman Khan. 2025. How good is my video lmm? complex video reasoning and robustness evaluation suite for video-lmms. In Proceedings of the IEEE/CVF Conference on Comp… find
  19. Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J Kim. 2023. Large language models are temporal and causal reasoners for video question answering. arXiv preprint arXiv:2310.15747. find
  20. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. TVQA: Localized, compositional video question answering. In EMNLP.
  21. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
  22. Chenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, and Jiaqi Wang. 2026. Videothinker: Building agentic videollms with llm-guided tool reasoning. Preprint, arXiv:2601.15724. — [arXiv:2601.15724](https://arxiv.org/abs/2601.15724) find
  23. Chenglin Li, Feng Han, Yikun Wang, Ruilin Li, Shuai Dong, Haowen Hou, Haitao Li, Qianglong Chen, Feng Tao, Jingqi Tong, Yin Zhang, and Jiaqi Wang. 2025a. Videopro: Adaptive program reasoning for long video understanding. Preprint, arXiv:2509.17743.
  24. Hongji Li, Manjiang Yu, Priyanka Singh, Xue Li, Di Wang, Lijie Hu, and 1 others. 2025b. Towards reasoning-preserving unlearning in multimodal large language models. arXiv preprint arXiv:2512.17911. — [arXiv:2512.17911](https://arxiv.org/abs/2512.17911) find
  25. Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Wei Li, Shufei Zhang, Mao Su, Wanli Ouyang, Yuqiang Li, and Dongzhan Zhou. 2025c. Chemvlm: Exploring the power of multimodal large language models i… find
  26. Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, and Jindong Wang. 2025d. Understanding and mitigating the bias inheritance in LLM-based data augmentation on downstream tasks. arXiv preprint arXiv:2502.04419. — [arXiv:2502.04419](https://arxiv.org/abs/2502.04419) find
  27. Shuo Li, Tao Ji, Xiaoran Fan, Linsheng Lu, Leyi Yang, Yuming Yang, Zhiheng Xi, Rui Zheng, Yuran Wang, xh. zhao, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025e. Have the vlms lost confidence? a study of sycophancy in vlms. In The Thirteenth International Conference on Learning Representations. find
  28. Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, and 1 others. 2024b. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574. — [arXiv:2501.00574](https://arxiv.org/abs/2501.00574) find
  29. Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang, Conghui He, Bin Cui, Chong Chen, and Wentao Zhang. 2024. KeyVideoLLM: Towards large-scale video keyframe selection. arXiv preprint arXiv:2407.03104. — [arXiv:2407.03104](https://arxiv.org/abs/2407.03104) find
  30. Lars Malmqvist. 2024. Sycophancy in large language models: Causes and mitigations. arXiv preprint arXiv:2411.15287.
  31. Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl Vondrick, Mikhail Sirotenko, Cordelia Schmid, and Tobias Weyand. 2025. MINERVA: Evaluating complex video reasoning. Preprint, arXiv:2505.00681. — [arXiv:2505.00681](https://arxiv.org/abs/2505.00681) find
  32. Ethan Perez, Saffron Huang, Floris Chan, Jack Val-madre, Yaru revanche, Scott Heiner, Jeff Z. HaoTrent, Andy Zou, Amanda Askell, Newton Cheng, Anna Chen, Vlad Schogol, Nicholas Joseph, Nelson Elhage, Ben Mann, Danny Hernandez, kamile lukosiute, Zac Hatfield-Dodds, Jackson Kernion, and 8 others. 2022… — Discovering Language Model Behaviors with Model-Written Evaluations
  33. Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. 2025. Omnia de egotempo: Benchmarking temporal understanding of multi-modal LLMs in egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24129–24138. find
  34. Mrinal Sharma, Tuka Alhanai, and Marzyeh Ghassemi. 2023. Flattering to deceive: The impact of sycophantic behavior on user trust in large language model. arXiv preprint arXiv:2311.06013. — [arXiv:2311.06013](https://arxiv.org/abs/2311.06013) find
  35. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, and 1 others. 2024. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations.
  36. Dan Shi, Renren Jin, Tianhao Shen, Weilong Dong, Xinwei Wu, and Deyi Xiong. 2024. IRCAN: Mitigating knowledge conflicts in LLM generation via identifying and reweighting context-aware neurons. arXiv preprint arXiv:2406.18406. — [arXiv:2406.18406](https://arxiv.org/abs/2406.18406) find
  37. Haochuan Xu, Yun Sing Koh, Shuhuai Huang, Zirun Zhou, Di Wang, Jun Sakuma, and Jingfeng Zhang. 2025. Model-agnostic adversarial attack and defense for vision-language-action models. arXiv preprint arXiv:2510.13237. — [arXiv:2510.13237](https://arxiv.org/abs/2510.13237) find
  38. Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chenliang Xu. 2023. Video understanding with large language models: A survey. arXiv prep…
  39. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296. find
  40. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  41. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  42. Keyu Wang, Jin Li, Shu Yang, Zhuoran Zhang, and Di Wang. 2026. When truth is overridden: Uncovering the internal origins of sycophancy in large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33566–33574. find
  43. Suyuchen Wang, Rui Li, Yejia Liu, Zongqian Wu, Zipeng Li, Yizhou Wang, Chuang Gan, Min-Yen Kan, and Ziwei Liu. 2024. Multitrust: A comprehensive benchmark for trustworthy multimodal large language models. arXiv preprint arXiv:2406.07057. — [arXiv:2406.07057](https://arxiv.org/abs/2406.07057) find
  44. Jason Wei, Dieuwke Hupkes, Slav Petrov, Mostafa Dehghani, Vincent Zhao, Orhan Firat, Aakanksha Chowdhery, Quoc V. Le, Denny Zhou, Diyi Yang, and Adam Roberts. 2023. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958. find
  45. Siqi Wen, Shu Yang, Shaopeng Fu, Jingfeng Zhang, Lijie Hu, and Di Wang. 2026. Concept-based dictionary learning for inference-time safety in vision language action models. arXiv preprint arXiv:2602.01834. — [arXiv:2602.01834](https://arxiv.org/abs/2602.01834) find
  46. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786. find
  47. Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia, pages 1645–1653, New York, NY, USA. ACM. find
  48. Junqi Yang, Yuecong Min, Jie Zhang, Shiguang Shan, and Xilin Chen. 2026. Infact: A diagnostic benchmark for induced faithfulness and factuality hallucinations in video-llms. Preprint, arXiv:2603.11481.
  49. Tiancheng Yang, Lin Zhang, Jiaye Lin, Guimin Hu, Di Wang, and Lijie Hu. 2025. Tracing and mitigating hallucinations in multimodal llms via dynamic attention localization. arXiv preprint arXiv:2509.07864. — [arXiv:2509.07864](https://arxiv.org/abs/2509.07864) find
  50. Junchi Yao, Lokranjan Lakshmikanthan, Annie Zhao, Danielle Zhao, Shu Yang, Zikang Ding, Di Wang, and Lijie Hu. 2026. Hearing is believing? evaluating and analyzing audio language model sycophancy with syaudio. arXiv preprint arXiv:2601.23149. — [arXiv:2601.23149](https://arxiv.org/abs/2601.23149) find
  51. Manjiang Yu, Hongji Li, Priyanka Singh, Xue Li, Di Wang, and Lijie Hu. 2025. Pixel: Adaptive steering via position-wise injection with exact estimated levels under subspace calibration. arXiv preprint arXiv:2510.10205. — [arXiv:2510.10205](https://arxiv.org/abs/2510.10205) find
  52. Qianchi Zhang, Hainan Zhang, Liang Pang, Yongxin Tong, Hongwei Zheng, and Zhiming Zheng. 2025a. Less is more: Compact clue selection for efficient retrieval-augmented generation reasoning. arXiv preprint arXiv:2502.11811. — [arXiv:2502.11811](https://arxiv.org/abs/2502.11811) find
  53. Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. 2026. Stable-rag: Mitigating retrieval-permutation-induced hallucinations in retrieval-augmented generation. arXiv preprint arXiv:2601.02993.
  54. Zhuoran Zhang, Tengyue Wang, Xilin Gong, Yang Shi, Haotian Wang, Di Wang, and Lijie Hu. 2025b. When modalities conflict: How unimodal reasoning uncertainty governs preference dynamics in mllms. arXiv preprint arXiv:2511.02243. — [arXiv:2511.02243](https://arxiv.org/abs/2511.02243) find
  55. Yao Zhao, Yuheng Bu, Xuesong Gao, Yao Fu, Michael Zhang, and Qiang Xu. 2023. On evaluating adversarial robustness of large vision-language models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  56. Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, Chengye Wang, Ziyao Shangguan, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan. 2025. MMVU: Measuring expert-level multi-discipline video understanding. In Proceedings of the… find
  57. Weihua Zheng, Xin Huang, Zhengyuan Liu, Tarun Kumar Vangani, Bowei Zou, Xiyan Tao, Yuhao Wu, Ai Ti Aw, Nancy F. Chen, and Roy Ka-Wei Lee. 2025. Adamcot: Rethinking cross-lingual factual reasoning through adaptive multilingual chain-of-thought. Preprint, arXiv:2501.16154. — [arXiv:2501.16154](https://arxiv.org/abs/2501.16154) find
  58. Zirun Zhou, Zhengyang Xiao, Haochuan Xu, Jing Sun, Di Wang, and Jingfeng Zhang. 2025. Goal-oriented backdoor attack against vision-language-action models via physical objects. arXiv preprint arXiv:2510.09269. — [arXiv:2510.09269](https://arxiv.org/abs/2510.09269) find
  59. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2023. Representa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-flattery-motion-benchmarking-analyzing-sycophancy/