Elamur Research⌕
☰
home / papers / 2026 / Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models

Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models

Paper · 2026 · Ziyao Tang, Pengkun Jiao, Bin Zhu et al.

Summary

Video Large Language Models (Vid-LLMs) do well on video QA, but it is unclear how stable their answers are when a user pushes back. The authors describe "spatiotemporal sycophancy": a model gives a correct, visually grounded answer, then drops it when the user simply denies it. The model often goes further and invents temporal or spatial evidence to support the new, wrong answer.

To measure this, the paper formalizes a gaslighting operator with three pressure styles: Direct Denial, Authority Appeal, and Emotional Pressure. It also defines a Sycophancy Rate, the share of initially correct answers that flip after the pushback. Using this framework, the authors test many open-source and proprietary Vid-LLMs on established video benchmarks and on GasVideo-1000, a new curated benchmark.

The vulnerability appears across models and is severe, even for systems with high baseline accuracy. Prompt-level hardening reduces it in some models, but it does not stop belief reversal or fabricated justifications. This matters for interactive deployments, such as surveillance or embodied agents, where user feedback can override real visual evidence.

Key points

  • GasVideo-1000 contains 1,013 samples balanced across spatial, temporal, and general video-understanding tasks. It was built for lightweight, controlled sycophancy evaluation.
  • Source benchmarks: Video-MME, MVBench, NExT-QA, Perception Test, EgoSchema, ActivityNet-QA, and MSRVTT-QA.
  • Metrics:
    • Sycophancy Rate (SR) is the conditional probability that a correct answer becomes wrong after gaslighting. It is computed from text outputs only, because black-box APIs do not expose logits.
    • The accuracy gap ΔAcc compares accuracy with and without gaslighting.
  • Robustness check: the effect persists under greedy decoding (T = 0), so sampling noise is ruled out as the main cause.
  • Illustrative example: Qwen3-VL-235B-A22B-Instruct changes a correct person count after a denial and cites invented frame timestamps. In another case it switches a correct shape answer to a wrong option.
  • Stated limitations:
    • The only defense examined is prompt hardening.
    • Training-time and tool-based mitigations are left for future work.
    • Automatic scoring of open-ended answers may miss nuances such as partial correctness or hedging.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓Singapore Management UniversitySingapore · Singapore27 works · 2,382 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ziyao Tang, Pengkun Jiao, Bin Zhu, Huiyan Qi, Jingjing Chen and Yu-Gang Jiang. 2026. Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.729

@inproceedings{tang2026spatiotemporal,
  title = {{Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models}},
  author = {Ziyao Tang, Pengkun Jiao, Bin Zhu, Huiyan Qi, Jingjing Chen and Yu-Gang Jiang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.729},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.729}
}

References

36 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip H.S. Torr, Yoon Kim, and Marzyeh Ghassemi. 2025. Vision-language models do not understand negation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29612–29622. \#\# Acknowledgment K… find
  2. This work is supported by the Science and Technology Commission of Shanghai Municipality (No. 24511103100) and the National Research Foundation, Singapore, under the AI Singapore Programme (AISG Award No: AISG3-RPGV-2025-017). find
  3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  4. Data Usage Statement William Croft. 1991. The evolution of negation. Journal of linguistics, 27(1):1–27. find
  5. We use only publicly available benchmark datasets collected and released for research purposes, with consent handled by the original dataset creators. No new data involving human subjects were collected. \#\# Ethics Statement This study is conducted strictly for research purposes, with the primary obj… — Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  6. Google DeepMind. 2025. Gemini 3 pro model card. https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf. ↗
  7. Pengkun Jiao, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, and Yu-Gang Jiang. 2025. Don’t deceive me: Mitigating gaslighting through attention reallocation in lmms. arXiv preprint arXiv:2504.09456. — [arXiv:2504.09456](https://arxiv.org/abs/2504.09456) find
  8. Chaoyu Li, Eun Woo Im, and Pooyan Fazli. 2025a. Vidhalluc: Evaluating temporal hallucinations in multimodal large language models for video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pages 13723–13733. Computer V… find
  9. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024a. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895. find
  10. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2025b. Videochat: Chat-centric video understanding. Science China Information Sciences, 68(10):200102.
  11. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024b. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206.
  12. Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. 2025c. Videohallu: Evaluating and mitigating multi-modal hallucinations on synthetic video understanding. In The Thirty-ninth Annual Conference on Neural Information Processing Syste… find
  13. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12585– 12602.
  14. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244. find
  15. Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, and 1 others. 2023. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36:42… find
  16. Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, Jacob Chalk, Zhifan Zhu, Rhodri Guerrier, Fahd Abdelazim, Bin Zhu, Davide Moltisanti, Michael Wray, Hazel Doughty, and Dima Damen. 2025. Hd-epic:… find
  17. Ruchit Rawal, Reza Shirkavand, Heng Huang, Gowthami Somepalli, and Tom Goldstein. 2025. Argus: Hallucination and omission evaluation in video-llms. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20280–20290. find
  18. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, and 1 others. 2024. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations.
  19. Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. 2025. LongVU: Spatiotemporal adaptive c… find
  20. Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. 2024. Learn" no" to say" yes" better: Improving vision-language models via negations. arXiv preprint arXiv:2403.20312. — [arXiv:2403.20312](https://arxiv.org/abs/2403.20312) find
  21. Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, and 1 others. 2025. Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology.
  22. Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023. Language models are not naysayers: an analysis of language models on negation benchmarks. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 101–114, Toronto, Canada. Associa…
  23. Boshi Wang, Xiang Yue, and Huan Sun. 2023a. Can ChatGPT defend its belief in truth? evaluating LLM reasoning via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11865–11881, Singapore. Association for Computational Linguistics. find
  24. Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. 2023b. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1802–1812. find
  25. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  26. Jinyang Wu, Bin Zhu, Xiandong Zou, Qiquan Zhang, Xu Fang, and Pan Zhou. 2026. Benchmarking gaslighting attacks against speech large language models. In IEEE International Conference on Acoustics, Speech and Signal Processing. find
  27. Bin Zhu, Yinxuan Gui, Huiyan Qi, Jingjing Chen, Chong-Wah Ngo, and Ee-Peng Lim. 2026a. Benchmarking gaslighting negation attacks against multimodal large language models. In ACM International Conference on Multimedia Retrieval. find
  28. Bin Zhu, Hailong Yin, Jingjing Chen, and Yu-Gang Jiang. 2026b. Benchmarking gaslighting negation attacks against reasoning models. In International Conference on Multimedia Modeling, pages 188–202. find
  29. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786. find
  30. Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653. find
  31. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296. find
  32. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134.
  33. Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations. find
  34. Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, and 1 others. 2025a. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106. — [arXiv:2501.13106](https://arxiv.org/abs/2501.13106) find
  35. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun MA, Ziwei Liu, and Chunyuan Li. 2025b. LLaVA-video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research. find
  36. Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. 2023. Learning video representations from large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6586–6597. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2026-spatiotemporal-sycophancy-negation-based-gaslighting/