Elamur Research⌕
☰
home / papers / 2025 / Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts

Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts

Paper · 2025 · Michal Golovanevsky, William Rudman, Michael A. Lepori et al.

Summary

Multimodal LLMs answer visual questions well, but it is unclear whether they rely on knowledge memorized during pretraining or on the image they are shown. The authors study this with Visual CounterFact, a new dataset of realistic images in which object attributes contradict common knowledge. Examples include a blue strawberry, or a fly shown larger than a strawberry. The dataset covers two tasks, color and size. Probing LLaVA-Next, Qwen2-VL and Janus-Pro layer by layer, they find that early predictions follow stored priors. In mid-to-late layers the answer moves toward the visual evidence, often flipping back and forth along the way. In the final output, pixels usually win, even when the question asks about objects in general. The authors then introduce Pixels Versus Priors (PvP) steering vectors. These are activation-level interventions that push the model toward either the visual answer or the memorized one without changing its weights. The work brings counterfactual-style interpretability from NLP to vision-language models and offers a practical way to control which information source a model trusts.

Key points

  • Dataset construction: Visual CounterFact is built in four steps:
    • objects with strong priors are taken from the McRae norms, plus GPT-4o attribute estimates for CIFAR-100 and ImageNet classes;
    • canonical images on white backgrounds are retrieved via the Google Images API and scored by GPT-4o;
    • counterfactual colors are drawn from the five least likely colors according to LLaVA-Next.
  • Steering results: On average, PvP flips the answer from the prior to the counterfactual one in 99.3% of color cases and 80.8% of size cases.
  • Attention shift: For LLaVA-Next on color, rewording the prompt from "most" to "this" raises attention to image tokens by about 13%. The PvP vector produces a much stronger shift, around 40%.
  • Task differences: Size steering has to start earlier and span more layers than color steering, which is more localized.
  • Asymmetry: Returning a model from visual evidence to its world-knowledge prior is harder than the reverse. Code is available at github.com/rsinghlab/pixels_vs_priors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brown UniversityProvidence · United States76 works · 25,952 citations🎓Tel Aviv UniversityTel Aviv · Israel82 works · 10,655 citations🎓University of TübingenTübingen · Germany75 works · 41,081 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Michal Golovanevsky, William Rudman, Michael A. Lepori, Amir Bar, Ritambhara Singh and Carsten Eickhoff. 2025. Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1262

@inproceedings{golovanevsky2025pixels,
  title = {{Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts}},
  author = {Michal Golovanevsky, William Rudman, Michael A. Lepori, Amir Bar, Ritambhara Singh and Carsten Eickhoff},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1262},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1262}
}

References

48 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang. 2024. Perceptionclip: Visual classification by inferring and conditioning on contexts. Preprint, arXiv:2308.01313. — [arXiv:2308.01313](https://arxiv.org/abs/2308.01313) find
  2. Suraj Anand, Michael A Lepori, Jack Merullo, and Ellie Pavlick. 2025. Dual process learning: Controlling use of in-context vs. in-weights strategies with weight forgetting. International Conference on Learning Representations. find
  3. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. Preprint, arXiv:2303.08112.
  4. Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. Advances in neural information processing systems, 35:18878–18891. find
  5. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. Preprint, arXiv:2501.17811.
  6. Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang. 2023. Can we edit multimodal large language models? arXiv preprint arXiv:2310.08475. find
  7. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696. find
  8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. find
  9. Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. 2025. Inside-out: Hidden factual knowledge in llms. arXiv preprint arXiv:2503.15299. — [arXiv:2503.15299](https://arxiv.org/abs/2503.15299) find
  10. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767. find
  11. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913. find
  12. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. Preprint, arXiv:2401.06102. find
  13. Michal Golovanevsky, William Rudman, Vedant Palit, Ritambhara Singh, and Carsten Eickhoff. 2025. What do vlms notice? a mechanistic interpretability pipeline for gaussian-noise-free text-image corruption and evaluation. Preprint, arXiv:2406.16320. find
  14. Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language transformers for verb understanding. arXiv preprint arXiv:2106.09141. find
  15. Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, and Amir Bar. 2024. Finding visual task vectors. In European Conference on Computer Vision, pages 257–273. Springer. find
  16. Han Huang, Haitian Zhong, Tao Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2024. Vlkeb: A large vision-language model knowledge editing benchmark. arXiv preprint arXiv:2403.07350. — [arXiv:2403.07350](https://arxiv.org/abs/2403.07350) find
  17. Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. 2024. Interpreting and editing vision-language representations to mitigate hallucinations. arXiv preprint arXiv:2410.02762. — [arXiv:2410.02762](https://arxiv.org/abs/2410.02762) find
  18. Tiep Le, Vasudev Lal, and Phillip Howard. 2023. Coco-counterfactuals: Automatically constructed counterfactual examples for image-text pairs. Advances in Neural Information Processing Systems, 36:71195– 71221. find
  19. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. Preprint, arXiv:2407.07895. find
  20. Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2024. Reducing hallucinations in vision-language models via latent space steering. Preprint, arXiv:2410.15778. — [arXiv:2410.15778](https://arxiv.org/abs/2410.15778) find
  21. Grace Luo, Trevor Darrell, and Amir Bar. 2024. Task vectors are cross-modal. arXiv preprint arXiv:2410.22330. find
  22. Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4):547–559. find
  23. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. find
  24. Mazda Moayeri, Wenxiao Wang, Sahil Singla, and Soheil Feizi. 2023. Spuriosity rankings: Sorting data to measure and mitigate biases. Preprint, arXiv:2212.02648. nostalgebraist. 2020. interpreting gpt: the logit lens. LessWrong. — [arXiv:2212.02648](https://arxiv.org/abs/2212.02648) find
  25. Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. 2023. Future lens: Anticipating subsequent tokens from a single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL). Association for Computational Linguistics.
  26. Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. 2023. Finding and editing multi-modal neurons in pre-trained transformers. arXiv preprint arXiv:2311.07470. find
  27. Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2024. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. arXiv preprint arXiv:2402.18154. find
  28. Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2021. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. arXiv preprint arXiv:2112.07566. find
  29. Alex Krizhevsky. 2012. Learning multiple layers of features from tiny images. University of Toronto. find
  30. Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. 2024. Competition dynamics shape algorithmic phases of in-context learning. arXiv preprint arXiv:2412.01003. — [arXiv:2412.01003](https://arxiv.org/abs/2412.01003) find
  31. Andrew Kyle Lampinen, Stephanie CY Chan, Aaditya K Singh, and Murray Shanahan. 2024. The broader spectrum of in-context learning. arXiv preprint arXiv:2412.03782. — [arXiv:2412.03782](https://arxiv.org/abs/2412.03782) find
  32. Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. 2024. Sam 2: Segm… find
  33. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations. find
  34. William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, and Ritambhara Singh. 2025. Forgotten polygons: Multimodal large language models are shape-blind. arXiv preprint arXiv:2502.15969. find
  35. Fawaz Sammani and Nikos Deligiannis. 2024. Interpreting and analyzing clip’s zero-shot image classification via mutual knowledge. arXiv preprint arXiv:2410.13016. — [arXiv:2410.13016](https://arxiv.org/abs/2410.13016) find
  36. Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017. Foil it! find one mismatch between image and language caption. arXiv preprint arXiv:1705.01359.
  37. Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. 2023. The transient nature of emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 36:27801–27819. find
  38. Nishant Subramani, Nivedita Suresh, and Matthew E Peters. 2022. Extracting latent steering vectors from pretrained language models. arXiv preprint arXiv:2205.05124.
  39. Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238– 5248. find
  40. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  41. Martina G Vilas, Timothy Schaumlöffel, and Gemma Roig. 2023. Analyzing vision transformers for image classification in class embedding space. Advances in neural information processing systems, 36:40030– 40041. find
  42. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  43. Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. 2021. Noise or signal: The role of image backgrounds in object recognition. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find
  44. Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. 2023. Mitigating spurious correlations in multi-modal models during fine-tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find
  45. Qinan Yu, Jack Merullo, and Ellie Pavlick. 2023. Characterizing mechanisms for factual recall in language models. arXiv preprint arXiv:2310.15910. find
  46. Tingwei Zhang, Collin Zhang, John X Morris, Eugene Bagdasarian, and Vitaly Shmatikov. 2024. Soft prompts go hard: Steering visual language models with hidden meta-instructions. arXiv preprint arXiv:2407.08970. — [arXiv:2407.08970](https://arxiv.org/abs/2407.08970) find
  47. Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. 2022. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221. — [arXiv:2207.00221](https://arxiv.org/abs/2207.00221) find
  48. Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, and Soham De. 2025. How do language models learn facts? dynamics, curricula and hallucinations. arXiv preprint arXiv:2503.21676. — [arXiv:2503.21676](https://arxiv.org/abs/2503.21676) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/golovanevsky-2025-pixels-versus-priors-controlling-knowledge/