Elamur Research⌕
☰
home / papers / 2026 / Finding Culture-Sensitive Neurons in Vision-Language Models

Finding Culture-Sensitive Neurons in Vision-Language Models

Paper · 2026 · Xiutian Zhao, Rochelle Choenni, Rohit Saxena et al.

Summary

Vision-language models (VLMs) often answer worse on culturally specific images and questions, but it is unclear where culture-related processing happens inside them. This paper asks whether individual decoder neurons respond more strongly to inputs from particular cultures, and whether those neurons matter for performance. Using the CVQA benchmark, the authors split cultures by CVQA's country–language categories and used English prompts only, so that language skill would not confound the results. They recorded SwiGLU MLP activations in the decoder on questions each model answered correctly, ranked neurons by culture selectivity with several scoring methods, and then switched off the top 1% at inference time. They also propose a new margin-based scorer, Contrastive Activation Margin (ConAct). Across three 7B VLMs and 25 cultures, removing these neurons hurt accuracy mainly on the matching culture and had little effect on other cultures. ConAct gave the cleanest separation on two of the three models. These findings suggest that some cultural knowledge is stored locally in the network. That opens a route to diagnosing cultural bias and to targeted interventions, such as sparse fine-tuning or activation steering.

Key points

  • Models: Qwen2.5-VL-7B, LLaVA-v1.6-Mistral-7B and Pangea-7B were tested on CVQA across 25 cultural groups, using English-only inputs. Code is available at github.com/xiutian/vlm-culture-neuron.
  • Methods compared: random selection (RND), LAP, LAPE, MAD and the new ConAct. ConAct scores a neuron by the activation gap between its most responsive culture and the closest competing culture.
  • Main result: ConAct gave the biggest accuracy drop on the target culture with the least spillover to other cultures on Qwen2.5-VL-7B and Pangea-7B. LLaVA-v1.6-Mistral-7B was harder to target selectively.
  • Ablation behaviour: RND masks barely changed answers. LAPE and ConAct masks pushed models toward wrong but plausible options while keeping the multiple-choice answer format intact.
  • Location of neurons: culture-sensitive neurons are unevenly spread across decoder layers, and the pattern depends on model and selector. For Qwen2.5-VL-7B they concentrate in layer 0 and layers 6–8. The analysis covers decoder MLPs only, not attention heads or vision encoders.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 worksInterpretability · 130 works

Cite

Cite as Xiutian Zhao, Rochelle Choenni, Rohit Saxena and Ivan Titov. 2026. Finding Culture-Sensitive Neurons in Vision-Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.155

@inproceedings{zhao2026finding,
  title = {{Finding Culture-Sensitive Neurons in Vision-Language Models}},
  author = {Xiutian Zhao, Rochelle Choenni, Rohit Saxena and Ivan Titov},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.155},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.155}
}

References

44 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ameen Ali Ali, Shahar Katz, Lior Wolf, and Ivan Titov. 2025. Detecting and pruning prominent but detrimental neurons in large language models. In Second Conference on Language Modeling. find
  2. Amith Ananthram, Elias Stengel-Eskin, Mohit Bansal, and Kathleen McKeown. 2025. See it from my perspective: How language affects cultural bias in image understanding. In The Thirteenth International Conference on Learning Representations. find
  3. Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1778–1796, Dublin, Ireland. Association for Computatio… find
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:25…
  5. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations. find
  6. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network Dissection: Quantifying Interpretability of Deep Visual Representations . In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3319–3327, Los Alamitos, CA, USA. IEEE Computer Society. find
  7. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. 2020. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071–30078. find
  8. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland. Asso…
  9. Olena Burda-Lassen, Aman Chadha, Shashank Goswami, and Vinija Jain. 2025. How culturally aware are vision-language models? In 2025 IEEE 6th International Conference on Image Processing, Applications and Systems (IPAS), pages 1–6. IEEE. find
  10. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  11. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applica… find
  12. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  13. Junfeng Fang, Zongze Bi, Ruipeng Wang, Houcheng Jiang, Yuan Gao, Kun Wang, An Zhang, Jie Shi, Xiang Wang, and Tat-Seng Chua. 2024. Towards neuron attributions in multimodal large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24,… find
  14. Danial Namazifard and Lukas Galke Poech. 2025. Isolating culture neurons in multilingual large language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics… find
  15. Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. 2024. Universal neurons in GPT2 language models. Transactions on Machine Learning Research. find
  16. Geert Hofstede. 1980. Culture and organizations. International studies of management & organization, 10(4):15–41. find
  17. Kaichen Huang, Jiahao Huo, Yibo Yan, Kun Wang, Yutao Yue, and Xuming Hu. 2024. Miner: Mining the underlying pattern of modality-specific neurons in multimodal large language models. Preprint, arXiv:2410.04819. — [arXiv:2410.04819](https://arxiv.org/abs/2410.04819) find
  18. Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu. 2024. MMNeuron: Discovering neuron-level domain-specific interpretation in multimodal large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6801–6816, Miami, Florida, USA. Assoc… find
  19. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  20. Claire Kramsch. 2014. Language and culture. AILA review, 27(1):30–55. find
  21. Wen Lai, Viktor Hangya, and Alexander Fraser. 2024. Style-specific neurons for steering LLMs in text style transfer. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13427–13443, Miami, Florida, USA. Association for Computational Linguistics. find
  22. Avinash Madasu, Vasudev Lal, and Phillip Howard. 2025. Cultural awareness in vision-language models: A cross-country exploration. In CVPR 2025 Workshop Vision Language Models For All. find
  23. Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishwarya Agrawal. 2024. Benchmarking vision language models for cultural understanding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,…
  24. Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. 2025. Towards interpreting visual information processing in vision-language models. In The Thirteenth International Conference on Learning Representations. find
  25. Jean De Dieu Nyandwi, Yueqi Song, Simran Khanuja, and Graham Neubig. 2025. Grounding multilingual multimodal LLMs with cultural knowledge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24187– 24231, Suzhou, China. Association for Computational Lingu…
  26. Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. 2024. Finding and editing multi-modal neurons in pre-trained transformers. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1012–1037, Bangkok, Thailand. Association for Computational Linguistics. find
  27. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, 51(3):907–1004.
  28. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  29. Taisei Yamamoto, Ryoma Kumon, Danushka Bollegala, and Hitomi Yanaka. 2025. Neuron-level analysis of cultural understanding in large language models. Preprint, arXiv:2510.08284. — [arXiv:2510.08284](https://arxiv.org/abs/2510.08284) find
  30. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thaila… find
  31. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. Qwen2 technical report. Preprint, arXiv:240… find
  32. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cu… find
  33. Zeping Yu and Sophia Ananiadou. 2024. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3267–3280, Miami, Florida, USA. Association for Computational Linguistics. find
  34. Noam Shazeer. 2020. Glu variants improve transformer. Preprint, arXiv:2002.05202. find
  35. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
  36. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
  37. Edward Burnett Tylor. 1871. Primitive culture: researches into the development of mythology, philosophy, religion, art, and custom, volume 2. J. Murray. find
  38. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2024. Neurons in large language models: Dead, n-gram, positional. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1288–1301, Bangkok, Thailand. Association for Computational Linguistics. find
  39. Jiaqi Xu, Cuiling Lan, and Yan Lu. 2025. Deciphering functions of neurons in vision-language models. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 3173–3181. find
  40. Srishti Yadav, Zhi Zhang, Daniel Hershcovich, and Ekaterina Shutova. 2025. Beyond words: Exploring cultural value sensitivity in multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7592–7608, Albuquerque, New Mexico. Association for Computational Lingui… find
  41. Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2025. Pangea: A fully open multilingual multimodal LLM for 39 languages. In The Thirteenth International Conference on Learning… find
  42. Xiutian Zhao, Björn Schuller, and Berrak Sisman. 2026. Discovering and causally validating emotion-sensitive neurons in large audio-language models. Preprint, arXiv:2601.03115.
  43. Xiutian Zhao, Ke Wang, and Wei Peng. 2024. Measuring the inconsistency of large language models in preferential ranking. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 171–176, Bangkok, Thailand. Association for Computational Linguistics. find
  44. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-finding-culture-sensitive-neurons-vision/