Elamur Research⌕
☰
home / papers / 2026 / Causal Localization of the English Pivot in LLaVA: Mechanistic VLM Analysis and Training-Free Multilingual Steering

Causal Localization of the English Pivot in LLaVA: Mechanistic VLM Analysis and Training-Free Multilingual Steering

Paper · 2026 · Abrar Zahin Raihan and Aurchi Chowdhury

Summary

The authors ask why multilingual vision-language models do worse on non-English visual questions, and where inside the network this English preference appears. They study LLaVA-1.5-7B as a single case. They built a controlled probing set from COCO val2017 with object-naming questions in English, Portuguese, German and Russian. They then used two tools: logit-lens readouts scored with PMI, and causal activation patching measured by the Average Indirect Effect (AIE).

Non-English queries pass through an English-leaning intermediate stage in layers 5–17. This carries the English-pivot idea of Wendler et al. (2024) over to the multimodal setting. The causal signal flows through text-token positions and the residual stream, and it only appears when the image has real content.

From these pivot layers the authors derive steering vectors that need no training. These vectors raise MMMB accuracy for Russian and Portuguese without any fine-tuning. The work suggests that the bias comes from the Vicuna backbone and survives multimodal pre-training. It also offers a cheap diagnostic and fix for deployed VLMs.

Key points

  • Probing set: 195 COCO val2017 examples covering 15 object categories, posed as forced-choice questions in en/pt/de/ru. Evaluation: MMMB, with 200 multiple-choice items each for en, pt and ru.
  • Causal localization: AIE peaks at layer 8 (0.49 averaged across languages). Per-language peaks are PT 0.52 at layer 8, DE 0.62 at layer 10, and RU 0.85 at layer 10. Patching only the attention or MLP outputs gives near-zero effect.
  • Blank-image condition: when the image is replaced with a blank one, PMI stays mostly negative at every layer. The pivot therefore depends on meaningful visual content.
  • Steering results: steering at layers 5–17 with α=0.2 adds +6.5 pp for Russian and +4.0 pp for Portuguese, with Portuguese ending above the English baseline. Performance collapses at larger α; Russian falls to 0% at α=0.5.
  • Transfer and limitations: a German-derived vector matches own-language results for both PT (67.0%) and RU (58.0%), which hints at one shared "non-English" direction. Because α was tuned on the MMMB test set, the reported gains are upper bounds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bangladesh University of Engineering and TechnologyDhaka · Bangladesh7 works · 308 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.English · 2,263 worksMultilingual & cross-lingual NLP · 828 worksVision & language · 657 worksInterpretability · 130 works

Cite

Cite as Abrar Zahin Raihan and Aurchi Chowdhury. 2026. Causal Localization of the English Pivot in LLaVA: Mechanistic VLM Analysis and Training-Free Multilingual Steering. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.25

@inproceedings{raihan2026causal,
  title = {{Causal Localization of the English Pivot in LLaVA: Mechanistic VLM Analysis and Training-Free Multilingual Steering}},
  author = {Abrar Zahin Raihan and Aurchi Chowdhury},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.25},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.25}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jannik Brinkmann, Chris Wendler, Christian Bartelt, and Aaron Mueller. 2025. Large language models share representations of latent grammatical concepts across typologically diverse languages. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Lin… find
  2. Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2025. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics.
  3. Jean de Dieu Nyandwi, Yueqi Song, Simran Khanuja, and Graham Neubig. 2025. Grounding multilingual multimodal LLMs with cultural knowledge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
  4. Yuchun Fan, Yilong Wang, Yongyu Mu, Lei Huang, Bei Li, Xiaocheng Feng, Tong Xiao, and Jingbo Zhu. 2025. Language-specific layer matters: Efficient multilingual enhancement for large vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2025. find
  5. Lucas Resck, Isabelle Augenstein, and Anna Korhonen. 2025. Explainability and interpretability of multilingual large language models: A survey. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
  6. Javier Ferrando and Marta R. Costa-jussà. 2024. On the similarity of circuits across languages: a case study on the subject-verb agreement task. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10115–10125. find
  7. Yueqi Song, Simran Khanuja, and Graham Neubig. 2025. What is missing in multilingual visual reasoning and how to fix it. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2654–2667. find
  8. Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavaš. 2024. mBLIP: Efficient bootstrapping of multilingual vision-LLMs. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research, pages 7–25. find
  9. Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. 2025. Parrot: Multilingual visual instruction tuning. In Proceedings of the 42nd International Conference on Machine Learning. find
  10. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. Preprint, arXiv:2203.14680.
  11. Michal Golovanevsky, William Rudman, Vedant Palit, Ritambhara Singh, and Carsten Eickhoff. 2025. What do VLMs NOTICE? A mechanistic interpretability pipeline for gaussian-noise-free text-image corruption and evaluation. In Proceedings of the 2025 Conference of the North American Chapter of the Assoc… find
  12. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do Llamas work in English? On the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15209–15228.
  13. Andy Zou, Long Phan, Sarah Chen, and 1 others. 2023. Representation engineering: A top-down approach to AI transparency. Preprint, arXiv:2310.01405.
  14. A Tsung-Yi Lin, Michael Maire, Serge Belongie, and 1 others. 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision.
  15. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  16. Muhammad Maaz, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, and Fahad S. Khan. 2024. PALO: A polyglot large multimodal model for 5b people. arXiv preprint arXiv:2402.14818. — [arXiv:2402.14818](https://arxiv.org/abs/2402.14818) find
  17. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/raihan-2026-causal-localization-english-pivot-llava/