Elamur Research⌕
☰
home / papers / 2026 / Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models

Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models

Paper · 2026 · Hanqi Yan, Xiangxiang Cui, Lu Yin et al.

Summary

Vision-language models are usually judged by how well they align images and text. Most prior work treats the remaining modality gap as a defect. This paper argues instead that the gap is common and can be useful. It introduces the Modality Dominance Score (MDS), which compares how strongly each final-layer feature of a CLIP model responds to image versus text inputs, averaged over more than 10k image-text pairs. Based on this score, features are sorted into image-dominant (ImgD), text-dominant (TextD) and cross-modal (CrossD) groups.

The grouping is checked in three ways:

  • ablation probes on COCO;
  • inspection of the images and texts that most activate each feature;
  • new embedding-based metrics for monosemanticity and cross-modal fidelity, which scale better than LLM-based judging.

ImgD features tend to encode low-level visual cues such as patterns and colours. TextD features capture abstract notions such as emotions and atmosphere. The groups also support training-free editing for:

  • reducing gender bias;
  • defending against cross-modal adversarial attacks;
  • steering text-to-image generation.

This suggests that modality-specific components are practical handles for interpreting and lightly editing multimodal models.

Key points

  • Ablation results (CLIP ViT-H/14, LAION-2B; logistic-regression probes on COCO):
    • Zeroing ImgD features (426 removed) lowered image classification accuracy from 0.776 to 0.750 (−2.6%), versus −2.1% for random removal.
    • Zeroing TextD features (554 removed) lowered text classification accuracy from 0.713 to 0.683 (−3.0%), versus −2.0% for random removal.
    • Removing CrossD features had little effect on either task.
  • Feature groups: a feature is labelled TextD or ImgD when its MDS lies more than one standard deviation below or above the mean. Features in between are CrossD.
  • Jailbreak defense: alignment training restricted to TextD features defended best against adversarial samples, followed by CrossD and then ImgD. The setup used 50 adversarial samples, each attacked 100 times.
  • Generation control (Stable Diffusion v2): the authors blended the prompt embedding with a reference-image embedding at α values from 0 to 0.7, but only on the indices of one feature group.
    • Blending on TextD transferred the main subject (a horse) without the reference background.
    • Blending on ImgD carried over background texture and caused distortions at larger α.
  • Limitations: the interpretability metrics were not validated with human studies, and experiments cover only CLIP-family models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States76 works · 32,207 citations🔬The Alan Turing InstituteLondon · United Kingdom31 works · 4,535 citations🎓King's College LondonLondon · United Kingdom46 works · 1,714 citations🎓University of SurreyGuildford · United Kingdom22 works · 677 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Hanqi Yan, Xiangxiang Cui, Lu Yin, Jindong Gu, Paul Pu Liang, Yulan He and Yifei Wang. 2026. Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.588

@inproceedings{yan2026beyond,
  title = {{Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models}},
  author = {Hanqi Yan, Xiangxiang Cui, Lu Yin, Jindong Gu, Paul Pu Liang, Yulan He and Yifei Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.588},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.588}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. URL https://openaipublic.blob. core. windows. net/neuron-explainer/paper/index. html.(Date accessed:… ↗
  2. Gemma Calvert, Charles Spence, and Barry E Stein, editors. 2004. The Handbook of Multisensory Processes. MIT Press. find
  3. Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. 2024. On the robustness of large multimodal models against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24625– 24634. find
  4. Hoagy Cunningham, Aidan Ewart, Logan Riggs, R. Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. International Conference on Learning Representations. find
  5. DeepSeek-AI, Aixin Liu, and Bei Feng et al. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  6. Nelson Elhage, Neel Nanda, Catherine Olsson, and Others. A mathematical framework for transformer circuits. Transformer Circuits Thread (2022). find
  7. Lingzhong Fan, Hai Li, Junjie Zhuo, Yu Zhang, Liangfu Chen, Zhengyi Yang, Congying Chu, Sangma Xie, Angela Laird, Peter Fox, Simon Eickhoff, Chunshui Yu, and Tianzi Jiang. 2016. The human brainnetome atlas: A new brain atlas based on connectional architecture. Cerebral Cortex, 26:bhw157. find
  8. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. find
  9. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational… find
  10. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  11. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. ArXiv, abs/2305.01610.
  12. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. Openclip. If you use this software, please cite it as below. find
  13. Nicholas Jiang, Anish Kachinthaya, Suzanne Petryk, and Yossi Gandelsman. 2025. Interpreting and editing vision-language representations to mitigate hallucinations. In The Thirteenth International Conference on Learning Representations. find
  14. Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. arXiv preprint arXiv:2102.03334.
  15. Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. 2024. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36. find
  16. Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. 2022. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations. find
  17. Zhaowei Li, Wei Wang, YiQing Cai, Qi Xu, Pengyu Wang, Dong Zhang, Hang Song, Botian Jiang, Zhida Huang, and Tao Wang. 2025. Unifiedmllm: Enabling unified representation for multi-modal multi-tasks with large language model. In Findings of the Association for Computational Linguistics: NAACL 2025, pa… find
  18. Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2024. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10):1–42. find
  19. Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems. find
  20. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147.
  21. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
  22. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved baselines with visual instruction tuning.
  23. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, pages 13–23.
  24. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 689–696. nostalgebraist. 2020. interpreting gpt the logit lens. URL https://www. lesswrong.com/po… find
  25. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill, 5(3):e00024– 001. find
  26. Allan Paivio. 1991. Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue canadienne de psychologie, 45(3):255. find
  27. Letitia Parcalabescu and Anette Frank. 2023. Mm-shap: A performance-agnostic metric for measuring multimodal contributions in vision and language models & tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4032–4059. find
  28. Letitia Parcalabescu and Anette Frank. 2025. Do vision & language decoders use images and text equally? how self-consistent are their explanations? In The Thirteenth International Conference on Learning Representations. find
  29. Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. 2024. A concept-based explainability framework for large multimodal models. Advances in Neural Information Processing Systems, 37:135783–135818. find
  30. Anirudh Phukan, Divyansh, Harshit Kumar Morj, Vaishnavi, Apoorv Saxena, and Koustava Goswami. 2025. Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in VLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com… find
  31. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.
  32. Isha Rawal, Shantanu Jaiswal, Basura Fernando, and Cheston Tan. 2023. Dissecting multimodality in videoqa transformer models by impairing modality fusion. In International Conference on Machine Learning. find
  33. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695.
  34. Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. 2025. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models. In The Thirteenth International Conference on Learning Representations. find
  35. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL.
  36. Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2024. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. In The Twelfth International Conference on Learning Representations.
  37. Charles Spence. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73(4):971–995. find
  38. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, and 1 others. 2024. Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13088–…
  39. Vinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, and Mary-Anne Hartley. 2024. Multimodn—multimodal, multi-task, interpretable modular networks. Advances in Neural Information Processing Systems, 36. find
  40. Adly Templeton. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic. find
  41. Leslie G. Ungerleider and James V. Haxby. 1994. ‘what’ and ‘where’ in the human brain. Current Opinion in Neurobiology, 4(2):157–165. find
  42. Yifei Wang, Qi Zhang, Yaoyu Guo, and Yisen Wang. 2024. Non-negative contrastive learning. ICLR. find
  43. Hanqi Yan, Yanzheng Xiang, Guangyi Chen, Yifei Wang, Lin Gui, and Yulan He. 2024a. Encourage or inhibit monosemanticity? revisit monosemanticity from a feature decorrelation perspective. ArXiv, abs/2406.17969. find
  44. Hanqi Yan, Qinglin Zhu, Xinyu Wang, Lin Gui, and Yulan He. 2024b. Mirror: Multiple-perspective self-reflection method for knowledge-rich reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7086–7103, Bangkok, Thailand.… find
  45. Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. 2024. Vlattack: Multimodal adversarial attacks on vision-language tasks via pre-trained models. Advances in Neural Information Processing Systems, 36. find
  46. Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, and 1 others. 2024. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. Advances in Neural Information Processing Systems, 36. find
  47. Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, and Min Zhang. 2025. Evaluating and steering modality preferences in multimodal large language model. arXiv preprint arXiv:2505.20977. — [arXiv:2505.20977](https://arxiv.org/abs/2505.20977) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yan-2026-beyond-cross-modal-alignment-measuring/