Elamur Research⌕
☰
home / papers / 2026 / Cross-Modal Taxonomic Generalization in (Vision-) Language Models

Cross-Modal Taxonomic Generalization in (Vision-) Language Models

Paper · 2026 · Tianyang Xu, Marcelo Sandoval-Castañeda, Karen Livescu et al.

Summary

The paper asks whether hypernym knowledge that a language model acquires purely from text can carry over when inputs arrive through vision. The authors build VLMs in which a pretrained image encoder and a pretrained LM stay frozen, and only a projector between them is trained. Training uses yes/no questions about whether an object category appears in an image. They then progressively remove supervision on higher-level categories (hypernyms such as bird or animal) and test whether the model can still detect those categories in unseen images. Even with zero hypernym exposure during projector training, the models score above chance, so the LM's taxonomic knowledge transfers across modalities. Counterfactual training gives a more nuanced result. Generalization survives when image–label pairings are shuffled within a category, but it collapses to chance when pairings are shuffled across categories. Cross-modal generalization therefore appears to depend on two things: relational knowledge from language and visual coherence in the input. This matters for understanding how much meaning LMs bring to multimodal systems.

Key points

  • Data: a subset of the THINGS image database released by Rodriguez et al. (2025), with 1,216 leaf categories and 17,336 images. The leaves are grouped under 53 hypernyms, and each leaf has 1.67 hypernyms on average (range 1–6).
  • Task format: English polar questions of the type "Is there a {category} in this image?". Each image is paired with a positive question and a negative one, where the negative category is sampled uniformly from other categories.
  • Two counterfactual controls:
    • Within-category label shuffling preserves the category structure, and models perform close to those trained on the original data.
    • Across-category shuffling breaks that structure, and models drop to chance.
  • Limitations: the study uses a single simple QA-style task, one modality pair, modest model sizes, and English only. An attempt to study spatial relations failed because the models could not learn the base task.
  • Code: available at github.com/sally-xu-42/cross-modal-taxonomic-gen.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States119 works · 57,950 citations🎓Toyota Technological Institute at ChicagoChicago · United States35 works · 13,377 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Tianyang Xu, Marcelo Sandoval-Castañeda, Karen Livescu, Greg Shakhnarovich and Kanishka Misra. 2026. Cross-Modal Taxonomic Generalization in (Vision-) Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.742

@inproceedings{xu2026cross,
  title = {{Cross-Modal Taxonomic Generalization in (Vision-) Language Models}},
  author = {Tianyang Xu, Marcelo Sandoval-Castañeda, Karen Livescu, Greg Shakhnarovich and Kanishka Misra},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.742},
  url = {https://doi.org/10.18653/v1/2026.acl-long.742}
}

References

70 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021. Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color. In Proceedings of the 25th Conference on Computational Natural Language Learning.
  2. Lawrence W Barsalou. 1985. Ideals, central tendency, and frequency of instantiation as determinants of graded structure in categories. Journal of experimental psychology: learning, memory, and cognition, 11(4):629. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… — Language Models are Few-Shot Learners
  4. Rosa Cao and Daniel Yamins. 2021. Explanatory Models in Neuroscience: Part 2 – Constraint-based Intelligibility. arXiv preprint arXiv:2104.01489. — [arXiv:2104.01489](https://arxiv.org/abs/2104.01489) find
  5. Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. 2025. Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas. In Forty-second International Conference on Machine Learning. find
  6. Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. 2023. PaLI-3 visi… — [arXiv:2310.09199](https://arxiv.org/abs/2310.09199) find
  7. Andrei Cimpian and Ellen M Markman. 2008. Preschool children’s use of cues to generic meaning. Cognition, 107(1):19–53. 16328 10 find
  8. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, and 1 others. 2025. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models. In Proceedings of the Computer Vision and…
  9. Jeffrey L Elman. 2004. An Alternative View of the Mental Lexicon. Trends in Cognitive Sciences, 8(7):301– 306. find
  10. John Firth. 1957. A Synopsis of Linguistic Theory, 1930-1955. Studies in Linguistic Analysis, 6(4):10– 32. find
  11. Gilbert Harman. 1982. Conceptual role semantics. Notre Dame Journal of Formal Logic, 23(2):242– 256. find
  12. Stevan Harnad. 1990. The Symbol Grounding Problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346. find
  13. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
  14. Marti A. Hearst. 1992. Automatic Acquisition of Hyponyms from Large Text Corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics.
  15. Stella Frank and Emily Allaway. 2025. VISaGE: Understanding Visual Generics and Exceptions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. find
  16. Martin N. Hebart, Adam H. Dickter, Alexis Kidder, Wan Y. Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris I. Baker. 2019. THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLOS ONE, 14(10):1–24. find
  17. Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. 2025. Hidden in Plain Sight: VLMs Overlook their Visual Representations. In Second Conference on Language Modeling. find
  18. Tianze Hua, Tian Yun, and Ellie Pavlick. 2025. How do Vision-Language Models Process Conflicting Information across Modalities? arXiv preprint arXiv:2507.01790. find
  19. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. BLINK: Multimodal Large Language Models Can See But Not Perceive. In ECCV.
  20. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. Position: The Platonic Representation Hypothesis. In Proceedings of the 41st International Conference on Machine Learning.
  21. Maayan Geffet and Ido Dagan. 2005. The Distributional Inclusion Hypotheses and Lexical Entailment. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics.
  22. Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021. Language models use monotonicity to assess NPI licensing. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. find
  23. Susan A Gelman, Elizabeth A Ware, and Felicia Kleinberg. 2010. Effects of Generic Language on Category Content and Structure. Cognitive Psychology, 61(3):273–301. find
  24. Michal Golovanevsky, William Rudman, Michael A. Lepori, Amir Bar, Ritambhara Singh, and Carsten Eickhoff. 2025. Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts. In EMNLP.
  25. Alison Gopnik and Henry M Wellman. 1994. The Theory Theory. Mapping the Mind: Domain Specificity in Cognition and Culture, page 257. find
  26. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  27. Michael Hanna and David Mareček. 2021. Analyzing BERT’s Knowledge of Hypernymy via Prompting. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. find
  28. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. Mission: Impossible language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  29. Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s “up” with Vision-Language Models? Investigating their Struggle with Spatial Reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. find
  30. Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. 2024. Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. In Forty-first International Conference on Machine Learning. find
  31. Charles Kemp and Joshua B Tenenbaum. 2009. Structured Statistical Models of Inductive Reasoning. Psychological Review, 116(1):20. find
  32. Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. VLind-Bench: Measuring Language Priors in Large Vision-Language Models. In Findings of the Association for Computational Linguistics: NAACL 2025. 16329 11 find
  33. Cara Su-Yi Leong and Tal Linzen. 2024. Testing Learning Hypotheses using Neural Networks by Manipulating Learning Data. arXiv preprint arXiv:2407.04593.
  34. M Lynne Murphy. 2003. Semantic Relations and the Lexicon: Antonymy, Synonymy and other Paradigms. Cambridge University Press. find
  35. Sarah-Jane Leslie. 2008. Generics: Cognition and Acquisition. Philosophical Review, 117(1):1–47. find
  36. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, and 7 o… — DINOv2: Learning Robust Visual Features without Supervision
  37. Molly Lewis, Martin Zettersten, and Gary Lupyan. 2019. Distributional Semantics as a Source of Visual Knowledge. Proceedings of the National Academy of Sciences, 116(39):19237–19238. find
  38. Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, and Anders Søgaard. 2024. Do vision and language models share concepts? a vector space alignment study. Transactions of the Association for Computational Linguistics, 12:1232–1249. find
  39. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In Thirty-seventh Conference on Neural Information Processing Systems.
  40. Qiawen Liu, Jeroen van Paridon, and Gary Lupyan. 2025. Learning about Color from Language. Communications Psychology, 3(1):60. find
  41. Gary Lupyan, Rasha Abdel Rahman, Lera Boroditsky, and Andy Clark. 2020. Effects of Language on Visual Perception. Trends in Cognitive Sciences, 24(11):930–944. find
  42. Matthew Mandelkern and Tal Linzen. 2024. Do Language Models’ Words Refer? Computational Linguistics, 50(3):1191–1200. find
  43. Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2023. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representations. find
  44. Kanishka Misra and Kyle Mahowald. 2024. Language models learn rare phenomena from less rare phenomena: The case of the missing AANNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 913–929, Miami, Florida, USA. Association for Computational Linguisti…
  45. Dimitri Coelho Mollo and Raphaël Millière. 2024. The Vector Grounding Problem. Philosophy and the Mind Sciences (Forthcoming).
  46. Viktor Moskvoretskii, Alexander Panchenko, and Irina Nikishina. 2024. Are Large Language Models good at lexical semantics? A case of taxonomy learning. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). find
  47. Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations. find
  48. Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie Wang, Clevis Willrich, and Shane Steinert-Threlkeld. 2024. Filtered Corpus Training (FICT) shows that Language Models can Generalize from Indirect Evidence. Transactions of the Association for Computational Linguistics, 12:…
  49. Ellie Pavlick. 2023. Symbols and Grounding in Large Language Models. Philosophical Transactions of the Royal Society A, 381(2251):20220041. find
  50. Steven Piantadosi and Felix Hill. 2022. Meaning without Reference in Large Language Models. In NeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI). find
  51. Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, and Najoung Kim. 2025. Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It. In Advances in Neural Information Processing Systems. find
  52. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In ICML.
  53. Michaela Regneri, Alhassan Abdelhalim, and Soeren Laue. 2024. Detecting Conceptual Abstraction in LLMs. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). find
  54. Gregory Murphy. 2004. The Big Book of Concepts. MIT Press. find
  55. Juan Diego Rodriguez, Aaron Mueller, and Kanishka Misra. 2025. Characterizing the role of similarity in the property inferences of language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologi… find
  56. Gregory L Murphy and Douglas L Medin. 1985. The Role of Theories in Conceptual Coherence. Psychological Review, 92(3):289. find
  57. Stephen Roller and Katrin Erk. 2016. Relations such as Hypernymy: Identifying and Exploiting Hearst patterns in Distributional Vectors for Lexical Entailment. 16330 12 find
  58. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find
  59. Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. 2022. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43):e2200800119. find
  60. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing vision-language model’s perception of the w…
  61. Ziwen Wang, Akshi, Sophia Keil, Judy Kim, and Marina Bedny. 2026. Constructing Meaning from Language: Visual knowledge in People Born Blind and in Large Language Models. Annual Review of Linguistics, 12:In press. find
  62. Sandra R Waxman and Dana B Markow. 1995. Words as Invitations to form Categories: Evidence from 12-to 13-month-old Infants. Cognitive Psychology, 29(3):257–302. find
  63. Kyra Wilson, Michael C Frank, and Abdellah Fourtassi. 2023. Conceptual Hierarchy in Child-directed Speech: Implicit Cues are more Reliable. Journal of Cognition and Development, 24(4):563–580. find
  64. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Huggin…
  65. Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. 2025. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. In The Thirteenth International Conference on Learning Representations. find
  66. Tianyang Xu, Tatsuki Kuribayashi, Yohei Oseki, Ryan Cotterell, and Alex Warstadt. 2025. Can Language Models Learn Typologically Implausible Languages? arXiv preprint arXiv:2502.12317.
  67. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.
  68. Qing Yao, Kanishka Misra, Leonie Weissweiler, and Kyle Mahowald. 2025. Both direct and indirect evidence contribute to dative alternation preferences in language models. In Second Conference on Language Modeling. find
  69. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  70. Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. 2025. Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities. In The Thirteenth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-cross-modal-taxonomic-generalization-vision/