Elamur Research⌕
☰
home / papers / 2026 / Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding

Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding

Paper · 2026 · Ayaan Siddiqui

Summary

Vision–language models such as CLIP and BLIP-2 handle literal image–text matching well, but they perform poorly when an image must be matched to the non-literal sense of an idiom, metaphor or simile. They usually pick the image that depicts the words literally. The author argues this comes from a systematic preference for literal matches, not from too little model capacity. The task is therefore recast as contrastive "semantic deviation", in which the model is trained to score the figurative image higher than a literal distractor that looks visually plausible. To do this, CLIP is fine-tuned with LoRA on triples of a phrase, its figurative image and a mined hard literal negative.

On the IRFL benchmark, this beats zero-shot CLIP and BLIP-2, weighted and MLP ensembles of CLIP backbones, and ConceptNet-enriched text. The paper also releases FIGMENT, a multilingual figurative grounding test in five languages. Although trained only on English data, the adapted model transfers well to these languages. This suggests that lightweight, targeted adaptation can noticeably improve non-literal multimodal reasoning.

Key points

  • On the IRFL "mixed" task, LoRA-CLIP improves on zero-shot CLIP for all three categories:
    • idioms: about 72% vs 22%
    • metaphors: 48% vs 30%
    • similes: 62% vs 44%
    • Humans score 99.7%, so a gap remains.
  • Common alternatives give little benefit:
    • Ensembles of ViT-B/32, ViT-B/16, RN50 and RN50×64 fail to consistently beat the strongest single backbone (RN50×64).
    • ConceptNet augmentation of the text also adds little.
  • FIGMENT covers French, Chinese, Spanish, Hindi and Arabic.
    • The adapted model reaches 68.7–84.6% accuracy.
    • Zero-shot mCLIP reaches only 4.1–28.8%.
    • Literal bias falls to 15–31%, compared with up to 96% for mCLIP.
  • The zero-shot models show near-zero or negative similarity margins between figurative and literal images. The adapted model shows clearly positive margins, which line up with its lower literal bias.
  • Adapting the higher-level visual attention layers gives the largest gains. Text-side adaptation helps mainly for idioms when definitions are available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birla Institute of Technology and Science - Hyderabad CampusHyderabad · India2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksIdioms & multiword expressions · 217 works

Cite

Cite as Ayaan Siddiqui. 2026. Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.12

@inproceedings{siddiqui2026semantic,
  title = {{Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding}},
  author = {Ayaan Siddiqui},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.12},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.12}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. 2025. Vision-language models do not understand negation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). find
  2. Julia Birke and Anoop Sarkar. 2006. A clustering approach for nearly unsupervised recognition of nonliteral language. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics. find
  3. Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, and Wenping Wang. 2023. mclip: Multilingual clip via cross-lingual transfer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. A… find
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML).
  5. Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  6. Dan Fass. 1991. met*: A method for discriminating metonymy and metaphor by computer. Computational Linguistics, 17:49–90.
  7. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
  8. Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed (Ehsan) Hoque. 2019. UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Proces… find
  9. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, et al. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML). find
  10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR).
  11. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision.
  12. Hui Li, Peng Wang, and Chunhua Shen. 2020. Scene text recognition from two-dimensional perspective. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  13. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. ICML’23. JMLR.org.
  14. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. find
  15. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR.
  16. Ron Yosef, Yonatan Bitton, and Dafna Shahaf. 2023. IRFL: Image recognition of figurative language. In Findings of the Association for Computational Linguistics: EMNLP 2023. find
  17. Siting Li, Pang Wei Koh, and Simon Shaolei Du. 2024. On erroneous agreements of CLIP image embeddings. arXiv preprint arXiv:2411.05195. — [arXiv:2411.05195](https://arxiv.org/abs/2411.05195) find
  18. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL). find
  19. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. find
  20. Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. 2023. Filtering, distillation, and hard negatives for vision–language pre-training. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition… find
  21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International C…
  22. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2025. Understanding figurative meaning through explainable visual entailment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL).
  23. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  24. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. Laion-5b: an open larg…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/siddiqui-2026-semantic-contrastive-adaptation-multimodal-figurative/