Elamur Research⌕
☰
home / papers / 2021 / Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation

Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation

Paper · 2021 · Sandro Pezzelle, Ece Takmaz and Raquel Fernández

Summary

Pre-trained vision-and-language Transformers are said to learn general-purpose word meanings, but it was unclear how well these meanings match human intuitions. The authors test this directly, without relying on downstream task scores. They build a multimodal dataset, VICO, made of image–caption samples from VIST and COCO, chosen to cover as many words as possible from five human similarity benchmarks. Following Bommasani et al. (2020), they pool contextualized embeddings over many contexts to get static word vectors. They do this for LXMERT, UNITER, ViLBERT, VisualBERT, Vokenization, and the text-only baseline BERT. They then correlate model similarities with human ratings, layer by layer. Every multimodal model beats BERT on concrete word pairs but gains nothing on abstract ones. Vokenization is the most robust overall, which the authors attribute to its token-level visual supervision. Layer-wise trends show that the models combine the two modalities in different ways and at different depths. The study supplies a simple, interpretable check of grounding in multimodal Transformers.

Key points

  • Five benchmarks are used: RG65, WordSim353, SimLex999, MEN, and SimVerb3500. Together they contain 7917 word pairs over 2453 unique words, covering nouns, verbs, and adjectives.
  • Six models are compared: LXMERT, UNITER, ViLBERT, VisualBERT, and Vokenization against BERT. Static vectors are obtained by aggregating contextual embeddings from the VICO samples.
  • The advantage of visual grounding appears for concrete pairs, as in MEN, but not for abstract ones. On MEN, pairs that only the multimodal models get right (0.09) slightly outnumber BERT-only pairs (0.08). SimLex999 shows the opposite pattern (0.05 vs 0.10).
  • Example: visual context pushes the concrete pair bakery–zebra apart, matching human judgments. It also separates creator–maker, which humans judge highly similar.
  • Data and code are released at github.com/sandropezzelle/multimodal-evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Sandro Pezzelle, Ece Takmaz and Raquel Fernández. 2021. Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00443

@article{pezzelle2021word,
  title = {{Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation}},
  author = {Sandro Pezzelle, Ece Takmaz and Raquel Fernández},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00443},
  url = {https://doi.org/10.1162/tacl_a_00443}
}

References

64 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6077–6086. https://… — [doi:10.1109/cvpr.2018.00636](https://doi.org/10.1109/cvpr.2018.00636) find
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425–2433. https://doi.org/10 .1109/ICCV.2015.279 find
  3. Marco Baroni. 2016. Grounding distributional semantics in the visual world. Language and Linguistics Compass, 10(1):3–13. https://doi.org/10.1111/lnc3.12170 — [doi:10.1111/lnc3.12170](https://doi.org/10.1111/lnc3.12170) ↗
  4. Lawrence W. Barsalou. 2008. Grounded cognition. Annual Review of Psychology, 59:617–645. https://doi.org/10.1146/annurev .psych.59.103006.093639, PubMed: 17705682 — [doi:10.1146/annurev](https://doi.org/10.1146/annurev) ↗
  5. Lisa Beinborn, Teresa Botschen, and Iryna Gurevych. 2018. Multimodal grounding for language processing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2325–2339. Association for Computational Linguistics. find
  6. Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting pretrained contextualized representations via reductions to static embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758–4781. https://doi .org/10.18653/v1/2020.acl-main.4…
  7. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. https://doi.org/10.1613/jair .4135 ↗
  8. Elia Bruni, Jasper Uijlings, Marco Baroni, and Nicu Sebe. 2012. Distributional semantics with eyes: Using image analysis to improve computational representations of word meaning. In Proceedings of the 20th ACM International Conference on Multimedia, pages 1219–1228. https://doi.org/10.1145/2393347 .… — [doi:10.1145/2393347](https://doi.org/10.1145/2393347) ↗
  9. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46(3):904–911. https://doi.org/10.3758/s13428-013-0403-5, PubMed: 24142837 ↗
  10. Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2021. Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language BERTs. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00408 ↗
  11. Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. 2020. Behind the scene: Revealing the secrets of pre-trained vision-and-language models. In European Conference on Computer Vision, pages 565–580. Springer. https://doi.org/10.1007/978-3-030-58539-6 34 — [doi:10.1007/978-3-030-58539-6](https://doi.org/10.1007/978-3-030-58539-6) ↗
  12. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: Universal image-text representation learning. In European Conference on Computer Vision, pages 104–120. Springer. https://doi.org/10.1007/978-3-030-58577-8 7 ↗
  13. Guillem Collell Talleda and Marie-Francine Moens. 2016. Is an image worth more than a thousand words? On the fine-grain semantic differences between visual and linguistic representations. In Proceedings of the 26th International Conference on Computational Linguistics, pages 2807–2817. ACL. find
  14. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  15. Christopher Davis, Luana Bulat, Anita Lilla Verő, and Ekaterina Shutova. 2019. Deconstructing multimodality: Visual properties and visual context in human semantic processing. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019), pages 118–124. find
  16. Manuel De Vega, Arthur Glenberg, and Arthur Graesser. 2012. Symbols and Embodiment: Debates on Meaning and Cognition, Oxford University Press. find
  17. Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. GuessWhat?! Visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5503–5512. https://doi.org/10 .1109/CV… find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  19. Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. In Proceedings of the 1st Work shop on Evaluating Vector-Space Representations for NLP, pages 30–35. https://doi.org/10.18653/v1/W16-2506 ↗
  20. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. https://doi.org/10.1145/503104.503110 ↗
  21. John R. Firth. 1957. A synopsis of linguistic theory, 1930–1955. Studies in Linguistic Analysis. find
  22. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2173–2182. https://doi.org/10 .18653/v1/D16-1235
  23. Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346. https://doi.org/10 .1016/0167-2789(90)90087-6 find
  24. Zellig S. Harris. 1954. Distributional structure. Word, 10(2-3):146–162. https://doi.org/10.1080/00437956.1954.11659520 ↗
  25. Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, and Aida Nematzadeh. 2021. Decoupling the role of data, attention, and losses in multimodal Transformers. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00385 ↗
  26. Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language Transformers for verb understanding. arXiv preprint arXiv: 2106.09141. https://doi.org/10.18653/v1/2021.findings-acl.318 ↗
  27. Felix Hill and Anna Korhonen. 2014. Learning abstract concept embeddings from multi-modal data: Since you probably can’t see what I mean. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 255–265. https://doi.org/10.3115/v1/D14-1032 ↗
  28. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. https://doi.org/10.1162/COLI a 00237 ↗
  29. Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, and et al.. 2016. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational…
  30. Satwik Kottur, Ramakrishna Vedantam, José MF Moura, and Devi Parikh. 2016. Visual word2vec (vis-w2v): Learning visually grounded word embeddings using abstract scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4985–4994. https://doi.org/10 .1109/CVPR.20… find
  31. Thomas K. Landauer and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211. https://doi.org/10.1037/0033-295X.104.2.211 — [doi:10.1037/0033-295x.104.2.211](https://doi.org/10.1037/0033-295x.104.2.211) ↗
  32. Juan J. Lastra-Dı́az, Josu Goikoetxea, Mohamed Ali Hadj Taieb, Ana Garcı́a-Serrano, Mohamed Ben Aouicha, and Eneko Agirre. 2019. A reproducible survey on word embeddings and ontology-based methods for word similarity: linear combinations outperform the state of the art. Engineering Applications of A… — [doi:10.1016/j.engappai.2019.07.010](https://doi.org/10.1016/j.engappai.2019.07.010) find
  33. Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 153–163. https://doi .…
  34. Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. 2021. Probing contextual language models for common ground with visual representations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  35. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
  36. Douwe Kiela and Léon Bottou. 2014. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 36–45. https://doi.org/10.3115/v1/D14-1005 ↗
  37. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265–5275. https://doi.org/10 .18653/v1/2020.acl-main.469 find
  38. Douwe Kiela, Anita Lilla Verő, and Stephen Clark. 2016. Comparing data sources and architectures for deep visual representation learning Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. find
  39. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer. https://doi.org/10.1007/978-3-319-10602-1 48 ↗
  40. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  41. Timo Lüddecke, Alejandro Agostini, Michael Fauth, Minija Tamosiunaite, and Florentin Wörgötter. 2019. Distributional semantics of objects in visual scenes in comparison to text. Artificial Intelligence, 274:44–65. https://doi.org/10.1016/j.artint.2018.12.009 https://doi.org/10.1017/S135132491 90… — [doi:10.1016/j.artint.2018.12.009](https://doi.org/10.1016/j.artint.2018.12.009) ↗
  42. Letitia Parcalabescu, Albert Gatt, Anette Frank, and Iacer Calixto. 2021. Seeing past words: Testing the cross-modal capabilities of pre-trained V&L models on counting tasks. In Proceedings of the ‘Beyond Language: Multimodal Semantic Representations’ Workshop. find
  43. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. https://doi.org/10 .3115/v1/D14-1162
  44. Chunpeng Ma, Aili Shen, Hiyori Yoshikawa, Tomoya Iwakura, Daniel Beck, and Timothy Baldwin. 2021. On the (in)effectiveness of images for text classification. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 42–48. find
  45. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  46. Lotte Meteyard, Sara Rodriguez Cuadrado, Bahador Bahrami, and Gabriella Vigliocco. 2012. Coming of age: A review of embodiment and the neuroscience of semantics. Cortex, 48(7):788–804. https://doi.org/10.1016/j.cortex.2010.11.002, PubMed: 21163473 ↗
  47. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. https://doi.org/10.1162/tacl_a_00349 ↗
  48. Timothee Mickus, Mathieu Constant, Denis Paperno, and Kees van Deemter. 2020. What do you mean, BERT? Assessing BERT as a Distributional Semantics Model. Proceedings of the Society for Computation in Linguistics, volume 3. find
  49. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2–4, 2013, Workshop Track Proceedings.
  50. Roberto Navigli and Federico Martelli. 2019. An overview of word and sense similarity. Natural Language Engineering, 25(6):693–714. find
  51. Armand S. Rotaru and Gabriella Vigliocco. 2020. Constructing semantic models from words, images, and emojis. Cognitive Science, 44(4):e12830. https://doi.org/10.1111/cogs.12830, PubMed: 32237093 ↗
  52. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. https://doi.org/10.1145/365628.365657 — [doi:10.1145/365628.365657](https://doi.org/10.1145/365628.365657) ↗
  53. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565.…
  54. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 721–732. https://doi.org/10.3115/v1/P14-1068 ↗
  55. Amanpreet Singh, Vedanuj Goswami, and Devi Parikh. 2020. Are we pretraining it right? Digging deeper into visio-linguistic pretraining. arXiv preprint arXiv:2004.08744. find
  56. Mohamed Ali Hadj Taieb, Torsten Zesch, and Mohamed Ben Aouicha. 2020. A survey of semantic relatedness evaluation datasets and procedures. Artificial Intelligence Review, 53(6):4407–4448. https://doi.org/10 .1007/s10462-019-09796-3 find
  57. Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–51…
  58. Hao Tan and Mohit Bansal. 2020. Vokenization: Improving language understanding via contextualized, visually-grounded supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2066–2080. https://doi.org/10.18653/v1/2020.emnlp-main.162 ↗
  59. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601. https://doi.org/10.18653/v1/P19-1452 ↗
  60. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. https://doi.org/10.1613/jair.2934 ↗
  61. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019. https://doi.org/10.18653/v1/W18 -5446 ↗
  62. Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2018. Learning multimodal word representation via dynamic fusion methods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  63. Matthijs Westera and Gemma Boleda. 2019. Don’t blame distributional semantics if it can’t do entailment. In Proceedings of the 13th International Conference on Computational Semantics-Long Papers, pages 120–133. https://doi.org/10.18653/v1/W19-0410 ↗
  64. Eloi Zablocki, Benjamin Piwowarski, Laure Soulier, and Patrick Gallinari. 2018. Learning multi-modal word representation grounded in visual context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pezzelle-2021-word-representation-learning-multimodal-pre/