Elamur Research⌕
☰
home / papers / 2023 / Acquiring Linguistic Knowledge from Multimodal Input

Acquiring Linguistic Knowledge from Multimodal Input

Summary

The authors test the hypothesis that language models learn worse than children partly because they receive no visual input. In prior work, multimodal training sometimes even degraded the models' purely linguistic abilities. The authors attribute this to a pretrained language model being further trained on short, formulaic image captions, which caused it to lose more complex language. To avoid this, they train FLAVA from scratch on the Wikipedia-based corpus WiT, where images are paired with both captions and longer article passages. Training is multitask: text-only MLM is retained alongside the multimodal objectives. The amount of text and the number of images are varied independently, and the models are evaluated on grammar (BLiMP), language understanding (GLUE/SuperGLUE) and generalization (MSGS). The result is largely negative. Images do not harm language performance, but they do not yield a consistent gain either. Only at the 10M-word scale is a small advantage on grammatical tests sometimes visible. The work undermines the simple explanation that it is specifically vision that helps humans, and offers an open codebase for further experiments within the BabyLM Challenge.

Key points

  • The experiment covers eight configurations: 10M or 100M words combined with 0, 40K, 400K or 4M images. Each configuration was run only once, and the whole experiment took about 1728 GPU-hours.
  • The data come from the English portion of WiT, which contains roughly 5.5M image–text pairs. The code is published on GitHub (amariucaitheodor/acquiring-linguistic-knowledge).
  • Going from 10M to 100M words raises the GLUE score by about 5%. Adding images at the same amount of text yields no reliable improvement.
  • On MSGS, all models have negative scores. This means they generalize based on surface features rather than linguistic ones.
  • In zero-shot caption retrieval on ImageNet-1k, the trained models perform at chance level: top1 0.1%, top5 0.5%. For comparison, FLAVA pretrained on PMD achieves 32% and 60%. The authors therefore consider it unlikely that cross-situational learning takes place in these models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.conll-babylm.11 · arXiv 2402.17936

@inproceedings{amariucai2023acquiring,
  title = {{Acquiring Linguistic Knowledge from Multimodal Input}},
  author = {Amariucai, Theodor and Warstadt, Alexander Scott},
  booktitle = {Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning},
  year = {2023},
  doi = {10.18653/v1/2023.conll-babylm.11},
  eprint = {2402.17936},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.conll-babylm.11}
}

References

58 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Theodor Amariucai. 2023. Acquiring linguistic knowledge from multimodal input. Master thesis, ETH Zürich, Zürich. find
  2. Lukas Biewald. 2020. weights and biases. wandb.com. find
  3. Experiment tracking with Software available from Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2021. Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language BERTs. Transactions of the Association for Computational Linguistics, 9:97… find
  4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft COCO Captions: Data Collection and Evaluation Server. In European conference on computer vision, pages 740– 755. Springer. ArXiv: 1504.00325. find our card
  5. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: UNiversal Image-TExt Representation Learning. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX, page 104–120, Berlin,… find our card
  6. Grzegorz Chrupała, Àkos Kádár, and Afra Alishahi. 2015. Learning language through pictures. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 112– 118, B… find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  8. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Intern… find
  9. William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning. find
  10. Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9847–9857, Online and Punta Cana, Dominican… find
  11. Jon Gauthier, Roger Levy, and Joshua B. Tenenbaum. 2018. Word learning and the acquisition of syntactic–semantic overhypotheses. In Proceedings of the 40th annual meeting of the cognitive science society, Madison, Wisconsin. find
  12. Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Judith K. Montgomery, Charles R. Greenwood, D. Kim-brough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. Mapping the Early Language Environment Using All-Day Recordings and Automated Analysis. American Journal of Speech-Language Pathology… find
  13. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A Systematic Assessment of Syntactic Generalization in Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… find
  14. Taichi Iki and Akiko Aizawa. 2021. Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2189–2196, Online and Punta Cana, Dominican Republic. Association for Compu… find
  15. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
  16. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find our card
  17. Avinash Madasu and Vasudev Lal. 2023. Is multimodal vision supervision beneficial to language? find
  18. George Kachergis, Chen Yu, and Richard M. Shiffrin. 2014. Cross-situational word learning is both implicit and strategic. Frontiers in Psychology, 5. find
  19. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. International Journ… find our card
  20. Tatsuki Kuribayashi. 2023. Does Vision Accelerate Hierarchical Generalization of Neural Language Learners? ArXiv:2302.00667 [cs]. — [arXiv:2302.00667](https://arxiv.org/abs/2302.00667) find
  21. Angeliki Lazaridou, Marco Marelli, and Marco Baroni. 2017. Multimodal Word Meaning Induction From Minimal Exposure to Natural Text. Cognitive Science, 41(S4):677–705. find
  22. Quentin Lhoest, Albert Villanova del Moral, Patrick von Platen, Thomas Wolf, Mario Šaško, Yacine Jernite, Abhishek Thakur, Lewis Tunstall, Suraj Patil, Mariama Drame, Julien Chaumond, Julien Plu, Joe Davison, Simon Brandeis, Victor Sanh, Teven Le Scao, Kevin Canwen Xu, Nicolas Patry, Steven Liu, Ang… find our card
  23. Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020. Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training. Proceedings of the AAAI Conference on Artificial Intelligence, 34:11336–11344. find
  24. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. find our card
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenbaum, and Jiajun Wu. 2019. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences from Natural Supervision. In Proceedings of IC… find
  26. Rebecca Marvin and Tal Linzen. 2018. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202. find
  27. Michael McCloskey and Neal J. Cohen. 1989. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. In Psychology of Learning and Motivation, volume 24, pages 109–165. Elsevier. find
  28. R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 217–2… find
  29. Mitja Nikolaus and Abdellah Fourtassi. 2021. Evaluating the acquisition of semantic knowledge from cross-situational learning in artificial neural networks. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 200–210, Online. Association for Computational Lingui… find
  30. Eva Portelance, Michael C. Frank, and Dan Jurafsky. 2023. Learning the meanings of function words from grounded language using a visual question answering model. ArXiv:2308.08628 [cs]. find
  31. Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik Shah, Yann LeCun, and Ramalingam Chellappa. 2022. VoLTA: Vision-language transformer with weakly-supervised local-feature alignment. ArXiv, abs/2210.04135. — [arXiv:2210.04135](https://arxiv.org/abs/2210.04135) find
  32. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio… find our card
  33. Deb K Roy and Alex P Pentland. 2002. Learning words from sights and sounds: a computational model. Cognitive Science, 26(1):113–146. find
  34. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–25… find our card
  35. Hassan Shahmohammadi, Maria Heitmeier, Elnaz Shafaei-Bajestan, Hendrik P. A. Lensch, and Harald Baayen. 2022. Language with vision: a study on grounded word and sentence embeddings. find
  36. A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela. 2022. FLAVA: A foundational language and vision alignment model. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15617–15629, Los Alamitos, CA, USA. IEEE Computer Society. find
  37. Andrew D. M. Smith and Kenny Smith. 2012. Cross-Situational Learning, pages 864–866. Springer US, Boston, MA. find
  38. Kenny Smith, Andrew D. M. Smith, and Richard A. Blythe. 2011. Cross-Situational Learning: An Experimental Study of Word-Learning Mechanisms. Cognitive Science, 35(3):480–498. find
  39. Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’2… find
  40. Jessica Sullivan, Michelle Mei, Andrew Perfors, Erica Wojcik, and Michael C. Frank. 2021. SAYCam: A Large, Longitudinal Audiovisual Dataset Recorded From the Infant’s Perspective. Open Mind, 5:20–29. find
  41. Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–51… find our card
  42. Wai Keen Vong and Brenden M. Lake. 2022. Cross-Situational Word Learning With Multimodal Neural Networks. Cognitive Science, 46(4):e13122. find
  43. Alex Wang and Kyunghyun Cho. 2019. BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30–36, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  44. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In 33rd Conference on Neural Information Processing Systems. Journal Abbreviation: arX… find our card
  45. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  46. Ruocheng Wang, Jiayuan Mao, Samuel Gershman, and Jiajun Wu. 2021. Language-Mediated, Object-Centric Representation Learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2033–2046, Online. Association for Computational Linguistics. find
  47. Wentao Wang, Wai Keen Vong, Najoung Kim, and Brenden M. Lake. 2023. Finding Structure in One Child’s Linguistic Experience. Cognitive Science, 47(6):e13305. find
  48. Alex Warstadt and Samuel R Bowman. 2022. What artificial neural networks can tell us about human language acquisition. In Shalom Lappin and Jean-Philippe Bernardy, editors, Algebraic Structures in Natural Language, pages 17–60. CRC Press. Publisher: CRC Press. find
  49. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Gotlieb Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Adina Williams, Bhargavi Paranjabe, Tal Linzen, and Ryan Cotterell. 2023. Findings of the 2023 BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proce… find
  50. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020a. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. _eprint: https://doi.org/10.1162/tacl_a_00321. ↗
  51. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020b. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 217–23… find
  52. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… find our card
  53. Yiqun Yao and Rada Mihalcea. 2022. Modality-specific learning rates for effective multimodal additive late-fusion. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1824–1834, Dublin, Ireland. Association for Computational Linguistics. find
  54. Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3208– 3216. find
  55. Tian Yun, Chen Sun, and Ellie Pavlick. 2021. Does Vision-and-Language Pretraining Improve Lexical Grounding? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4357– 4366, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  56. Yayun Zhang, Chi hsin Chen, and Chen Yu. 2019. Chapter Two - Mechanisms of Cross-situational Learning: Behavioral and Computational Evidence. volume 56 of Advances in Child Development and Behavior, pages 37–63. JAI. find
  57. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… find
  58. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified Vision-Language Pre-Training for Image Captioning and VQA. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):13041–13049. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/amariucai-2023-acquiring-linguistic-knowledge-multimodal-input/