Elamur Research⌕
☰
home / papers / 2021 / Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

Paper · 2021 · Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich et al.

Summary

It is an open question whether language models trained only on text capture structure grounded in human perception, rather than only relational facts. The authors test this on color, a domain with a perceptually uniform reference space. They use the Color Lexicon of American English, which pairs Munsell chips with free-naming judgments, and place each color in CIELAB. They then build word-type embeddings for color terms from BERT, RoBERTa and ELECTRA under three extraction setups: no context, random Common Crawl sentences, and templated controlled sentences. PMI vectors and FastText serve as baselines. Two methods measure how well the spaces align: Representation Similarity Analysis and a learned linear mapping. Both reveal significant correspondence. Even PMI statistics carry some signal, but large pretrained LMs match the perceptual geometry noticeably better, and bigger models beat smaller ones. Warm colors align better than cool ones, which echoes research on how efficiently colors are communicated. Corpus analysis links weaker alignment to collocational use, and stronger alignment to varied syntactic roles and frequent adjectival use. The work offers empirical evidence that some perceptual topology can be recovered from text alone.

Key points

  • Color data: The lexicon (Lindsey and Brown, 2014) covers 330 Munsell chips, each named by 51 subjects, yielding 122 color terms. After a frequency cutoff of f = 100, the analysis keeps the 18 most common terms. Their CIELAB centroids are compared with the text embeddings.
  • Models and baselines:
    • The LMs are bert-large-uncased, roberta-large and electra-large-discriminator, probed layer by layer.
    • The baselines are PMI vectors from Common Crawl (windows 1–3: pmi-1, pmi-2, pmi-3) and FastText.
  • Controlled-context set: Three frames (COPULA, POSSESSION, SPATIAL) are combined with object nouns from McRae et al. (2005). This produces 6588 sentences, 366 per term.
  • Evaluation and main findings:
    • RSA uses Kendall's τ between similarity matrices, complemented by a linear mapping into CIELAB.
    • Larger LMs align more closely, and low-dimensional subspaces of the embeddings explain much of the CIELAB variance.
    • Higher color chip surprisal is associated with lower alignment.
  • Regression analysis:
    • Collocationality (pmi-col) predicts lower alignment; examples are metaphorical uses such as Red Army.
    • Dependency-relation entropy and the share of amod uses (UD relations) predict higher alignment.
    • The proportions of adjective POS tags and copula uses are not significant.
    • In preliminary tests, VisualBERT and VideoBERT showed no major gains over text-only models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brown UniversityProvidence · United States58 works · 25,310 citations🎓University of CopenhagenCopenhagen · Denmark78 works · 14,228 citations🎓University of TrentoTrento · Italy62 works · 14,868 citations🎓Uppsala UniversityUppsala · Sweden56 works · 13,046 citations

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksPhysics and Astronomy · 25 worksAtomic and Molecular Physics, and Optics · 2 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick and Anders Søgaard. 2021. Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color. Proceedings of the 25th Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.conll-1.9

@inproceedings{abdou2021can,
  title = {{Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color}},
  author = {Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick and Anders Søgaard},
  booktitle = {Proceedings of the 25th Conference on Computational Natural Language Learning},
  year = {2021},
  doi = {10.18653/v1/2021.conll-1.9},
  url = {https://doi.org/10.18653/v1/2021.conll-1.9}
}

References

60 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches.
  2. Yonatan Belinkov. 2021. Probing classifiers: Promises, shortcomings, and alternatives. arXiv preprint arXiv:2102.12452. — [arXiv:2102.12452](https://arxiv.org/abs/2102.12452) find
  3. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics.
  4. Brent Berlin and Paul Kay. 1991. Basic color terms: Their universality and evolution. Univ of California Press. find
  5. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020. Experience grounds language. arXiv preprint arXiv:2004.10151.
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.
  7. Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758– 4781, Online. Association for Computational Lingu…
  8. Klaus Brinker. 2003. Incorporating diversity in active learning with support vector machines. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 59–66. find
  9. Robert J Ennis and Qasim Zaidi. 2019. Geometrical structure of perceptual color space: mental representations and adaptation invariance. Journal of vision, 19(12):1–1. find
  10. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 136–145.
  11. Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019a. Do neural language representations learn physical commonsense? arXiv preprint arXiv:1908.02899. find
  12. Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. 2021. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences, 118(12). find
  13. M Chastrette. 1997. Trends in structure-odor relationship. SAR and QSAR in Environmental Research, 6(3-4):215–254. find
  14. Lin Chen. 1982. Topological structure in visual perception. Science, 218(4573):699–700. find
  15. Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? when it’s like a rabbi! multi-prototype BERT embeddings for estimating semantic relationships. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 227–244, Online. Association for Computatio… find
  16. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. find
  17. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre-trained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–…
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  19. Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4865–4880, Online. Association for Computational Linguistics. find
  20. Maxwell Forbes, Christine Kaeser-Chen, Piyush Sharma, and Serge Belongie. 2019b. Neural naturalist: Generating fine-grained image comparisons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc… find
  21. Andrzej Gałecki and Tomasz Burzykowski. 2013. Linear mixed-effects model. In Linear Mixed-Effects Models Using R, pages 245–273. Springer. find
  22. Edward Gibson, Richard Futrell, Julian Jara-Ettinger, Kyle Mahowald, Leon Bergen, Sivalogeswaran Ratnasingam, Mitchell Gibson, Steven T Piantadosi, and Bevil R Conway. 2017. Color naming across languages reflects color use. Proceedings of the National Academy of Sciences, 114(40):10785–10790. find
  23. Evan Hernandez and Jacob Andreas. 2021. The low-dimensional linear geometry of contextualized word representations. arXiv preprint arXiv:2105.07109. find
  24. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. arXiv preprint arXiv:1909.03368. find
  25. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695.
  26. David Hume. 1938. An Abstract of a Treatise of Human Nature, 1740. CUP Archive. find
  27. Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. 2020. Probing text models for common ground with visual representations. arXiv preprint arXiv:2005.00619.
  28. Frank Jackson. 1982. Epiphenomenal qualia. The Philosophical Quarterly (1950-), 32(127):127–136. find
  29. Paul Kay, Brent Berlin, Luisa Maffi, William R Merrifield, and Richard Cook. 2009. The world color survey. CSLI Publications Stanford, CA. find
  30. Paul Kay, Brent Berlin, and William Merrifield. 1991. Biocultural implications of systems of color naming. Journal of Linguistic Anthropology, 1(1):12–25. find
  31. Paul Kay and Chad K McDaniel. 1978. The linguistic significance of the meanings of basic color terms. Language, pages 610–646. find
  32. Judy Sein Kim, Brianna Aheimer, Verónica Montané Manrara, and Marina Bedny. 2020. Shared understanding of color among congenitally blind and sighted adults. find
  33. Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020. Pareto probing: Trading off accuracy for complexity. arXiv preprint arXiv:2010.02180. find
  34. Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4. find
  35. Edoardo Provenzi. 2020. Geometry of color perception. part 1: structures and metrics of a homogeneous color space. The Journal of Mathematical Neuroscience, 10(1):1–19. find
  36. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
  37. Terry Regier, Paul Kay, and Naveen Khetarpal. 2007. Color naming reflects optimal partitions of color space. Proceedings of the National Academy of Sciences, 104(4):1436–1441. find
  38. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019a. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  39. Delwin T Lindsey and Angela M Brown. 2014. The color lexicon of american english. Journal of vision, 14(2):17–17. find
  40. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  41. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019b. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32:8594–8603.
  42. John Locke. 1847. An essay concerning human understanding. Kay & Troutman. find
  43. Karen J Rossiter. 1996. Structure- odor relationships. Chemical reviews, 96(8):3201–3240. find
  44. Li Lucy and Jon Gauthier. 2017. Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning. arXiv preprint arXiv:1705.11168. find
  45. Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rappoport. 2015. How well do distributional models capture different types of semantic knowledge? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Languag… find
  46. Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4):547–559. find
  47. William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A Smith. 2021. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? arXiv preprint arXiv:2104.10809.
  48. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluat…
  49. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  50. Armin Saysani, Michael C Corballis, and Paul M Corballis. 2018. Colour envisioned: Concepts of colour in the blind and sighted. Visual Cognition, 26(5):382–392. find
  51. Hinrich Schütze. 1992. Dimensions of meaning. In SC, pages 787–796. find
  52. Vered Shwartz and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863–6870, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  53. Milan Straka, Jan Hajic, and Jana Straková. 2016. Udpipe: trainable pipeline for processing conll-u files performing tokenization, morphological analysis, pos tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4290– 4297.
  54. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464–7473.
  55. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
  56. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962.
  57. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. arXiv preprint arXiv:2010.05731.
  58. Nathaniel Weir, Adam Poliak, and Benjamin Van Durme. 2020. Probing neural language models for human tacit assumptions. find
  59. Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. 2015. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113(2):113–127. find
  60. Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. 2018. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31):7937–7942. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdou-2021-can-language-models-encode-perceptual/