Elamur Research⌕
☰
home / papers / 2021 / Probing for idiomaticity in vector space models

Probing for idiomaticity in vector space models

Summary

Do vector space models actually encode the idiomatic meaning of multiword expressions? Garcia et al. study this for noun compounds (NCs), whose meanings range from transparent (graduate student) to opaque (eager beaver). Drawing on the semantic priming paradigm from psycholinguistics, they design four probing measures. Each compares cosine similarities between embeddings of an NC out of context, an NC inside a sentence, and the whole sentence. These similarities are then correlated (Spearman ρ) with human idiomaticity ratings. To support the probes, the authors build the Noun Compound Senses (NCS) dataset for English and Portuguese, derived from the NC Compositionality dataset. In NCS, each compound is replaced by a synonym, by synonyms of its components, or by its head or modifier alone. These variants appear in both naturalistic corpus sentences and uninformative template sentences. Testing GloVe, ELMo and BERT-based models without fine-tuning, the authors find that contextualised models largely miss that idiomatic compounds resist component substitution, and that context changes little. The work provides a controlled benchmark for idiomaticity-aware representations.

Key points

  • NCS covers 280 English and 180 Portuguese NCs, totalling 9,220 sentences (5,620 EN, 3,600 PT). Neutral sentences follow the template "This is a/an <NC>" (Portuguese: "Este/a é um(a) <NC>"). The dataset is freely available on GitHub (marcospln/noun_compound_senses).
  • Five models were compared: the static GloVe and four contextualised ones (ELMo, BERT, DistilBERT, SBERT). Component synonyms came from WordNet and from synonym dictionaries, and the variants were reviewed by experts.
  • Naturalistic and neutral contexts gave strongly correlated results. For example, SBERT's in-context NC similarities correlated at ρ > 0.85 (EN) and > 0.76 (PT) across probes P1–P3. Sentence length also correlated moderately to strongly with the similarity scores, which inflated sentence-level similarities.
  • Learning single-token NC vectors via attentive mimicking and injecting them into multilingual BERT with BERTRAM improved the probes. For instance, the P2 correlation on naturalistic sentences reached ρ = 0.45 in English, but this approach requires retraining the model.
  • Concatenating subword vectors instead of averaging them raised BERT-based correlations slightly (about 0.06 on P3 for 273 English sentences).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

51 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 39 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.310

@inproceedings{garcia2021probing,
  title = {{Probing for idiomaticity in vector space models}},
  author = {Garcia, Marcos and Vieira, Tiago Kramer and Scarton, Carolina and Idiart, Marco and Villavicencio, Aline},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.310},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.310}
}

References

45 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Aina, Kristina Gulordava, and Gemma Boleda. 2019. Putting words in context: LSTM language models and lexical ambiguity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3342–3348, Florence, Italy. Association for Computational Linguistics. find
  2. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638–1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  3. Timothy Baldwin, Colin Bannard, Takaaki Tanaka, and Dominic Widdows. 2003. An empirical model of multiword expression decomposability. In Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment, pages 89–96, Sapporo, Japan. Association for Computational Lin… find our card
  4. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Language resources and evaluation, 43(3):209–226. find
  5. Pedro Vitor Quinta de Castro, Nádia Félix Felipe da Silva, and Anderson da Silva Soares. 2018. Portuguese Named Entity Recognition Using LSTM-CRF. In Proceedings of the 13th International Conference on the Computational Processing of the Portuguese Language (PROPOR 2018), pages 83–92, Canela–RS, B… find
  6. Ting-Yun Chang and Yun-Nung Chen. 2019. What does this word mean? explaining contextualized embeddings with natural language definition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  7. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find our card
  8. Silvio Cordeiro, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019. Unsupervised compositionality prediction of nominal compounds. Computational Linguistics, 45(1):1–57. find
  9. David Alan Cruse. 1986. Lexical semantics. Cambridge university press. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  11. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  12. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. find our card
  13. Meghdad Farahmand and James Henderson. 2016. Modeling the non-substitutability of multiword expressions with distributional semantics and a log-linear model. In Proceedings of the 12th Workshop on Multiword Expressions, pages 61–66, Berlin, Germany. Association for Computational Linguistics. find
  14. Afsaneh Fazly, Paul Cook, and Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. Computational Linguistics, 35(1):61–103. find our card
  15. Christina L Gagné and Thomas L Spalding. 2009. Constituent integration during the processing of compound words: Does it involve the use of relational structures? Journal of Memory and Language, 60(1):20–35. find
  16. Rachel Giora. 1999. On the priority of salient meanings: Studies of literal and figurative language. Journal of pragmatics, 31(7):919–929. find
  17. Yoav Goldberg. 2019. Assessing BERT’s Syntactic Abilities. arXiv preprint arXiv:1901.05287. find our card
  18. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find our card
  19. Nathan Hartmann, Erick Fonseca, Christopher Shulby, Marcos Treviso, Jéssica Silva, and Sandra Aluı́sio. 2017. Portuguese word embeddings: Evaluating on word analogies and natural language tasks. In Proceedings of the 11th Brazilian Symposium in Information and Human Language Technology, pages 122–1… find
  20. Hongbo Ji, Christina L Gagné, and Thomas L Spalding. 2011. Benefits and costs of lexical decomposition and semantic integration during the processing of transparent and opaque English compounds. Journal of Memory and Language, 65(4):406–430. find
  21. Milton King and Paul Cook. 2017. Supervised and unsupervised approaches to measuring usage similarity. In Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and their Applications, pages 47–52, Valencia, Spain. Association for Computational Linguistics. find
  22. Milton King and Paul Cook. 2018. Leveraging distributed representations and lexico-syntactic fixedness for token-level prediction of the idiomaticity of english verb-noun combinations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Au… find
  23. Jean-Paul Laurent, Guy Denhières, Christine Passerieux, Galina Iakimova, and Marie-Christine Hardy-Baylé. 2006. On understanding idiomatic language: The salience hypothesis assessed by ERPs. Brain Research, 1068(1):151–160. find
  24. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. find our card
  25. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find our card
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems -Volume 2, NIPS’13, pages 3111–3119, USA. Curran As… find
  27. George A. Miller. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11):39–41. find our card
  28. Navnita Nandakumar, Timothy Baldwin, and Bahar Salehi. 2019. How Well Do Embedding Models Capture Non-compositionality? A View from Multiword Expressions. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 27–34, Minneapolis, USA. Association for Computation… find
  29. James H. Neely, Dennis E. Keefe, and Kent L. Ross. 1989. Semantic priming in the lexical decision task: Roles of prospective prime-generated expectancies and retrospective semantic matching. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15(6):1003–1019. find
  30. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  31. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  32. Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 66–76, Hong Kong, China. Association for Comput… find
  33. Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Fifth International Joint Conference on Natural Language Processing, IJCNLP 2011, Chiang Mai, Thailand, November 8-13, 2011, pages 210–218. The Association for Computer Linguistics. find our card
  34. Nils Reimers and Iryna Gurevych. 2019a. Alternative Weighting Schemes for ELMo Embeddings. CoRR, abs/1904.02954. — [arXiv:1904.02954](https://arxiv.org/abs/1904.02954) find
  35. Nils Reimers and Iryna Gurevych. 2019b. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  36. Joost Rommers, Ton Dijkstra, and Marcel Bastiaansen. 2013. Context-dependent semantic processing in the human brain: Evidence from idiom comprehension. Journal of Cognitive Neuroscience, 25(5):762–776. find
  37. Andreas Rücklé, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. 2018. Concatenated power mean word embeddings as universal cross-lingual sentence representations. arXiv preprint arXiv:1803.01400. — [arXiv:1803.01400](https://arxiv.org/abs/1803.01400) find
  38. Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2002), pages 1–15, Mexico City, Mexico. S… find our card
  39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing at NeurIPS 2019, Vancouver, Canada. find
  40. Timo Schick and Hinrich Schütze. 2019. Attentive mimicking: Better word embeddings by attending to informative contexts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers… find
  41. Timo Schick and Hinrich Schütze. 2020a. BERTRAM: Improved word embeddings have big impact on contextualized model performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3996–4007, Online. Association for Computational Linguistics. find
  42. Timo Schick and Hinrich Schütze. 2020b. Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 8766–8774. find
  43. Marten van Schijndel and Tal Linzen. 2018. A neural model of adaptation in reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4704–4710, Brussels, Belgium. Association for Computational Linguistics. find
  44. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find
  45. Weijia Shi, Muhao Chen, Pei Zhou, and Kai-Wei Chang. 2019. Retrofitting contextualized word embeddings with paraphrases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/garcia-2021-probing-idiomaticity-vector-space-models/