Elamur Research
☰
home / papers / 2020 / Probing Pretrained Language Models for Lexical Semantics

Probing Pretrained Language Models for Lexical Semantics

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.586.pdf

Summary

The paper asks how much type-level (context-independent) lexical knowledge pretrained Transformer LMs such as BERT contain, and how best to extract it. The authors turn contextual encoders into static "BERT-based" word vectors and test them on five lexical tasks: LSIM, WA, BLI, CLIR, and RELP. The experiments cover six typologically varied languages (EN, DE, RU, FI, ZH, TR). They vary four parts of the extraction process: the source model (monolingual BERT vs. mBERT), the input context (isolated word vs. averaging over many sentences), the treatment of the [CLS] and [SEP] tokens, and single-layer vs. averaged-layer representations. The vectors are compared against fastText, and linear CKA is used to measure cross-lingual structural similarity.

The best setup differs by language and task, but some patterns recur. Monolingual models work better than mBERT, and multiple contexts help. Leaving out special tokens and averaging the lower layers also pays off. With these choices, static vectors from monolingual BERT match or beat fastText on LSIM, WA, and RELP. The study offers practical guidance for building lexical resources from LMs, including for cross-lingual use.

Key points

  • The four extraction components combine into 24 configurations. Vocabularies are the 100K most frequent words of each language's Wikipedia-trained fastText model.
  • Context sentences come from 1M sentences sampled per language: Europarl for EN/DE/FI, the United Nations Parallel Corpus for RU/ZH, and WMT17 monolingual data for TR. LSIM is evaluated on Multi-SimLex.
  • Peak scores come from layers L1–L5, and no single layer matches averaging over layers. Lexical knowledge is therefore concentrated in the lower layers but spread across several of them.
  • Translation pairs receive similar representations even from independently trained monolingual BERTs. This similarity is stronger for closely related languages. CKA similarity correlates strongly with BLI performance (ρ = 1.0, 0.83, 0.99 for EN–FI, EN–DE, DE–FI).
  • Scale helps somewhat. EN BERT-Large raises peak EN LSIM from 0.518 to 0.531. For EN–IT, an Italian BERT trained on 81GB instead of 13GB improves BLI from 0.548 to 0.572 and CLIR from 0.148 to 0.160.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

140 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 135 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.586

@inproceedings{vulic2020probing,
  title = {{Probing Pretrained Language Models for Lexical Semantics}},
  author = {Vulić, Ivan and Ponti, Edoardo Maria and Litschko, Robert and Glavaš, Goran and Korhonen, Anna},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.586},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.586}
}

References

66 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of ACL, pages 789–798. find our card
  2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of ACL, pages 4623–4637. find our card
  3. Yonatan Belinkov and James R. Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association of Computational Linguistics, 7:49–72. find our card
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135–146. find
  6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Transla… find our card
  7. Danushka Bollegala and Cong Bao. 2018. Learning word meta-embeddings by autoencoding. In Proceedings of COLING, pages 1650–1661. find
  8. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45. find our card
  9. Ting-Yun Chang and Yun-Nung Chen. 2019. What does this word mean? Explaining contextualized embeddings with natural language definition. In Proceedings of EMNLP-IJCNLP, pages 6064–6070. find
  10. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of ACL, pages 5564–5577. find
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL, pages 8440–8451. find
  12. Daniel Edmiston. 2020. A systematic analysis of morphological content in BERT models for multiple languages. CoRR, abs/2004.03032. — [arXiv:2004.03032](https://arxiv.org/abs/2004.03032) find
  13. Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingual language model fine-tuning. In Proceedings of EMNLP-IJCNLP, pages 5701–5706. find
  14. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of EMNLP-IJCNLP, pages 55–65. find
  15. Christiane Fellbaum. 1998. WordNet. MIT Press. find
  16. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of EMNLP, pages 316–327. find
  17. Goran Glavaš and Ivan Vulić. 2018. Discriminating between lexico-semantic relations with the specialization tensor model. In Proceedings of NAACL-HLT, pages 181–187. find
  18. Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In Proceedings of ACL, pages 710–721. find
  19. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS, pages 249– 256. find
  20. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. BilBOWA: Fast bilingual distributed representations without word alignments. In Proceedings of ICML, pages 748–756. find
  21. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of NAACL-HLT, pages 4129–4138. find our card
  22. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. find our card
  23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186. find
  24. Valentin Hofmann, Janet B. Pierrehumbert, and Hinrich Schütze. 2020. Generating derivational morphology with BERT. CoRR, abs/2005.00672. — [arXiv:2005.00672](https://arxiv.org/abs/2005.00672) find
  25. Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. 2016. Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen. In Proceedings of COLING, pages 3519– 3530. find
  26. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In Proceedings of ICML. find our card
  27. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of ACL, pages 3651– 3657. find our card
  28. Douwe Kiela, Changhan Wang, and Kyunghyun Cho. 2018. Dynamic meta-embeddings for improved sentence representations. In Proceedings of EMNLP, pages 1466–1477. find
  29. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit (MT SUMMIT), pages 79–86. find
  30. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of ICML, pages 3519–3529. find our card
  31. Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proceedings of ACL, pages 4077–4091. find
  32. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS, pages 3111– 3119. find
  33. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532– 1543. find
  34. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. find
  35. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of EMNLP. find our card
  36. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of ACL, pages 4609–4622. find our card
  37. Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised cross-lingual information retrieval using monolingual data only. In Proceedings of SIGIR, pages 1253–1256. find
  38. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of EMNLP. find our card
  39. Robert Litschko, Goran Glavaš, Ivan Vulić, and Laura Dietz. 2019. Evaluating resource-lean cross-lingual embedding models in unsupervised retrieval. In Proceedings of SIGIR, pages 1109–1112. find
  40. Yifan Qiao, Chenyan Xiong, Zheng-Hao Liu, and Zhiyuan Liu. 2019. Understanding the behaviors of BERT in ranking. CoRR, abs/1904.07531. — [arXiv:1904.07531](https://arxiv.org/abs/1904.07531) find
  41. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of NAACL-HLT, pages 1073–1094. find our card
  42. Qianchu Liu, Diana McCarthy, Ivan Vulić, and Anna Korhonen. 2019b. Investigating cross-lingual alignment methods for contextualized embeddings with token-level evaluation. In Proceedings of CoNLL, pages 33–43. find
  43. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  44. Timothee Mickus, Denis Paperno, Mathieu Constant, and Kees van Deemter. 2020. What do you mean, BERT? Assessing BERT as a distributional semantics model. Proceedings of the Society for Computation in Linguistics, 3(34). find
  45. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013a. Exploiting similarities among languages for machine translation. arXiv preprint, CoRR, abs/1309.4168. find
  46. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.08271. find our card
  47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. find
  48. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Proceedings of NeurIPS, pages 8594– 8603. find our card
  49. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: what we know about how BERT works. Transactions of the ACL. find our card
  50. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual embedding models. Journal of Artificial Intelligence Research, 65:569– 631. find
  51. Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47–55. find
  52. Samuel L. Smith, David H.P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of ICLR (Conference Track). find our card
  53. Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of ACL, pages 778–788. find our card
  54. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP, pages 833–844. find our card
  55. Wenpeng Yin and Hinrich Schütze. 2016. Learning word meta-embeddings. In Proceedings of ACL, pages 1351–1360. find
  56. Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In Proceedings of LREC. find our card
  57. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of ACL, pages 4593–4601. find our card
  58. Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In Proceedings of RANLP, pages 237– 248. find
  59. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proceedings EMNLP-IJCNLP, pages 3632–3636. find
  60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS, pages 6000– 6010. find
  61. Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of EMNLP. find our card
  62. Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, and Anna Korhonen. 2020. Multi-Simlex: A large-scale evaluation of multilingual and cross-lingual lexical semantic similarity. Computational Lingui… find
  63. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-Adapter: Infusing knowledge into pre-trained models with adapters. CoRR, abs/2002.01808. find our card
  64. Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of EMNLP-IJCNLP, pages 5721–5727. find
  65. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  66. Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of ACL, pages 6022–6034. find our card