Elamur Research⌕
☰
home / papers / 2021 / Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses

Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses

Paper · 2021 · Aina Garí Soler and Marianna Apidianaki

Summary

The paper asks whether pre-trained contextual language models capture how polysemous a word is, and how cleanly its uses split into distinct senses, rather than only how well they disambiguate. The authors built controlled sentence pools from SemCor 3.0, using WordNet 3.0 sense counts accessed through NLTK. The sense annotations only shaped pool composition and were never used for training. Each monosemous word gets ten random instances. Each polysemous word gets three ten-sentence pools: one covering many senses, one drawn at random, and one restricted to a single sense. Word frequency and part of speech were balanced, since both correlate strongly with polysemy. Similarity patterns among the resulting token embeddings were then analysed across layers for English BERT, ELMo, and context2vec, monolingual French, Spanish, and Greek BERT models, and mBERT. BERT separates mono from poly words in every layer, even when all instances share one sense. This suggests the knowledge is acquired in pre-training and not merely read off context variation. English BERT also predicts how easily a word's meanings cluster into senses. This matters for lexicography, annotation planning, WSD, cross-lingual transfer, and semantic change research.

Key points

  • Data come from SemCor 3.0. Only words with at least ten occurrences are used, organised into the pool settings mono, poly-bal, poly-rand, and poly-same.
  • The polysemy signal is strongest in English BERT. The French, Spanish, and Greek models also distinguish polysemy levels, while mBERT does so to a lesser extent.
  • For partitionability, the best combination pairs BERT representations with agglomerative clustering (BERT-AGG) and the silhouette (SIL) metric. It reaches Spearman ρ = 0.69 with the gold Uiaa score. BERT-REP with SIL reaches ρ = −0.46 with Umid.
  • Raw BERT representations beat substitute-based estimates. Manual substitutes (Gold-SUB) reach only about 0.20–0.34 for Uiaa and 0.16–0.38 for Umid. Gold-AGG, built on Usim scores, reaches 0.80.
  • Code and datasets are released at github.com/ainagari/monopoly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France169 works · 62,488 citations🎓University of HelsinkiHelsinki · Finland51 works · 5,318 citations🔬Laboratoire d'Informatique pour la Mécanique et les Sciences de l'IngénieurOrsay · France25 works · 7,462 citations🎓Université Paris-SaclayGif-sur-Yvette · France48 works · 6,336 citations🔬Laboratoire Interdisciplinaire des Sciences du NumériqueGif-sur-Yvette · France15 works · 597 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Lexical semantics & word embeddings · 65 works

Cite

Cite as Aina Garí Soler and Marianna Apidianaki. 2021. Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00400

@article{soler2021let,
  title = {{Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses}},
  author = {Aina Garí Soler and Marianna Apidianaki},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00400},
  url = {https://doi.org/10.1162/tacl_a_00400}
}

References

60 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Margareta Ackerman and Shai Ben-David. 2009. Clusterability: A Theoretical Study. Journal of Machine Learning Research, 5:1–8. find
  2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR. Toulon, France.
  3. José Cañete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Pérez. 2020. Spanish pre-trained BERT model and evaluation data. In Proceedings of the ICLR 2020 Workshop on Practical ML for Developing Countries (PML4DC). Addis Ababa, Ethiopia. find
  4. Eneko Agirre and David Martinez. 2004. Unsupervised WSD based on automatically retrieved examples: The importance of bias. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 25–32, Barcelona, Spain. Association for Computational Linguistics. find
  5. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the ACL 2019 Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… find
  6. Laura Aina, Kristina Gulordava, and Gemma Boleda. 2019. Putting words in context: LSTM language models and lexical ambiguity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3342–3348, Florence, Italy. Association for Computational Linguistics. find
  7. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  8. Papers), pages 2126–2136, Melbourne, Australia. Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-1198 ↗
  9. Claudio Delli Bovi, Jose Camacho-Collados, Alessandro Raganato, and Roberto Navigli. 2017. EuroSense: Automatic harvesting of multilingual sense annotations from parallel text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages… find
  10. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Language, Speech, and Communication. Cambridge, MA. MIT Press. https://doi.org/10 .7551/mitpress/7287.001.0001 find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The para phrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764, Atlanta, Georgia. Association for C… find
  13. Aina Garı́ Soler, Marianna Apidianaki, and Alexandre Allauzen. 2019. Word usage similarity estimation with sentence representations and automatic substitutes. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 9–21, Minneapolis, Minnesota. Associa… find
  14. Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-Out: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457–470, Florence, Italy. Association for… find
  15. Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020. Analysing lexical semantic change with contextualized word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960–3973, Online. Association for Computational Linguistic… — [doi:10.18653/v1/2020](https://doi.org/10.18653/v1/2020) find
  16. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10–18, Suntec, Singapore.… find
  17. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Katrin Erk, Diana McCarthy, and… find
  18. Language Processing (EMNLP-IJCNLP), pages 4365–4374, Hong Kong, China. Association for Computational Linguistics. https://doi.org/10.18653/v1/D19 -1445 — [doi:10.18653/v1/d19](https://doi.org/10.18653/v1/d19) ↗
  19. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129…
  20. Hang Le, Loı̈c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoı̂t Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model Pre-training for French. In Proceedings of The 12th Language Resources and Evaluation Conferen…
  21. Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882, Jeju Island, Korea.…
  22. Claudia Leacock, Martin Chodorow, and George A. Miller. 1998. Using corpus statistics and WordNet relations for sense identification. Computational Linguistics, 24(1):147–165.
  23. Tal Linzen. 2018. What can linguistics and deep learning contribute to each other? arXiv preprint:1809.04179v2. https://doi.org/10 .1162/tacl a 00115 — [arXiv:1809.04179](https://arxiv.org/abs/1809.04179) find
  24. Alexander Jakubowski, Milica Gasic, and Marcus Zibrowius. 2020. Topology of word embeddings: Singularities reflect polysemy. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 103–113, Barcelona, Spain (Online). Association for Computational Linguistics. find
  25. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.
  26. Adam Kilgarriff. 2004. How dominant is the commonest sense of a word? In Lecture Notes in Computer Science (vol. 3206), Text, Speech and Dialogue, Sojka Petr, Kopeček Ivan, Pala Karel (eds.), pages 103–112, Springer. Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30120-2_14 ↗
  27. Daniel Loureiro and Jose Camacho-Collados. 2020. Don’t neglect the obvious: On the role of unambiguous words in word sense disambiguation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3514–3520, Online. Association for Computational Linguis… find
  28. John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020. GREEK-BERT: The Greeks visiting Sesame Street. In Proceedings of the 11th Hellenic Conference on Artificial Intelligence (SETN 2020), pages 110–117, Athens, Greece. https://doi.org/10.1145/3411408 .3411440 — [doi:10.1145/3411408](https://doi.org/10.1145/3411408) ↗
  29. Diana McCarthy, Marianna Apidianaki, and Katrin Erk. 2016. Word sense clustering and clusterability. Computational Linguistics, 42(2):245–275. https://doi.org/10.1162/COLI a 00247 ↗
  30. Diana McCarthy, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in untagged text. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 279–286. Barcelona, Spain. https://doi.org/10.3115/1218955.1218991 ↗
  31. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Diana McCarthy and Roberto Navigli. 2007. SemEval-…
  32. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational…
  33. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics. https://doi.… ↗
  34. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  35. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint:1301.3781v3.
  36. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro. New Jersey. https://doi.org/10 .3115/1075671.1075742
  37. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: The Word-in-Context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  38. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. https://doi.org/10.1016/j.artint.2012.07.001 ↗
  39. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1059–1069, Doha, Qatar. Assoc…
  40. Tiago Pimentel, Rowan Hall Maudslay, Damian Blasi, and Ryan Cotterell. 2020. Speakers fill lexical semantic gaps with context. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4004–4015, Online. Association for Computational Linguistics. https:… find
  41. Rafail Ostrovsky, Yuval Rabani, Leonard J. Schulman, and Chaitanya Swamy. 2012. The effectiveness of Lloyd-type methods for the k-means problem. Journal of the ACM (JACM), 59(6):28. https://doi.org/10 .1145/2395116.2395117 find
  42. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  43. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems, pages 8592–8600, Vancouver, Canada.
  44. Cardiff, UK. Leibniz-Institut für Deutsche Sprache. find
  45. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics—On what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743–758. https://doi.org/10.1162/tacl_a_00342 ↗
  46. Joseph Reisinger and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109–117, Los Angeles, California. Association for Comput…
  47. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19 -14… ↗
  48. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866.
  49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. Long Beach, California, USA.
  50. Alex Rosenfeld and Katrin Erk. 2018. Deep neural models of semantic shift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 474–484, New Orleans, Louisiana. Association for…
  51. Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joi…
  52. Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65. find
  53. Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 Task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1–23, Barcelona (online). International Committee for Com…
  54. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019b. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence,…
  55. Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In Proceedings of the 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). find
  56. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computation…
  57. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 161–170, Erlangen, Germany.… find
  58. George Kingsley Zipf. 1945. The meaning-frequency relationship of words. Journal of General Psychology, 33(2):251–256. https://doi.org/10.1080/00221309.1945 .10544509 — [doi:10.1080/00221309.1945](https://doi.org/10.1080/00221309.1945) ↗
  59. A Clusterability Metrics Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Ma… — Transformers: State-of-the-Art Natural Language Processing
  60. Variance Ratio. First, the variance of a cluster y is calculated: 1  σ 2 ( Y ) = ( y i − ȳ ) 2 (3) \| y \| iy where ȳ denotes the centroid of cluster y . Then the within-cluster variance W and the between-cluster variance B of a clustering solution C are calculated in the following way: W ( C ) =… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/soler-2021-let-s-play-mono-poly/