Elamur Research
☰
home / papers / 2020 / Analysing Lexical Semantic Change with Contextualised Word Representations

Analysing Lexical Semantic Change with Contextualised Word Representations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.365.pdf

Summary

Most earlier work on lexical semantic change gives each word form a single static vector or assumes a fixed set of senses. This hides the gradual, polysemous paths that meaning shifts usually follow. The authors propose an unsupervised, usage-based alternative built on contextualised embeddings. Each occurrence of a target word in COHA (1910–2009) is encoded with BERT base-uncased by summing the outputs of all hidden layers. The occurrence vectors are then grouped into usage types with K-Means, where the number of clusters is chosen per word by the silhouette score. Change over time is measured by three metrics: ED, JSD and APD. For evaluation, the authors build a new dataset of human similarity judgements on word usage pairs drawn from different decades. Both the representations and the detected shifts correlate positively with human judgements. Qualitative analysis shows the method separates senses, syntactic roles and literal versus metaphorical uses, and traces narrowing and broadening over time. This is the first approach of its kind that needs no lexicographic supervision.

Key points

  • Model and representations: Usage vectors come from BERT base-uncased (12 layers, 768 dimensions, 110M parameters) via Hugging Face's implementation.
    • Layers are summed because concatenation or using a subset of layers barely changed the relative distances between vectors.
  • Clustering: K-Means is run for K from 2 to 10, with 10 restarts per K, and the best K is picked by silhouette score. As a result, the setup cannot represent a word with only one usage type.
  • Metrics: ED, JSD and APD each have different strengths.
    • Negative ED signals narrowing well (e.g., coach). It misses narrowing when a new usage type appears at the same time (e.g., tenure).
    • JSD and APD best capture broadening (e.g., disk acquiring an optical-disk sense from the 1950s).
    • JSD picks up rare new uses such as curtain in "iron curtain". APD, which does not depend on clusters, is robust to clustering errors (e.g., address).
  • Resources: The new dataset contains human similarity judgements on more than 3K usage pairs and is available on Zenodo (doi 10.5281/zenodo.3773250). Code is at github.com/glnmario/cwr4lsc. Target words come from the GEMS set.
  • Limitations and follow-up: The metrics show that a word broadened but not how; for example, detecting metaphorisation would require comparing clusters. Follow-up work (Kutuzov and Giulianelli, 2020) extends the approach to four languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.365

@inproceedings{giulianelli2020analysing,
  title = {{Analysing Lexical Semantic Change with Contextualised Word Representations}},
  author = {Giulianelli, Mario and Tredici, Marco Del and Fernández, Raquel},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.365},
  url = {https://doi.org/10.18653/v1/2020.acl-main.365}
}

References

55 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Arthur and Sergei Vassilvitskii. 2007. k-means++: The Advantages of Careful Seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1027–1035. Society for Industrial and Applied Mathematics. find
  2. Robert Bamler and Stephan Mandt. 2017. Dynamic Word Embeddings. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 380–389. JMLR.org. find
  3. Susan Windisch Brown. 2008. Choosing Sense Distinctions for WSD: Psycholinguistic Evidence. In Proceedings of ACL-08: HLT, Short Papers, pages 249–252, Columbus, Ohio. Association for Computational Linguistics. find
  4. Claudia Marlea Brugman. 1988. The Story of Over: Polysemy, Semantics, and the Structure of the Lexicon. Garland, New York. find
  5. Joan Bybee. 2015. Language Change. Cambridge University Press. find
  6. Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel Word-Sense Identification. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1624–1635. find
  7. Andrew M Dai and Quoc V Le. 2015. Semi-supervised Sequence Learning. In Advances in Neural Information Processing Systems, pages 3079–3087. find
  8. Mark Davies. 2012. Expanding Horizons in Historical Linguistics with the 400-Million Word Corpus of Historical American English. Corpora, 7(2):121– 157. find
  9. Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2019. Short-Term Meaning Shift: A Distributional Exploration. In Proceedings of NAACL-HLT 2019 (Annual Conference of the North American Chapter of the Association for Computational Linguistics). find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  11. Haim Dubossarsky, Yulia Tsvetkov, Chris Dyer, and Eitan Grossman. 2015. A Bottom Up Approach to Category Mapping and Meaning Change. In Word Structure and Word Usage. Proceedings of the Net-WordS Final Conference, pages 66–70. find
  12. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on Word Senses and Word Usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10–18, Suntec, Singapore.… find
  13. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2013. Measuring Word Meaning in Context. Computational Linguistics, 39(3):511–554. find
  14. Katrin Erk and Sebastian Padó. 2008. A Structured Vector Space Model for Word Meaning in Context. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 897–906. find
  15. Katrin Erk and Sebastian Padó. 2010. Exemplar-Based Models for Word Meaning in Context. In Proceedings of the ACL 2010 Conference (Short Papers), pages 92–97. find
  16. Lea Frermann and Mirella Lapata. 2016. A Bayesian Model of Diachronic Meaning Change. Transactions of the Association for Computational Linguistics, 4:31–45. find
  17. Mario Giulianelli. 2019. Lexical Semantic Change Analysis with Contextualised Word Representations. Master’s thesis, University of Amsterdam, July. find
  18. Kristina Gulordava and Marco Baroni. 2011. A Distributional Similarity Approach to the Detection of Semantic Change in the Google Books Ngram Corpus. In Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics, pages 67–71. find
  19. William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501. find our card
  20. Paul J Hopper et al. 1991. On Some Principles of Grammaticization. Approaches to Grammaticalization, 1:17–35. find
  21. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. find
  22. Renfen Hu, Shen Li, and Shichen Liang. 2019. Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3899– 3908, Florence, Italy. Association for Computational Linguistic… find
  23. Adam Kilgarriff. 1997. I Don’t Believe in Word Senses. Computers and the Humanities, 31(2):91– 113. find our card
  24. Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal Analysis of Language through Neural Language Models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61–65. find
  25. Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically Significant Detection of Linguistic Change. In Proceedings of the 24th International Conference on World Wide Web, pages 625–635. International World Wide Web Conferences Steering Committee. find
  26. Andrey Kutuzov and Mario Giulianelli. 2020. UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. Forthcoming. find
  27. Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic Word Embeddings and Semantic Shifts: A Survey. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384–1397. find our card
  28. Jey Han Lau, Paul Cook, Diana McCarthy, Spandana Gella, and Timothy Baldwin. 2014. Learning Word Sense Distributions, Detecting Unattested Senses and Identifying Novel Senses Using Topic Models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  29. Jey Han Lau, Paul Cook, Diana McCarthy, David Newman, and Timothy Baldwin. 2012. Word Sense Induction for Novel Sense Detection. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 591–601. Association for Computational Linguistics. find
  30. Jianhua Lin. 1991. Divergence Measures Based on the Shannon Entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  31. Peter Ludlow. 2014. Living Words: Meaning Underdetermination and the Dynamic Lexicon. OUP Oxford. find
  32. Suresh Manandhar, Ioannis P Klapaftis, Dmitriy Dligach, and Sameer S Pradhan. 2010. SemEval-2010 Task 14: Word Sense Induction & Disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 63–68. Association for Computational Linguistics. find
  33. Nathan Mantel. 1967. The Detection of Disease Clustering and a Generalized Regression Approach. Cancer Research, 27(2):209–220. find
  34. Matej Martinc, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020. Capturing Evolution in Word Usage: Just Add More Clusters? In Companion Proceedings of the International World Wide Web Conference, pages 20–24. find
  35. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in Translation: Contextualized Word Vectors. In Advances in Neural Information Processing Systems, pages 6294–6305. find our card
  36. Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, et al. 2011. Quantitative Analysis of Culture Using Millions of Digitized Books. Science, 331(6014):176–182. find
  37. Sunny Mitra, Ritwik Mitra, Suman Kalyan Maity, Martin Riedl, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2015. An Automatic Approach to Identify Word Sense Changes in Text Media across Timescales. Natural Language Engineering, 21(5):773–798. find
  38. Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That’s Sick Dude! Automatic Identification of Word Sense Change across Different Timescales. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… find
  39. Roberto Navigli and Daniele Vannella. 2013. SemEval-2013 Task 11: Word Sense Induction and Disambiguation within an End-User Application. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (Sem… find
  40. Patrick Pantel and Dekang Lin. 2002. Discovering Word Senses from Text. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’02, page 613–619, New York, NY, USA. Association for Computing Machinery. find
  41. Carita Paradis. 2011. Metonymization: A Key Mechanism in Semantic Change. Defining Metonymy in Cognitive Linguistics: Towards a Consensus View, pages 61–98. find
  42. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  43. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find our card
  44. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. Technical report, OpenAI. find
  45. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI. find
  46. Miguel A Ré and Rajeev K Azad. 2014. Generalization of Entropy Based Divergence Measures for Symbolic Sequence Analysis. PloS one, 9(4):e93532. find
  47. Alex Rosenfeld and Katrin Erk. 2018. Deep Neural Models of Semantic Shift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 474–484. find
  48. Peter J. Rousseeuw. 1987. Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis. Journal of Computational and Applied Mathematics, 20:53–65. find
  49. Maja Rudolph and David Blei. 2018. Dynamic embeddings for Language evolution. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1003–1011. International World Wide Web Conferences Steering Committee. find
  50. Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Eckmann. 2018. Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  51. Hinrich Schütze. 1998. Automatic Word Sense Discrimination. Computational Linguistics, 24(1):97– 123. find our card
  52. Xuri Tang. 2018. A State-of-the-Art of Semantic Change computation. Natural Language Engineering, 24(5):649–676. find
  53. Xuri Tang, Weiguang Qu, and Xiaohe Chen. 2016. Semantic Change Computation: A Successive Approach. World Wide Web, 19(3):375–415. find
  54. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  55. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings. In Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019, Erlangen, Germany. find