Elamur Research
☰
home / papers / 2016 / Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-1141.pdf

Summary

How fast a word's meaning drifts, and which properties of a word drive that drift, has mostly been studied through hand-picked examples. Large-scale historical evidence about meaning is hard to obtain. Hamilton, Leskovec and Jurafsky develop diachronic word embeddings into a tested measurement tool. They train decade-by-decade PPMI, SVD and SGNS (word2vec) representations on six historical corpora: five from Google N-Grams plus COHA. Together these cover English, German, French and Chinese over roughly two centuries. The low-dimensional spaces are mapped onto shared axes with orthogonal Procrustes. Change is then quantified in two ways: as shifts in similarity between word pairs, or as a word's displacement across decades. The methods are validated against documented historical shifts. Applying the pipeline across languages yields two regularities. First, frequent words drift more slowly, following an inverse power law (law of conformity). Second, once frequency is controlled for, contextually diverse (polysemous) words change faster (law of innovation). These results give historical semantics a quantitative baseline. They also suggest that polysemy may drive semantic change, not merely result from it.

Key points

  • Data. The six datasets are ENGALL, ENGFIC, COHA, FREALL, GERALL and CHIALL.
    • The Google data spans 1800–1999, except Chinese, which spans 1950–1999. COHA spans 1810–2009.
    • All data is binned by decade, and the Google 2000s decade is dropped.
    • COHA lemmas were used to confirm that results hold for both lemmas and raw tokens.
  • Setup. The context window is symmetric, with 4 words on each side. SVD and SGNS vectors have 300 dimensions. Procrustes alignment is solved via SVD. Incremental initialization for SGNS barely affected the results.
  • Polysemy measure. Polysemy is estimated from a word's local clustering coefficient in a co-occurrence network.
    • The measure correlates with frequency (Pearson 0.45 < r < 0.8).
    • After controlling for frequency, change scales with polysemy as a power law, with exponent β_d between 0.37 and 0.77.
  • Explanatory power. Frequency and polysemy together explain 48%–88% of the variance in rates of semantic change (marginal R²).
  • Resources. The embeddings and analysis code are publicly released as HistWords (nlp.stanford.edu/projects/histwords).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Cited by

323 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 312 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1141 · arXiv 1605.09096

@inproceedings{hamilton2016diachronic,
  title = {{Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change}},
  author = {Hamilton, William L. and Leskovec, Jure and Jurafsky, Dan},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1141},
  eprint = {1605.09096},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1141}
}

References

62 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James S. Adelman, Gordon D. A. Brown, and José F. Quesada. 2006. Contextual diversity, not word frequency, determines word-naming and lexical decision times. Psychol. Sci., 17(9):814–823. find
  2. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python. O’Reilly Media, Inc. find
  3. Andreas Blank. 1999. Why do new meanings occur? A cognitive typology of the motivations for lexical semantic change. In Peter Koch and Andreas Blank, editors, Historical Semantics and Cognition. Walter de Gruyter, Berlin, Germany. find
  4. Robert Boyd and Peter J Richerson. 1988. Culture and the Evolutionary Process. University of Chicago Press, Chicago, IL. find
  5. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proc. ACL, pages 136–145. find
  6. Michel Bréal. 1897. Essai de Sémantique: Science des significations. Hachette, Paris, France. find
  7. John A. Bullinaria and Joseph P. Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behav. Res. Methods, 39(3):510–526. find
  8. John A. Bullinaria and Joseph P. Levy. 2012. Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD. Behav. Res. Methods, 44(3):890–907. find
  9. J.L. Bybee. 2007. Frequency of Use And the Organization of Language. Oxford University Press, New York City, NY. find
  10. Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel Word-sense Identification. In Proc. COLING, pages 1624–1635. find
  11. Scott Crossley, Tom Salsbury, and Danielle McNamara. 2010. The development of polysemy and frequency use in english second language speakers. Language Learning, 60(3):573–605. find
  12. Mark Davies. 2010. The Corpus of Historical American English: 400 million words, 1810-2009. http://corpus.byu.edu/coha/. ↗
  13. Beate Dorow and Dominic Widdows. 2003. Discovering corpus-specific word senses. In Proc. EACL, pages 79–82. find
  14. J.R. Firth. 1957. A Synopsis of Linguistic Theory, 1930-1955. In Studies in Linguistic Analysis. Special volume of the Philological Society. Basil Blackwell, Oxford, UK. find
  15. Lea Frermann and Mirella Lapata. 2016. A Bayesian Model of Diachronic Meaning Change. Trans. ACL, 4:31–45. find
  16. Dirk Geeraerts. 1997. Diachronic Prototype Semantics: A Contribution to Historical Lexicology. Clarendon Press, Oxford, UK. find
  17. Michael H. Graham. 2003. Confronting multicollinearity in ecological multiple regression. Ecology, 84(11):2809–2815. find
  18. Kristina Gulordava and Marco Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proc. find
  19. GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics, pages 67–71. Association for Computational Linguistics. find
  20. Zellig S. Harris. 1954. Distributional structure. Word, 10:146–162. find
  21. Paul J. Hopper and Elizabeth Closs Traugott. 2003. Grammaticalization. Cambridge University Press, Cambridge, UK. find
  22. Adam Jatowt and Kevin Duh. 2014. A framework for analyzing semantic change of words across time. In Proc. ACM/IEEE-CS Conf. on Digital Libraries, pages 229–238. IEEE Press. find
  23. R. Jeffers and Ilse Lehiste. 1979. Principles and Methods for Historical Linguistics. MIT Press, Cambridge, MA. find
  24. Adam Kilgarriff. 2004. How dominant is the commonest sense of a word? In Text, Speech and Dialogue, pages 103–111. Springer. find
  25. Yoon Kim, Yi-I. Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. arXiv preprint arXiv:1405.3515. find
  26. Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2014. Statistically significant detection of linguistic change. In Proc. WWW, pages 625–635. find
  27. Thomas K. Landauer and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol. Rev., 104(2):211. find
  28. Christiane Fellbaum. 1998. WordNet. Wiley Online Library. find
  29. Jey Han Lau, Paul Cook, Diana McCarthy, David Newman, and Timothy Baldwin. 2012. Word sense induction for novel sense detection. In Proc. EACL, pages 591–601. find
  30. Olivier Ferret. 2004. Discovering word senses from a network of lexical cooccurrences. In Proc. COL-ING, page 1326. find
  31. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. Trans. ACL, 3. find
  32. Qi Li and Jeffrey Scott Racine. 2007. Nonparametric econometrics: theory and practice. Princeton University Press, Princeton, NJ. find
  33. Erez Lieberman, Jean-Baptiste Michel, Joe Jackson, Tina Tang, and Martin A. Nowak. 2007. Quantifying the evolutionary dynamics of language. Nature, 449(7163):713–716. find
  34. Yuri Lin, Jean-Baptiste Michel, Erez Lieberman Aiden, Jon Orwant, Will Brockman, and Slav Petrov. 2012. Syntactic annotations for the google books ngram corpus. In Proc. ACL, System Demonstrations, pages 169–174. find
  35. Charles E McCulloch and John M Neuhaus. 2001. Generalized linear mixed models. Wiley-Interscience, Hoboken, NJ. find
  36. Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, and others. 2011. Quantitative analysis of culture using millions of digitized books. Science, 331(6014):176–182. find
  37. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  38. Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That’s sick dude!: Automatic identification of word sense change across different timescales. In Proc. ACL. find
  39. Sunny Mitra, Ritwik Mitra, Suman Kalyan Maity, Martin Riedl, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2015. An automatic approach to identify word sense changes in text media across timescales. Natural Language Engineering, 21(05):773–798. find
  40. Shinichi Nakagawa and Holger Schielzeth. 2013. A general and simple method for obtaining R 2 from generalized linear mixed-effects models. Methods Ecol. Evol., 4(2):133–142. find
  41. Mark Pagel, Quentin D. Atkinson, and Andrew Meade. 2007. Frequency of word-use predicts rates of lexical evolution throughout Indo-European history. Nature, 449(7163):717–720. find
  42. Eitan Adam Pechenick, Christopher M. Danforth, and Peter Sheridan Dodds. 2015. Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution. PLoS ONE, 10(10). find
  43. F. Reali and T. L. Griffiths. 2010. Words as alleles: connecting language evolution with Bayesian learners to models of genetic drift. Proc. R. Soc. B, 277(1680):429–436. find
  44. Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2011. Tracing semantic change with latent semantic analysis. In Kathryn Allan and Justyna A. Robinson, editors, Current Methods in Historical Semantics, page 161. De Gruyter Mouton, Berlin, Germany. find
  45. Benôit Sagot, Lionel Clément, Eric de La Clergerie, and Pierre Boullier. 2006. The Lefff 2 syntactic lexicon for French: architecture, acquisition, use. In Proc. LREC, pages 1–4. find
  46. Gerold Schneider and Martin Volk. 1998. Adding manual constraints and lexical look-up to a Brill-tagger for German. In Proceedings of the ESSLLI-98 Workshop on Recent Advances in Corpus Annotation, Saarbrücken. find
  47. Peter H Schönemann. 1966. A generalized solution of the orthogonal Procrustes problem. Psychometrika, 31(1):1–10. find
  48. J.S. Seabold and J. Perktold. 2010. Statsmodels: Econometric and statistical modeling with python. In Proc. 9th Python in Science Conference. find
  49. John Andrew Simpson, Edmund SC Weiner, et al. 1989. The Oxford English Dictionary, volume 2. Clarendon Press Oxford, Oxford, UK. find
  50. Elizabeth Closs Traugott and Richard B Dasher. 2001. Regularity in Semantic Change. Cambridge University Press, Cambridge, UK. find
  51. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. J. Artif. Intell. Res., 37(1):141–188. find
  52. S. Ullmann. 1962. Semantics: An Introduction to the Science of Meaning. Barnes & Noble, New York City, NY. find
  53. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(2579-2605):85. find
  54. Duncan J Watts and Steven H Strogatz. 1998. Collective dynamics of ‘small-world’networks. Nature, 393(6684):440–442. find
  55. Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In Proc. Workshop on Detecting and Exploiting Cultural Diversity on the Social Web, pages 35– 40. ACM. find
  56. David P Wilkins. 1993. From part to person: Natural tendencies of semantic change and the search for cognates. Cognitive Anthropology Research Group at the Max Planck Institute for Psycholinguistics. find
  57. B. Winter, Graham Thompson, and Matthias Urban. 2014. Cognitive Factors Motivating The Evolution Of Word Meanings: Evidence From Corpora, Behavioral Data And Encyclopedic Network Structure. In Proc. EVOLANG, pages 353–360. find
  58. Yang Xu and Charles Kemp. 2015. A computational evaluation of two laws of semantic change. In Proc. Annual Conf. of the Cognitive Science Society. find
  59. Yang Xu, Terry Regier, and Barbara C. Malt. 2015. Historical Semantic Chaining and Efficient Communication: The Case of Container Names. Cognitive Science. find
  60. Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The Penn Chinese TreeBank: Phrase structure annotation of a large corpus. Natural language engineering, 11(02):207–238. find
  61. George Kingsley Zipf. 1945. The meaning-frequency relationship of words. J. Gen. Psychol., 33(2):251– 256. find
  62. Bahar İlgen and Bahar Karaoglan. 2007. Investigation of Zipf’s ‘law-of-meaning’on Turkish corpora. In International Symposium on Computer and Information Sciences, pages 1–6. IEEE. find