Elamur Research⌕
☰
home / papers / 2017 / Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models

Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models

Summary

The paper asks whether recently proposed "laws" of semantic change reflect real linguistic behaviour or are artefacts of how word meaning is represented and measured. The authors propose a rule: an effect only counts as genuine if it shows up in real diachronic data but disappears or weakens in a matched control where meaning cannot have changed. They use Google Books English fiction 5-grams (1900–1999, grouped by decade) as the genuine condition. They build two controls: a corpus shuffled across decades, and repeated subsamples drawn from the single year 1999. They then measure change as the cosine distance between a word's vectors in neighbouring time bins, using counts, PPMI and SVD representations. In the controls, frequency and polysemy correlate with "change" much as they do in the real data. This leaves the Law of Conformity and the Law of Prototypicality with only small residual effects. The Law of Innovation is largely explained by frequency. Analytical proofs show that count-based vectors inherently tie cosine distance to frequency. This matters for anyone drawing linguistic conclusions from embedding comparisons.

Key points

  • Three proposed laws were tested:
    • Conformity: more frequent words change less.
    • Innovation: more polysemous words change more.
    • Prototypicality: more prototypical words change less.
  • Data setup:
    • 10 million 5-grams were sampled per year so that productive years would not dominate.
    • The vocabulary was the 100,000 most frequent words.
    • The analysis focused on the 10k most frequent words.
  • The subsample control drew 10 million 5-grams 30 times from the roughly 250 million 5-grams available for 1999. Meaning change in this control was the average cosine distance across samples.
  • Representation pipeline:
    • Co-occurrence counts.
    • PPMI with context smoothing α = 0.75.
    • Truncated SVD to 300 dimensions, followed by alignment across decades.
  • Effects were assessed with a linear mixed effects model (frequency as a fixed effect, a random intercept per word) in R using lme4 and MuMIn, reporting only results with p < .01. Spurious frequency effects were much weaker for plain PPMI than after the SVD step.

Interesting ideas

  • I1. Null-change baselines: Any diachronic claim can be checked against a corpus where change is impossible by construction, for example by shuffling texts across time bins or resampling one period. This is a cheap and reusable test for temporal analyses of news, social media or trade documents.
  • I2. Matched controls: The shuffled corpus keeps vocabulary, corpus size and bin sizes identical to the real data. Only the variable of interest is removed, so a surviving effect cannot be blamed on corpus mechanics.
  • I3. Construct-validity check: Before testing hypotheses, the authors confirm that the control shows less average change than the real data and a roughly constant rate over decades. Validating the baseline before trusting it is good practice in any evaluation.
  • I4. Sampling noise as a confound: Two samples of the same word's contexts are not expected to have zero cosine distance. The gap reflects sampling variance, which is larger for rarer words. Any similarity-based metric over corpora of different sizes (such as low-resource language corpora) should account for this.
  • I5. Collinearity check: Before attributing an effect to a predictor like polysemy, check whether it is strongly correlated with frequency. A predictor that overlaps heavily with frequency may add no independent explanatory power.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Historical linguistics & language change

Institutions

🎓Hebrew University of JerusalemJerusalem · Israel42 works · 8 354 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1118

@inproceedings{dubossarsky2017outta,
  title = {{Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models}},
  author = {Dubossarsky, Haim and Weinshall, Daphna and Grossman, Eitan},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1118},
  url = {https://doi.org/10.18653/v1/d17-1118}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of ACL, pages 238–247. find our card
  2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(4-5):993–1022. find our card
  3. Vladimir Bochkarev, Valery Solovyev, and Sören Wichmann. 2014. Universals versus historical contingencies in lexical evolution. Journal of The Royal Society Interface, 11:1–23. find
  4. John A Bullinaria and Joseph P Levy. 2012. Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and svd. Behavior research methods, 44(3):890–907. find
  5. Haim Dubossarsky, Yulia Tsvetkov, Chris Dyer, and Eitan Grossman. 2015. A bottom up approach to category mapping and meaning change. In Net-WordS 2015 Word Knowledge and Word Usage, pages 66–70. find
  6. Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2016. Verbs change more than nouns: A bottom up computational approach to semantic change. Lingue e Linguaggio, 1:5–25. find
  7. John Rupert Firth. 1957. Papers in Linguistics 1934– 1951. Oxford University Press, London. find
  8. Lea Frermann and Mirella Lapata. 2016. A Bayesian model of diachronic meaning change. TACL, 4:31– 45. find
  9. Kristina Gulordava and Marco Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 67–71. Association for Computational Linguistics. find our card
  10. William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of ACL. find our card
  11. Martin Hilpert and Florent Perek. 2015. Meaning change in a petri dish: constructions, semantic vector spaces, and motion charts. Linguistics Vanguard, 1(1):339–350. find
  12. Adam Jatowt and Kevin Duh. 2014. A framework for analyzing semantic change of words across time. In Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries, pages 229–238. find
  13. Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, pages 625–635. find
  14. Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems (NIPS), pages 2177–2185. find
  15. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. TACL, 3:211–225. find our card
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NIPS), pages 3111–3119. find our card
  17. Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That’s sick dude!: Automatic identification of word sense change across different timescales. In Proceedings of ACL, pages 1020–1029. find
  18. Elliott Moreton. 2008. Analytic bias and phonological typology. Phonology, 25(1):83–127. find
  19. Shinichi Nakagawa and Holger Schielzeth. 2013. A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2):133–142. find
  20. John Newman. 2015. Semantic shift. In Nick Rimer, editor, The Routledge Handbook of Semantics, pages 266–280. Routledge, New York. find
  21. Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2009. Semantic density analysis: Comparing word meaning across time and phonetic space. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 104–111. Association for Computational Linguistics. find
  22. Peter D Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research, 37:141–188. find our card
  23. Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In Proceedings of the 2011 international workshop on DETecting and Exploiting Cultural diversiTy on the social web, pages 35–40. ACM. find
  24. Yang Xu and Charles Kemp. 2015. A computational evaluation of two laws of semantic change. In CogSci. find
  25. Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. In Proceedings of ACL, pages 61–65. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dubossarsky-2017-outta-control-laws-semantic-change/