Elamur Research
☰
home / papers / 2018 / Diachronic word embeddings and semantic shifts: a survey

Diachronic word embeddings and semantic shifts: a survey

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/C18-1117.pdf

Summary

Since about 2011, research on detecting changes in word meaning with distributional models has grown quickly. However, the work is spread across NLP, information retrieval and political science, and these communities use inconsistent terminology and have no shared practices. This survey by Kutuzov, Øvrelid, Szymanski and Velldal organises that literature.

It first grounds the topic in historical linguistics. It contrasts slow linguistic drifts in a word's core meaning with culturally driven shifts in its associations. It then defines the computational task: given corpora sliced by time period, find words whose meaning differs between periods or has changed the most. Existing studies are compared along several dimensions:

  • which diachronic corpora they use and at what time granularity,
  • how they evaluate results,
  • how they extract shifts,
  • how they align or compare vectors across periods.

Later sections cover proposed statistical laws of semantic change, temporal relations between words, and applications ranging from historical linguistics to event detection in news. The paper offers no new model. Its value is a common map of the field and a list of open problems, useful for newcomers and for work on languages other than English.

Key points

  • Corpora discussed include the Helsinki Corpus, Google Books Ngrams (5-year granularity), Amazon Movie Reviews (yearly), Twitter (monthly) and the New York Times Annotated Corpus. Szymanski (2017) trained 21 yearly models covering 1987–2007, and Yao et al. (2018) used 27 yearly NYT subcorpora from 1990 to 2016.
  • Over time, studies have moved toward finer time slices, from centuries or 50-year spans down to single years or months. Kulkarni et al. (2015) showed that shift detection remains robust for periods shorter than a decade.
  • According to Dubossarsky et al. (2016), verbs are the most prone to shifting meaning, followed by nouns, while adjectives shift the least.
  • The main open challenges are:
    • coverage beyond English,
    • methods that work on small datasets,
    • reliable gold-standard test sets,
    • formal joint models across time spans,
    • sense-aware analysis that classifies the type and cause of a shift.
  • The authors call for dedicated workshops and shared tasks. They note that the SemEval-2015 diachronic task dated texts rather than tracking how individual words change meaning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

143 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 137 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kutuzov2018diachronic,
  title = {{Diachronic word embeddings and semantic shifts: a survey}},
  author = {Kutuzov, Andrey and Øvrelid, Lilja and Szymanski, Terrence and Velldal, Erik},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1117/}
}

References

73 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hosein Azarbonyad, Mostafa Dehghani, Kaspar Beelen, Alexandra Arkut, Maarten Marx, and Jaap Kamps. 2017. Words are malleable: Computing semantic shifts in political and media discourse. In Proceedings of the ACM on Conference on Information and Knowledge Management, pages 1509–1518, Singapore. find
  2. Robert Bamler and Stephan Mandt. 2017. Dynamic word embeddings. In Proceedings of the International Conference on Machine Learning, pages 380–389, Sydney, Australia. find
  3. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 238–247, Baltimore, USA. find our card
  4. Sergey Bartunov, Dmitry Kondrashkin, Anton Osokin, and Dmitry Vetrov. 2016. Breaking sticks and ambiguities with adaptive skip-gram. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pages 130–138, Cadiz, Spain. find
  5. Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014. Analysing word meaning over time by exploiting temporal random indexing. In Proceedings of the First Italian Conference on Computational Linguistics, pages 38–42, Turin, Italy. find
  6. Andreas Blank and Peter Koch. 1999. Historical semantics and cognition. Walter de Gruyter. find
  7. David M Blei and John D Lafferty. 2006. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine learning, pages 113–120, Pittsburgh, USA. find our card
  8. Leonard Bloomfield. 1933. Language. Allen & Unwin. find
  9. Michel Bréal. 1899. Essai de sémantique. Hachette, Paris. find
  10. John A Bullinaria and Joseph P Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior research methods, 39(3):510–526. find
  11. Hyunyoung Choi and Hal Varian. 2012. Predicting the present with Google trends. Economic Record, 88(s1):2–9. find
  12. Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science, 41(6):391–407. find
  13. Haim Dubossarsky, Yulia Tsvetkov, Chris Dyer, and Eitan Grossman. 2015. A bottom up approach to category mapping and meaning change. In Proceedings of the NetWordS 2015 Word Knowledge and Word Usage, pages 66–70, Pisa, Italy. find
  14. Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2016. Verbs change more than nouns: a bottom-up computational approach to semantic change. Lingue e linguaggio, 15(1):7–28. find
  15. Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017. Outta control: Laws of semantic change and inherent biases in word representation models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1147–1156, Copenhagen, Denmark. find
  16. Steffen Eger and Alexander Mehler. 2016. On the linearity of semantic change: Investigating meaning variation via dynamic graph models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 52–58, Berlin, Germany. find
  17. John Firth. 1957. A synopsis of linguistic theory, 1930-1955. Blackwell. find
  18. Lea Frermann and Mirella Lapata. 2016. A bayesian model of diachronic meaning change. Transactions of the Association of Computational Linguistics, 4:31–45. find
  19. Dirk Geeraerts. 1997. Diachronic prototype semantics: A contribution to historical lexicology. Clarendon Press, Oxford. find
  20. Stefan Th. Gries. 1999. Particle movement: a cognitive and functional approach. Cognitive Linguistics, 10:105– 145. find
  21. Joachim Grzega and Marion Schoener. 2007. English and general historical lexicology. Eichstätt-Ingolstadt: Katholische Universität. find
  22. Kristina Gulordava and Marco Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics, pages 67–71, Edinburgh, UK. find
  23. L. William Hamilton, Jure Leskovec, and Dan Jurafsky. 2016a. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1489–1501, Berlin, Germany. find our card
  24. William L. Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. 2016b. Inducing domain-specific sentiment lexicons from unlabeled corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 595–605, Austin, Texas. find
  25. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016c. Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2116–2121, Austin, Texas. find
  26. Gerhard Heyer, Florian Holz, and Sven Teresniak. 2009. Change of topics over time – tracking topics by their change of meaning. In Proceeding of the International Conference on Knowledge Discovery and Information Retrieval, pages 223–228, Madeira, Portugal. find
  27. Gerhard Heyer, Cathleen Kantner, Andreas Niekler, Max Overbeck, and Gregor Wiedemann. 2016. Modeling the dynamics of domain specific terminology in diachronic corpora. In Proceedings of the 12th International conference on Terminology and Knowledge Engineering (TKE 2016). find
  28. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. find our card
  29. Martin Hilpert and Stefan Th. Gries. 2009. Assessing frequency changes in multistage diachronic corpora: Applications for historical corpus linguistics and the study of language acquisition. Literary and Linguistic Computing, 24(4):385–401. find
  30. M. Hilpert. 2008. Germanic future constructions: A usage-based approach to language change. Benjamins, Amsterdam, Netherlands. find
  31. Patrick Juola. 2003. The time course of language change. Computers and the Humanities, 37(1):77–96. find
  32. David Jurgens and Keith Stevens. 2009. Event detection in blogs using Temporal Random Indexing. find
  33. Nobuhiro Kaji and Hayato Kobayashi. 2017. Incremental skip-gram model with negative sampling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 363–371, Copenhagen, Denmark. find
  34. Pentti Kanerva, Jan Kristofersson, and Anders Holst. 2000. Random indexing of text samples for latent semantic analysis. In Proceedings of the 22nd annual conference of the cognitive science society, volume 1036, pages 103–106, Mahwah, USA. find
  35. D. Kerremans, S. Stegmayr, and H.-J. Schmid. 2010. The neocrawler: Identifying and retrieving neologisms from the internet and monitoring ongoing change. In K. Allan and J. A. Robinson, editors, Current methods in historical semantics, pages 130–160. De Gruyter Mouton. find
  36. Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 61–65, Baltimore, USA. find
  37. Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, pages 625–635, Florence, Italy. find
  38. Andrey Kutuzov and Elizaveta Kuzmenko. 2018. Two centuries in two thousand words: Neural embedding models in detecting diachronic lexical changes. Quantitative Approaches to the Russian Language, pages 95–112. find
  39. Andrey Kutuzov, Elizaveta Kuzmenko, and Anna Marakasova. 2016. Exploration of register-dependent lexical semantics using word embeddings. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 26–34, Osaka, Japan. find
  40. Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2017a. Temporal dynamics of semantic relations in word embeddings: an application to predicting armed conflict participants. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1824–1829, Copenhagen, Denmark. find
  41. Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2017b. Tracing armed conflicts with diachronic word embedding models. In Proceedings of the Events and Stories in the News Workshop at ACL 2017, pages 31–36, Vancouver, Canada. find
  42. Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Proceedings of the 27th International Conference on Neural Information Processing Systems, pages 2177–2185, Montreal, Canada. find
  43. Xuanyi Liao and Guang Cheng. 2016. Analysing the semantic change based on word embedding. In Natural Language Understanding and Intelligent Applications, pages 213–223. Springer International Publishing. find
  44. Jefrey Lijffijt, Tanja Säily, and Terttu Nevalainen. 2012. CEECing the baseline: Lexical stability and significant change in a historical corpus. In Studies in Variation, Contacts and Change in English, volume 10. Research Unit for Variation, Contacts and Change in English (VARIENG). find
  45. Christopher D. Manning. 41(4):701–707. 2015. find
  46. Computational linguistics and deep learning. Computational Linguistics, Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Martin A. Nowak, and Erez Lieberman Aiden. 2011. Quan… find
  47. Rada Mihalcea and Vivi Nastase. 2012. Word epoch disambiguation: Finding how words change over time. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 259–263, Jeju Island, Korea. find
  48. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  49. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26:3111– 3119. find
  50. Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That’s sick dude!: Automatic identification of word sense change across different timescales. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1020–1029,… find
  51. Hannes Mueller and Christofer Rauh. 2017. Reading between the lines: Prediction of political violence using newspaper text. American Political Science Review, page 1–18. find
  52. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English Gigaword Fifth Edition LDC2011T07. Technical report, Technical Report. Linguistic Data Consortium, Philadelphia. find
  53. Hao Peng, Jianxin Li, Yangqiu Song, and Yaopeng Liu. 2017. Incrementally learning the hierarchical softmax function for neural language models. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3267–327, San Francisco, California USA. find
  54. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on Empirical Methods in Natural Language Processing, pages 1532–1543, Doha, Qatar. find
  55. Octavian Popescu and Carlo Strapparava. 2015. find
  56. SemEval 2015, task 7: Diachronic text evaluation. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 870–878, Denver, Colorado. find
  57. Matti Rissanen et al. 1993. The helsinki corpus of english texts. Kyttö et. al, pages 73–81. find
  58. Alex Rosenfeld and Katrin Erk. 2018. Deep neural models of semantic shift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 474–484, New Orleans, Louisiana, USA. find
  59. Guy D. Rosin, Eytan Adar, and Kira Radinsky. 2017. Learning word relatedness over time. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1179–1189, Copenhagen, Denmark. find
  60. Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2011. Tracing semantic change with latent semantic analysis. Current methods in historical semantics, pages 161–183. find
  61. Evan Sandhaus. 2008. The New York Times annotated corpus overview. Linguistic Data Consortium, Philadelphia, 6(12):e26752. find
  62. Gustaf Stern. 1931. Meaning and change of meaning; with special reference to the English language. Wettergren & Kerbers. find
  63. Terrence Szymanski. 2017. Temporal word analogies: Identifying lexical replacement with diachronic word embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 448–453, Vancouver, Canada. find
  64. Wayne A Taylor. 2000. Change-point analysis: a powerful new tool for detecting changes. find
  65. Elizabeth Closs Traugott and Richard B Dasher. 2001. Regularity in semantic change. Cambridge University Press. find
  66. Elizabeth Traugott. 2017. Semantic change. Oxford Research Encyclopedias: Linguistics. find
  67. Peter Turney, Patrick Pantel, et al. 2010. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research, 37(1):141–188. find
  68. Xuerui Wang and Andrew McCallum. 2006. Topics over time: a non-markov continuous-time model of topical trends. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 424–433, Philadelphia, PA, USA. find
  69. Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In Proceedings of the 2011 international workshop on Detecting and Exploiting Cultural diversity on the social web, pages 35–40. find
  70. Yang Xu and Charles Kemp. 2015. A computational evaluation of two laws of semantic change. In Proceedings of the 37th Annual Meeting of the Cognitive Science Society, Austin, TX, USA. find
  71. Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018. Dynamic word embeddings for evolving semantic discovery. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 673–681, Marina Del Rey, CA, USA. find
  72. Yating Zhang, Adam Jatowt, Sourav Bhowmick, and Katsumi Tanaka. 2015. Omnia mutantur, nihil interit: Connecting past with present by finding corresponding terms across time. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Con… find
  73. Yating Zhang, Adam Jatowt, Sourav S. Bhowmick, and Katsumi Tanaka. 2016. The past is not a foreign country: Detecting semantically similar terms across time. IEEE Transactions on Knowledge and Data Engineering, 28(10):2793–2807, October. find