Elamur Research⌕
☰
home / papers / 2016 / A Bayesian Model of Diachronic Meaning Change

A Bayesian Model of Diachronic Meaning Change

Paper · 2016 · Lea Frermann and Mirella Lapata

Summary

Frermann and Lapata address the automatic tracking of how word meanings evolve over history, which can support lexicography as well as information retrieval and question answering. They introduce SCAN, a dynamic Bayesian model trained separately for each target word. Its input is a set of short, year-stamped snippets containing the target word and its nearby context. Time is divided into discrete intervals. For each interval, the model learns how prevalent each of a fixed number of senses is, and it represents each sense as a distribution over vocabulary words. Earlier approaches learned each period in isolation. SCAN instead links neighbouring intervals through logistic normal priors, so representations drift smoothly and sparse periods borrow information from adjacent ones. The induced senses are interpretable. They reveal emerging meanings, such as the computer-device sense of mouse, and subtler shifts within a single sense. The model is competitive at detecting meaning change. On the SemEval-2015 diachronic text evaluation, its features match heavily tuned task-specific systems. It needs no parser or external resources, only a time-stamped corpus.

Key points

  • Model inputs and priors: SCAN takes a ±5-word symmetric context window around each target mention. Temporal smoothness comes from iGMRF-style dependencies between adjacent time slices. A Gamma-distributed precision parameter controls how quickly sense prevalence may change.
  • Evaluation:
    • The model is tested on meaning change detection across many intervals, not just two.
    • Target words are sampled from the corpus or taken from the Gulordava and Baroni (2011) judgment study.
    • Results are validated against WordNet, following the protocol of Mitra et al. (2014).
  • SemEval-2015 DTE results:
    • In subtask 1, both SCAN and the ablated SCAN-NOT (no iGMRFs) beat all participating systems.
    • The full model achieves consistently better accuracy, the metric that accounts for distance from the true interval.
  • Subtask 2 comparison: An SVM combining SCAN features with character n-grams (n = 1–3, kept if seen more than 20 times) performs comparably to UCD (Szymanski and Lynch, 2015), the best published system. Unlike UCD, it uses no feature selection.
  • Limitations noted by the authors:
    • The number of senses K is fixed across words and periods, although more than 800 target concepts were handled in SemEval.
    • Context is treated as a bag of words.
    • The model cannot detect words that appear or vanish over time.
    • Proposed extensions include a non-parametric variant and applications to Twitter data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Historical linguistics & language change · 159 works

Cite

Cite as Lea Frermann and Mirella Lapata. 2016. A Bayesian Model of Diachronic Meaning Change. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00081

@article{frermann2016bayesian,
  title = {{A Bayesian Model of Diachronic Meaning Change}},
  author = {Lea Frermann and Mirella Lapata},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00081},
  url = {https://doi.org/10.1162/tacl_a_00081}
}

References

39 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aitchison, Jean. 2001. Language Change: Progress Or Decay?. Cambridge Approaches to Linguistics. Cambridge University Press. find
  2. Biemann, Chris. 2006. Chinese Whispers - an Efficient Graph Clustering Algorithm and its Application to Natural Language Processing Problems. In Proceedings of TextGraphs: the 1st Workshop on Graph Based Methods for Natural Language Processing. New York City, NY, USA, pages 73– 80. our card
  3. Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media, Inc., 1st edition. find
  4. Blei, David M. and John D. Lafferty. 2006a. Correlated Topic Models. In Advances in Neural Information Processing Systems 18, Vancouver, BC, Canada, pages 147–154. find
  5. Blei, David M. and John D. Lafferty. 2006b. Dynamic Topic Models. In Proceedings of the 23rd International Conference on Machine Learning. Pittsburgh, PA, USA, pages 113–120. our card
  6. Brody, Samuel and Mirella Lapata. 2009. Bayesian Word Sense Induction. In Proceedings of the 12th Conference of the European Chapter of the ACL. Athens, Greece, pages 103–111. our card
  7. Chambers, Nathanael. 2012. Labeling Documents with Timestamps: Learning from their Time Expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Jeju Island, Korea, pages 98–106. find
  8. Chang, Chih-Chung and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2:27:1–27:27. Software available at http://www.csie.ntu.edu. tw/˜cjlin/libsvm. ↗
  9. Chen, Jianfei, Jun Zhu, Zi Wang, Xun Zheng, and Bo Zhang. 2013. Scalable Inference for Logistic-Normal Topic Models. In Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, pages 2445–2453. find
  10. Cook, Paul, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel Word-sense Identification. In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers. Dublin, Ireland, pages 1624– 1635. find
  11. Cook, Paul and Suzanne Stevenson. 2010. Automatically Identifying Changes in the Semantic Orientation of Words. In Proceedings of the Seventh International Conference on Language Resources and Evaluation. Valletta, Malta, pages 28–34. find
  12. Davies, Mark. 2010. The Corpus of Historical American English: 400 million words, 1810-2009. Available online at http://corpus.byu.edu/coha/. ↗
  13. Diller, Hans-Jürgen, Hendrik de Smet, and Jukka Tyrkkö. 2011. A European database of descriptors of english electronic texts. The European English messenger 19(2):29–35. find
  14. Fellbaum, Christiane. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  15. Groenewald, Pieter C. N. and Lucky Mokgatlhe. 2005. Bayesian Computation for Logistic Regression. Computational Statistics & Data Analysis 48(4):857–868. find
  16. Gulordava, Kristina and Marco Baroni. 2011. A Distributional Similarity Approach to the Detection of Semantic Change in the Google Books Ngram Corpus. In Proceedings of the Workshop on GEometrical Models of Natural Language Semantics. Edinburgh, Scotland, pages 67–71. our card
  17. Kilgarriff, Adam. 2009. Simple maths for keywords. In Proceedings of the Corpus Linguistics Conference. find
  18. Kim, Yoon, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal Analysis of Language through Neural Language Models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science. Baltimore, MD, USA, pages 61–65. our card
  19. Kulkarni, Vivek, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically Significant Detection of Linguistic Change. In Proceedings of the 24th International Conference on World Wide Web. Geneva, Switzerland, pages 625–635. find
  20. Lau, Han Jey, Paul Cook, Diana McCarthy, Spandana Gella, and Timothy Baldwin. 2014. Learning Word Sense Distributions, Detecting Unattested Senses and Identifying Novel Senses using Topic Models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Baltimore, M… find
  21. Lau, Jey Han, Paul Cook, Diana McCarthy, David Newman, and Timothy Baldwin. 2012. Word Sense Induction for Novel Sense Detection. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Avignon, France, pages 591– 601. our card
  22. McMahon, April M.S. 1994. Understanding Language Change. Cambridge University Press. find
  23. Mihalcea, Rada and Vivi Nastase. 2012. Word Epoch Disambiguation: Finding How Words Change over Time. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Jeju Island, Korea, pages 259– 263. find
  24. Mimno, David, Hanna Wallach, and Andrew Mc-Callum. 2008. Gibbs Sampling for Logistic Normal Topic Models with Graph-Based Priors. In NIPS Workshop on Analyzing Graphs. Vancouver, Canada. find
  25. Mitra, Sunny, Ritwik Mitra, Suman Kalyan Maity, Martin Riedl, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2015. An automatic approach to identify word sense changes in text media across timescales. Natural Language Engineering 21:773–798. find
  26. Mitra, Sunny, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal. 2014. That’s sick dude!: Automatic identification of word sense change across different timescales. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Baltimore, MD,… find
  27. Ó Séaghdha, Diarmuid. 2010. Latent Variable Models of Selectional Preference. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Uppsala, Sweden, pages 435–444. find
  28. Popescu, Octavian and Carlo Strapparava. 2013. Behind the Times: Detecting Epoch Changes using Large Corpora. In Proceedings of the Sixth International Joint Conference on Natural Language Processing. Nagoya, Japan, pages 347–355. find
  29. Popescu, Octavian and Carlo Strapparava. 2015. find
  30. SemEval 2015, Task 7: Diachronic Text Evaluation. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). Denver, CO, USA, pages 869–877. find
  31. Ritter, Alan, Mausam, and Oren Etzioni. 2010. A Latent Dirichlet Allocation Method for Selectional Preferences. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Uppsala, Sweden, pages 424– 434. find
  32. Rue, Håvard and Leonhard Held. 2005. Gaussian Markov Random Fields: Theory and Applications. Chapman & Hall/CRC Monographs on Statistics & Applied Probability. CRC Press. find
  33. Sagi, Eyal, Stefan Kaufmann, and Brady Clark. 2009. Semantic Density Analysis: Comparing Word Meaning across Time and Phonetic Space. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics. Athens, Greece, pages 104–111. find
  34. Salaberri, Haritz, Iker Salaberri, Olatz Arregi, and Beñat Zapirain. 2015. IXAGroupEHUDiac: A Multiple Approach System towards the Diachronic Evaluation of Texts. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). Denver, CO, USA, pages 840–845. find
  35. Stevenson, Angus, editor. 2010. The Oxford English Dictionary. Oxford University Press, third edition. find
  36. Szymanski, Terrence and Gerard Lynch. 2015. UCD: Diachronic Text Classification with Character, Word, and Syntactic N-grams. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). Denver, CO, USA, pages 879–883. find
  37. Tahmasebi, Nina, Thomas Risse, and Stefan Dietze. 2011. Towards automatic language evolution tracking, A study on word sense tracking. In Proceedings of the Joint Workshop on Knowledge Evolution and Ontology Dynamics (EvoDyn 2011). Bonn, Germany. find
  38. Wijaya, Derry Tanti and Reyyan Yeniterzi. 2011. Understanding Semantic Change of Words over Centuries. In Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural diversiTy on the Social Web. Glasgow, Scotland, UK, pages 35–40. find
  39. Zampieri, Marcos, Alina Maria Ciobanu, Vlad Niculae, and Liviu P. Dinu. 2015. AMBRA: A Ranking Approach to Temporal Text Classification. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). Denver, CO, USA, pages 851–855. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/frermann-2016-bayesian-model-diachronic-meaning-change/