Elamur Research
☰
home / papers / 2020 / SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection

SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.semeval-1.1.pdf

Summary

Research on lexical semantic change (LSC), meaning how word meanings shift over time, has had no shared gold standard, so published models could not be compared fairly. This SemEval-2020 shared task fills that gap. It provides an unsupervised evaluation setup with two subtasks. The first is a binary decision on whether a target word gained or lost a sense between two time-specific corpora. The second ranks target words by how strongly they changed. The organisers built pairs of corpora for English, German, Latin and Swedish. Target words were drawn from historical dictionaries and balanced against stable control words of matching POS and frequency. Gold labels come from about 100,000 human relatedness judgments made with an extended DURel framework. In total, 33 teams entered 186 systems. Neither subtask was solved. Static type embeddings unexpectedly beat contextual token embeddings, though their outputs were strongly biased by word frequency. The benchmark matters because it lets researchers test LSC models consistently across languages.

Key points

  • Corpora: The data comes from CCOHA (English), DTA versus BZ+ND (German), LatinISE (Latin) and Kubhist (Swedish). For copyright reasons, the released texts are lemmatised and their sentences shuffled.
  • Annotation: For each target word, up to 100 uses per period were sampled. Annotators rated pairs of uses on the 4-point DURel scale, from unrelated (1) to identical (4). There were 9 annotators for English, 8 for German and 5 for Swedish, all native speakers.
  • No training data: Participants received no training data, and the trial labels were random. For development, they were pointed to the existing datasets DURel, SemCor LSC and WSC.
  • Frequency bias: Type embeddings beat token embeddings on both subtasks, but some models' predictions correlated with frequency at above 0.8.
  • Polysemy: Gold change scores correlate substantially with polysemy, while model predictions are only moderately biased toward it. The authors suggest controlling future test sets for polysemy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

176 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.semeval-1.1

@inproceedings{schlechtweg2020semeval,
  title = {{SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection}},
  author = {Schlechtweg, Dominik and McGillivray, Barbara and Hengchen, Simon and Dubossarsky, Haim and Tahmasebi, Nina},
  booktitle = {Proceedings of the Fourteenth Workshop on Semantic Evaluation},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.1},
  url = {https://doi.org/10.18653/v1/2020.semeval-1.1}
}

References

85 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yvonne Adesam, Dana Dannélls, and Nina Tahmasebi. 2019. Exploring the quality of the digital historical newspaper archive KubHist. In Proceedings of the 2019 DHN conference, pages 9–17. find
  2. Reem Alatrash, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2020. CCOHA: Clean Corpus of Historical American English. In Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC’20). European Language Resources Association (ELRA). find our card
  3. Efrat Amar and Chaya Liebeskind. 2020. JCT at SemEval-2020 Task 1: Combined Semantic Vector Spaces Models for Unsupervised Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  4. Asaf Amrami and Yoav Goldberg. 2018. Word sense induction with neural bilm and symmetric patterns. arXiv preprint arXiv:1808.08518. find
  5. Asaf Amrami and Yoav Goldberg. 2019. Towards better substitution-based word sense induction. arXiv preprint arXiv:1905.12598. — [arXiv:1905.12598](https://arxiv.org/abs/1905.12598) find
  6. Nikolay Arefyev and Vasily Zhikov. 2020. BOS at SemEval-2020 Task 1: Word Sense Induction via Lexical Substitution for Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  7. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia… find our card
  8. Ehsaneddin Asgari, Christoph Ringlstetter, and Hinrich Schütze. 2020. EmbLexChange at SemEval-2020 Task 1: Unsupervised Embedding-based Detection of Lexical Semantic Changes. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Li… find
  9. Nikhil Bansal, Avrim Blum, and Shuchi Chawla. 2004. Correlation clustering. Machine Learning, 56(1-3):89– 113. find
  10. Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2015. Temporal random indexing: A system for analysing word meaning over time. Italian Journal of Computational Linguistics, 1(1):55–68. find
  11. Christin Beck. 2020. DiaSense at SemEval-2020 Task 1: Modeling sense change via pre-trained BERT embeddings. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  12. Berliner Zeitung. 2018. Diachronic newspaper corpus published by Staatsbibliothek zu Berlin. find
  13. Chris Biemann. 2006. Chinese whispers: An efficient graph clustering algorithm and its application to natural language processing problems. In Proceedings of the First Workshop on Graph Based Methods for Natural Language Processing, TextGraphs-1, page 73–80, USA. Association for Computational Lingui… find
  14. Chris Biemann. 2013. Creating a system for lexical substitutions from scratch using crowdsourcing. Lang. Resour. Eval., 47(1):97–122, March. find
  15. Andreas Blank. 1997. Prinzipien des lexikalischen Bedeutungswandels am Beispiel der romanischen Sprachen. Niemeyer, Tübingen. find
  16. Vincent D. Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10):10008, October. find
  17. Lars Borin, Markus Forsberg, and Johan Roxendal. 2012. Korp — the corpus infrastructure of Språkbanken. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 474–478, Istanbul, Turkey, May. European Language Resources Association (ELRA). find
  18. Pierluigi Cassotti, Annalina Caputo, Marco Polignano, and Pierpaolo Basile. 2020. GM-CTSC at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal Similarity Clustering. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguist… find
  19. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better ud parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55–64. find
  20. James Clackson. 2011. A Companion to the Latin Language. Wiley-Blackwell. find
  21. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  22. Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel word-sense identification. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1624–1635. find
  23. Mark Davies. 2012. Expanding Horizons in Historical Linguistics with the 400-Million Word Corpus of Historical American English. Corpora, 7(2):121–157. find
  24. Deutsches Textarchiv. 2017. Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache. Herausgegeben von der Berlin-Brandenburgischen Akademie der Wissenschaften. find
  25. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  26. Gonzalo Donoso and David Sanchez. 2017. Dialectometric analysis of language variation in Twitter. CoRR, abs/1702.06777. find
  27. Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017. Outta control: Laws of semantic change and inherent biases in word representation models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1147–1156, Copenhagen, Denmark. find
  28. Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457–470, Florence, Italy. Association for… find
  29. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2013. Measuring word meaning in context. Computational Linguistics, 39(3):511–554. find
  30. Brendan J. Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. Science, 315(5814):972–976. find
  31. Maurı́cio Gruppi, Sibel Adalı, and Pin-Yu Chen. 2020. SChME at SemEval-2020 Task 1: A Model Ensemble for Detecting Lexical Semantic Change. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  32. Amaru Cuba Gyllensten, Evangelia Gogoulou, Ariel Ekgren, and Magnus Sahlgren. 2020. SenseCluster at SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguisti… find
  33. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501, Berlin, Germany. find our card
  34. Anna Hätty, Dominik Schlechtweg, and Sabine Schulte im Walde. 2019. SURel: A gold standard for incorporating meaning shifts into term extraction. In Proceedings of the 8th Joint Conference on Lexical and Computational Semantics, pages 1–8, Minneapolis, MN, USA. find
  35. Genevieve Hayes. 2019. mlrose: Machine Learning, Randomized Optimization and SEarch package for Python. https://github.com/gkhayes/mlrose. Accessed: May 22, 2020. ↗
  36. Simon Hengchen, Ruben Ros, Jani Marjanen, and Mikko Tolonen. to appear. A data-driven approach to studying changing vocabularies in historical newspaper collections. Digital Scholarship in the Humanities. find
  37. Mark J Hill and Simon Hengchen. 2019. Quantifying the impact of dirty OCR on historical text analysis: Eighteenth Century Collections Online as a case study. Digital Scholarship in the Humanities, 34(4):825–843. find
  38. Ran Iwamoto and Masahiro Yukawa. 2020. RIJP at SemEval-2020 Task 1: Gaussian-based Embeddings for Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  39. Vaibhav Jain. 2020. GloVeInit at SemEval-2020 Task 1: Using GloVe Vector Initialization for Unsupervised Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  40. Vani K, Sandra Mitrović, Alessandro Antonucci, and Fabio Rinaldi. 2020. SST-BERT at SemEval-2020 Task 1: Semantic Shift Tracing by Clustering in BERT-based Embedding Spaces. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Lin… find
  41. Jens Kaiser, Dominik Schlechtweg, Sean Papay, and Sabine Schulte im Walde. 2020. IMS at SemEval-2020 Task 1: How low can you go? Dimensionality in Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computatio… find
  42. Anna Karnysheva and Pia Schwarz. 2020. TUE at SemEval-2020 Task 1: Detecting semantic change by clustering contextual word embeddings. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  43. Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of language through neural language models. In LTCSS@ACL, pages 61–65. Association for Computational Linguistics. find
  44. Andrey Kutuzov and Mario Giulianelli. 2020. UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  45. Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: A survey. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384–1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics… find our card
  46. Jey Han Lau, Paul Cook, Diana McCarthy, Spandana Gella, and Timothy Baldwin. 2014. Learning word sense distributions, detecting unattested senses and identifying novel senses using topic models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  47. Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  48. Matej Martinc, Syrielle Montariol, Elaine Zosa, and Lidia Pivovarova. 2020. Discovery Team at SemEval-2020 Task 1: Context-sensitive Embeddings not Always Better Than Static for Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Ass… find
  49. Diana McCarthy, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in untagged text. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume, pages 279–286, Barcelona, Spain, July. find our card
  50. Barbara McGillivray and Adam Kilgarriff. 2013. Tools for historical corpus research, and a corpus of Latin. In Paul Bennett, Martin Durrell, Silke Scheible, and Richard J. Whitt, editors, New Methods in Historical Corpus Linguistics, Tübingen. Narr. find
  51. Barbara McGillivray, Simon Hengchen, Viivi Lähteenoja, Marco Palma, and Alessandro Vatri. 2019. A computational approach to lexical polysemy in Ancient Greek. Digital Scholarship in the Humanities, 34(4):893–907. find
  52. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceed… find
  53. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. find
  54. Neues Deutschland. 2018. Diachronic newspaper corpus published by Staatsbibliothek zu Berlin. find
  55. Paul Nulty and David Lillis. 2020. The UCD-Net System at SemEval-2020 Task 1: Temporal Referencing with Semantic Network Distances. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  56. OED. 2009. Oxford English Dictionary. Oxford University Press. find
  57. Gergely Palla, Albert-László Barabási, and Tamás Vicsek. 2007. Quantifying social group evolution. Nature, 446(7136):664–667. find
  58. Hermann Paul. 2002. Deutsches Wörterbuch: Bedeutungsgeschichte und Aufbau unseres Wortschatzes. Niemeyer, Tübingen, 10. edition. find
  59. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics. find
  60. Valerio Perrone, Marco Palma, Simon Hengchen, Alessandro Vatri, Jim Q. Smith, and Barbara McGillivray. 2019. GASC: Genre-aware semantic change for Ancient Greek. In Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change, pages 56–66, Florence, Italy,… find
  61. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  62. Martin Pincus. 1970. A monte carlo method for the approximate solution of certain types of constrained optimization problems. Operations Research, 18(6):1225–1228. find
  63. Michael Piotrowski. 2012. Natural Language Processing for Historical Texts. Morgan & Claypool Publishers, San Rafael, California. find
  64. Martin Pömsl and Roman Lyapin. 2020. CIRCE at SemEval-2020 Task 1: Ensembling Context-Free and Context-Dependent Word Representations. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  65. Ondřej Pražák, Pavel Přibáň, Stephen Taylor, and Jakub Sido. 2020. UWB at SemEval-2020 Task 1: Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  66. David Rother, Thomas Haider, and Steffen Eger. 2020. CMCE at SemEval-2020 Task 1: Clustering on Manifolds of Contextualized Embeddings to Detect Historical Meaning Shifts. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Lingui… find
  67. Gerard Salton and Michael J McGill. 1983. Introduction to Modern Information Retrieval. McGraw - Hill Book Company, New York. find
  68. Eleri Sarsfield and Harish Tayyar Madabushi. 2020. UoB at SemEval-2020 Task 1: Automatic Identification of Novel Word Senses. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  69. Dominik Schlechtweg and Sabine Schulte im Walde. 2020. Simulating lexical semantic change from sense-annotated data. In A. Ravignani, C. Barbieri, M. Martins, M. Flaherty, Y. Jadoul, E. Lattenkamp, H. Little, K. Mudd, and T. Verhoef, editors, The Evolution of Language: Proceedings of the 13th Intern… find
  70. Dominik Schlechtweg, Stefanie Eckmann, Enrico Santus, Sabine Schulte im Walde, and Daniel Hole. 2017. German in flux: Detecting metaphoric change via word entropy. In Proceedings of the 21st Conference on Computational Natural Language Learning, pages 354–367, Vancouver, Canada. find
  71. Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Eckmann. 2018. Diachronic Usage Relatedness (DURel): A framework for the annotation of lexical semantic change. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  72. Dominik Schlechtweg, Anna Hätty, Marco del Tredici, and Sabine Schulte im Walde. 2019. A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732–746, Florence, It… find
  73. Hinrich Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–123, March. find our card
  74. Philippa Shoemark, Farhana Ferdousi Liza, Dong Nguyen, Scott Hale, and Barbara McGillivray. 2019. Room to glo: A systematic comparison of semantic change detection approaches with word embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th I… find
  75. Språkbanken, Downloaded in 2019. The Kubhist Corpus, v2. Department of Swedish, University of Gothenburg. find
  76. Svenska Akademien. 2009. Contemporary dictionary of the Swedish Academy. The changed words are extracted from a database managed by the research group that develops the Contemporary dictionary. find
  77. Nina Tahmasebi and Thomas Risse. 2017. Finding individual word sense changes and their delay in appearance. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 741– 749, Varna, Bulgaria. find
  78. Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2018. Survey of computational approaches to lexical semantic change. In Preprint at ArXiv 2018. find
  79. Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David M Blei. 2004. Sharing clusters among related groups: Hierarchical dirichlet processes. In Proceedings of the 17th International Conference on Neural Information Processing Systems, pages 1385–1392. MIT Press. find
  80. Daniel van Strien, Kaspar Beelen, Mariona Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colavizza. 2020. Assessing the impact of OCR quality on downstream NLP tasks. In ICAART (1), pages 484–496. find
  81. Luke Vilnis and Andrew McCallum. 2015. Word representations via Gaussian embedding. In ICLR. find
  82. Han Xiao. 2018. bert-as-service. https://github.com/hanxiao/bert-as-service. ↗
  83. Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018. Dynamic word embeddings for evolving semantic discovery. In WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining, volume 2018-Febua, pages 673–681. find
  84. Frank D. Zamora-Reina and Felipe Bravo-Marquez. 2020. DCC-Uchile at SemEval-2020 Task 1: Temporal Referencing Word Embeddings. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  85. Jinan Zhou and Jiaxin Li. 2020. TemporalTeller at SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection with Temporal Referencing. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find