Elamur Research⌕
☰
home / papers / 2026 / The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks

The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks

Summary

Lexical Semantic Change Detection (LSCD) is usually split into three stages. First, Word-in-Context (WiC) scores are assigned to pairs of word usages. Next, Word Sense Induction (WSI) clusters a usage graph built from those scores. Finally, the sense distributions of two time periods are compared. This modular setup has produced many incompatible task definitions, dataset versions, preprocessing choices and metrics, so results are hard to compare or reproduce.

The authors release an open benchmark repository, LSCDBenchmark. It standardizes evaluation across most existing human-annotated LSCD datasets in several languages and historical periods. It also scores the WiC and WSI subtasks on the same data, so model components can be mixed and assessed separately.

Using it, they test recent token-based models on the CoMeDi test split and report new state-of-the-art LSCD results. Their main finding is that WiC quality largely drives LSCD quality, and that models which more closely imitate the human annotation procedure tend to perform better. The benchmark gives the community a shared, reproducible testbed and links research on WiC, WSI and LSCD.

Key points

  • Code: github.com/Garrafao/LSCDBenchmark. It covers WiC (binary, graded and ordinal; scored with accuracy, Spearman/Pearson or Krippendorff's α), WSI as hard clustering, and LSCD as Binary Change and Graded Change (Jensen–Shannon Distance).
  • Datasets:
    • DWUG for DE, EN, SV, ES and IT, plus DiscoWUG and RefWUG
    • NorDiaChange1/2 and ChiWUG
    • DURel and SURel
    • RuSemShift1/2 and RuShiftEval1–3
  • Evaluation split: models are evaluated on the CoMeDi test split. This allows testing XL-DURel, which was fine-tuned on use pairs from many LSCD datasets. As a result, scores are only weakly comparable with earlier full-dataset results.
  • Findings:
    • Ordinal-trained WiC models match or beat binary-trained ones.
    • Discretizing graded predictions to ordinal values helps.
    • Older, less reliable dataset versions can mislead model selection.
    • Raw, unpreprocessed text generally works best, suggesting base embedders cope well with spelling variation.
  • Limitations: cluster-based models were not evaluated, and the small test split may cause sampling noise. Future plans include adding Slovenian and Japanese datasets and newer change metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.10

@inproceedings{schlechtweg2026lscd,
  title = {{The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks}},
  author = {Schlechtweg, Dominik and Yadav, Sachin and Kuhn, Jonas and Arefyev, Nikolay},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.10},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.10}
}

References

67 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adnan Ahmad, Kiflom Desta, Fabian Lang, and Dominik Schlechtweg. 2020. Shared task: Lexical semantic change detection in german. CoRR, arXiv:2001.07786. find
  2. N. Arefyev, M. Fedoseev, V. Protasov, D. Homskiy, A. Davletov, and A. Panchenko. 2021a. Deepmistake: Which senses are hard to distinguish for a word-in-context model. In Computational linguistics and intellectual technologies, 20, page 16 – 30, Russian Federation. find
  3. Nikolay Arefyev, Maksim Fedoseev, Vitaly Protasov, Daniil Homskiy, Adis Davletov, and Alexander Panchenko. 2021b. DeepMistake: Which senses are hard to distinguish for a word-in-context model. volume 2021-June, pages 16–30. find
  4. Carlos Santos Armendariz, Matthew Purver, Matej Ulčar, Senja Pollak, Nikola Ljubešić, and Mark Granroth-Wilding. 2020. CoSimLex: A resource for evaluating graded word similarity in context. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5878–5886, Marseille, Fran… find
  5. Gioia Baldissin, Dominik Schlechtweg, and Sabine Schulte im Walde. 2022. DiaWUG: A Dataset for Diatopic Lexical Semantic Variation in Spanish. In Proceedings of the 13th Language Resources and Evaluation Conference, Marseille, France. European Language Resources Association. find
  6. Nikhil Bansal, Avrim Blum, and Shuchi Chawla. 2004. Correlation clustering. Machine Learning, 56(1-3):89–113. find
  7. Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara. 2020. Overview of the EVALITA 2020 Diachronic Lexical Semantics (DIACR-Ita) Task. In Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020), On… find
  8. Christin Beck. 2020. DiaSense at SemEval-2020 Task 1: Modeling sense change via pre-trained BERT embeddings. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  9. Felix Blessing, Johannes S. Sax, Julian Kaufmann, Wei Zhao, Nikolay Arefyev, and Dominik Schlechtweg. 2026. APODICTUS: Automatic Processing Of DIC-Tionary Update candidateS. In LREC. find
  10. Pierluigi Cassotti, Pierpaolo Basile, and Nina Tahmasebi. 2024a. DWUGs-IT: Extending and standardizing lexical semantic change detection for Italian. In Proceedings of the 10th Italian Conference on Computational Linguistics, Pisa, Italy, December 4 - December 6, 2024, CEUR Workshop Proceedings. CEU… find
  11. Pierluigi Cassotti, Stefano De Pascale, and Nina Tahmasebi. 2024b. Using synchronic definitions and semantic relations to classify semantic change types. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4539–4553, Bangkok, Thai… find
  12. Pierluigi Cassotti, Lucia Siciliani, Marco de Gemmis, Giovanni Semeraro, and Pierpaolo Basile. 2023. Xl-lexeme: Wic pretrained model for cross-lingual lexical semantic change. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computat… find
  13. Jing Chen, Emmanuele Chersoni, Dominik Schlechtweg, Jelena Prokic, and Chu-Ren Huang. 2023. ChiWUG: A graph-based evaluation dataset for Chinese lexical semantic change detection. In Proceedings of the 4th International Workshop on Computational Approaches to Historical Language Change, Singapore. A… find
  14. Tejaswi Choppa, Michael Roth, and Dominik Schlechtweg. 2025. Predicting median, disagreement and noise label in ordinal Word-in-Context data. In Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation, pages 65–77, Abu Dhabi, UAE. International Committee on Computational Lingu… find
  15. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  16. Paul Cook, Jey Han Lau, Diana McCarthy, and Timothy Baldwin. 2014. Novel word-sense identification. In COLING, pages 1624–1635. ACL. 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. find
  17. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2013. Measuring word meaning in context. Computational Linguistics, 39(3):511–554. find
  18. Mariia Fedorova, Timothee Mickus, Niko Partanen, Janine Siewert, Elena Spaziani, and Andrey Kutuzov. 2024. AXOLOTL’24 shared task on multilingual explainable semantic change modeling. In Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change, pages 72–91, Bangkok,… find
  19. Mario Giulianelli, Marco del Tredici, and Raquel Fernández. 2020. Analysing lexical semantic change with contextualised word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960– 3973, Online. Association for Computational Linguistic… find our card
  20. Roksana Goworek and Haim Dubossarsky. 2026. Rethinking metrics for lexical semantic change detection. In The Proceedings for the 6th International Workshop on Computational Approaches to Language Change (LChange’26), pages 147–161, Rabat, Morocco. Association for Computational Linguistics. find
  21. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489– 1501, Berlin, Germany. find our card
  22. Anna Hätty, Dominik Schlechtweg, and Sabine Schulte im Walde. 2019. SURel: A gold standard for incorporating meaning shifts into term extraction. In Proceedings of the 8th Joint Conference on Lexical and Computational Semantics, pages 1–8, Minneapolis, MN, USA. find
  23. Daniil Homskiy and Nikolay Arefyev. 2022. DeepMistake at LSCDiscovery: Can a multilingual word-in-context model replace human annotators? In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 173– 179, Dublin, Ireland. Association for Computational Lingu… find
  24. Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882, Jeju Island, Korea.… find
  25. Jader Martins Camboim de Sá, Marcos Da Silveira, and Cédric Pruski. 2025. Survey in characterization of semantic change. Preprint, arXiv:2402.19088. — [arXiv:2402.19088](https://arxiv.org/abs/2402.19088) find
  26. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  27. Andres Karjus. 2025. Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence. Humanities and Social Sciences Communications, 12(1):1–18. find
  28. Adam Kilgarriff. 2007. Springer. find
  29. Word Senses, chapter 2. pages 3353–3359, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  30. Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology. SAGE Publications. find
  31. Suresh Manandhar and Ioannis Klapaftis. 2009. SemEval-2010 task 14: Evaluation setting for word sense induction & disambiguation systems. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-2009), pages 117–122, Boulder, Colorado. Association for Co… find
  32. Sinan Kurtyigit, Maike Park, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Lexical Semantic Change Discovery. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… find
  33. Andrey Kutuzov and Mario Giulianelli. 2020. UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguistics. find
  34. Andrey Kutuzov and Lidia Pivovarova. 2021. Rushifteval: a shared task on semantic shift detection for russian. Komp’yuternaya Lingvistika i Intellektual’nye Tekhnologii: Dialog conference. find
  35. Andrey Kutuzov, Samia Touileb, Petter Mæhlum, Tita Enstad, and Alexandra Wittemann. 2022. Nor-DiaChange: Diachronic semantic change dataset for Norwegian. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2563–2572, Marseille, France. European Language Resources As… find
  36. Severin Laicher. 2020. Historical word sense clustering with deep contextualized word embeddings. find
  37. Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Explaining and Improving BERT Performance on Lexical Semantic Change Detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Studen… find
  38. Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37:145–151. find
  39. Zhidong Ling, Taichi Aida, Teruaki Oka, and Mamoru Komachi. 2023. Construction of evaluation dataset for Japanese lexical semantic change detection. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 125– 136, Hong Kong, China. Association for Computat… find
  40. Daniel Loureiro, Aminette D’Souza, Areej Nasser Muhajab, Isabella A. White, Gabriel Wong, Luis Espinosa-Anke, Leonardo Neves, Francesco Barbieri, and Jose Camacho-Collados. 2022. TempoWiC: An evaluation benchmark for detecting meaning shift in social media. In Proceedings of the 29th International C… find
  41. Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2020. Leveraging contextual embeddings for detecting diachronic semantic shift. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4811–4819, Marseille, France. European Language Resources Association. find
  42. Syrielle Montariol, Matej Martinc, and Lidia Pivovarova. 2021. Scalable and interpretable semantic change detection. In 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  43. Alp Mujko. 2026. Bridging the thresholding gap: Ordinal classification for word-in-context using cumulative link models. Master thesis, University of Stuttgart. find
  44. Francesco Periti and Stefano Montanelli. 2024. Lexical semantic change through large language models: a survey. ACM Comput. Surv., 56(11). find
  45. Francesco Periti and Nina Tahmasebi. 2024. A systematic comparison of contextualized word embeddings for lexical semantic change. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),… find
  46. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find our card
  47. Marko Pranjić, Kaja Dobrovoljc, Senja Pollak, and Matej Martinc. 2025. Tracking semantic change in slovene: A novel dataset and optimal transport-based distance. Preprint, arXiv:2402.16596. — [arXiv:2402.16596](https://arxiv.org/abs/2402.16596) find
  48. Julia Rodina and Andrey Kutuzov. 2020. RuSemShift: a dataset of historical lexical semantic change in Russian. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1037– 1047, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  49. Dominik Schlechtweg. 2023. Human and Computational Measurement of Lexical Semantic Change. Stuttgart, Germany. find
  50. Dominik Schlechtweg, Pierluigi Cassotti, Bill Noble, David Alfter, Sabine Schulte im Walde, and Nina Tahmasebi. 2024a. More DWUGs: Extending and evaluating Word Usage Graph datasets in multiple languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page… find
  51. Dominik Schlechtweg, Tejaswi Choppa, Wei Zhao, and Michael Roth. 2025. CoMeDi shared task: Median judgment classification & mean disagreement ranking with ordinal Word-in-Context judgments. In Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation, pages 33–47, Abu Dhabi, UAE… find
  52. Dominik Schlechtweg, Anna Hätty, Marco del Tredici, and Sabine Schulte im Walde. 2019. A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732–746, Florence, Ita… find
  53. Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. In Proceedings of the 14th International Workshop on Semantic Evaluation, Barcelona, Spain. Association for Computational Linguis… find our card
  54. Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanie Eckmann. 2018. Diachronic Usage Relatedness (DURel): A framework for the annotation of lexical semantic change. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  55. Zhang, Nina Tahmasebi, Jonas Kuhn, and Sabine Schulte im Walde. 2024b. The DURel annotation tool: Human and computational measurement of semantic proximity, sense clusters and semantic change. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguist… find
  56. Dominik Schlechtweg, Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Nikolay Arefyev. 2024c. Sense through time: Diachronic word sense annotations for word sense induction and lexical semantic change detection. Language Resources and Evaluation. find
  57. Hinrich Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–123. find our card
  58. Emma Sköldberg, Shafqat Mumtaz Virk, Pauline Sander, Simon Hengchen, and Dominik Schlechtweg. 2024. Revealing semantic variation in Swedish using computational models of semantic proximity: Results from lexicographical experiments. In Proceedings of the 21st EURALEX International Congress Lexicograp… find
  59. Charles Spearman. 1904. The proof and measurement of association between two things. American Journal of Psychology, 15:88–103. find
  60. Sean Trott and Benjamin Bergen. 2021. RAW-C: Relatedness of ambiguous words in context (a new lexical resource for English). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… find
  61. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computationa… find our card
  62. Nash Whaley. 2025. Human and computational measurement of semantic relations. Master thesis, University of Stuttgart. find
  63. Dominik Schlechtweg, Nina Tahmasebi, Simon Hengchen, Haim Dubossarsky, and Barbara McGillivray. 2021. DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7079–7091, Online and Punta… find
  64. Sachin Yadav and Dominik Schlechtweg. 2025. XL-DURel: Finetuning sentence transformers for ordinal word-in-context classification. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Comput… find
  65. Dominik Schlechtweg, Shafqat Mumtaz Virk, Pauline Sander, Emma Sköldberg, Lukas Theuer Linke, Tuo Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022. LSCDiscovery: A shared task on semantic change discovery and detection in Spanish. In Proceedings of the 3rd International Wor… find
  66. Frank D. Zamora-Reina, Felipe Bravo-Marquez, Dominik Schlechtweg, and Nikolay Arefyev. 2025. Can large language models compete with specialized models in lexical semantic change detection? In Proceedings of the European Conference on Artificial Intelligence (ECAI). find
  67. Dataset MCL MCL→RSS ∆ chiwug_100 discowug_200 durel_300 dwug_de_300 dwug_de_resampled_100 dwug_en_300 dwug_en_resampled_100 dwug_es_402 dwug_sv_300 dwug_sv_resampled_100 nor_dia_change_1_101 nor_dia_change_2_101 rushifteval1_200 rushifteval2_200 rushifteval3_200 surel_300 0.71 0.62 0.67 0.69 0.49 0.… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schlechtweg-2026-lscd-benchmark-testbed-diachronic-word/