Elamur Research⌕
☰
home / papers / 2018 / Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change

Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change

Summary

Computational models of lexical semantic change are hard to evaluate because no language had a reliable benchmark beyond a few hand-picked examples. The authors propose DURel, an annotation framework that adapts graded synchronic polysemy annotation to diachronic data. Annotators never define or label word senses. They rate how related two uses of the same word are on a 4-point scale adapted from Brown (2008). Use pairs are drawn from an earlier period, a later period, and a mixed COMPARE group that pairs one use from each. Change is read from shifts in mean relatedness:

  • A drop from the earlier to the later period signals a new meaning.
  • A rise signals a lost meaning.
  • Low COMPARE relatedness signals strong change of either kind.

Applied to German texts from DTA, the framework produced a test set of 1,320 rated use pairs for 22 target words. Inter-annotator agreement was good, and the measures broadly matched the changes reported in the historical literature. The framework is language-independent, so it offers a reusable recipe for building semantic change benchmarks for other languages.

Key points

  • Data and design. Five native German speakers, all linguistics students, rated 1,320 use pairs for 22 target words. Each word had 20 pairs in each of three groups: EARLIER (1750–1800), LATER (1850–1900) and COMPARE. Uses came from DTA (Deutsches Textarchiv).
  • Scale. The scale runs from 4 (Identical) down to 1 (Unrelated), with 0 for "Cannot decide". Each use was shown with its preceding and following sentence.
  • Agreement. Mean pairwise Spearman's ρ was 0.66, with 0-judgments omitted and pairwise values ranging from 0.57 to 0.68. This is slightly above the 0.55–0.62 reported by Erk et al. (2013) for synchronic data. The two annotators with a historical-linguistics background agreed most closely.
  • Target words. Targets were chosen from changes documented by Paul (2002) and Osman (1971), checked against the corpus, giving 19 innovative and 9 reductive cases. Anstalt, Anstellung and Vorwort undergo both kinds of change and appear more than once.
  • Measures.
    • Mean ΔLATER was 0.39 for reductive cases and −0.18 for innovative ones.
    • ΔLATER failed for Presse, whose new meaning became dominant, but COMPARE caught that change.
    • COMPARE wrongly flagged the stably polysemous Feder as changed.
    • Guidelines and data are released at www.ims.uni-stuttgart.de/data/durel/.

Interesting ideas

  • I1. Rate relatedness instead of assigning senses. Graded relatedness judgments between two uses sidestep the need for a sense inventory, which is often missing for historical stages or low-resource languages. This would suit building semantic resources for Ukrainian, where fine-grained historical dictionaries may be incomplete.
  • I2. Add a cross-period comparison group. Pairing one early and one late use measures change directly. It does not rely on the old meaning still being present, which helps when a meaning is fully replaced. The same trick applies to comparing domains or genres, not only time periods.
  • I3. Normalize by baseline polysemy. The proposed ΔCOMPARE subtracts within-early relatedness from cross-period relatedness, so a word that is simply polysemous is not mistaken for one that changed. This idea transfers to any drift-detection metric that confuses inherent variability with genuine shift.
  • I4. Choose targets with a known spread of effects. Selecting words whose documented change is visible in the corpus, both weakly and strongly, keeps a test set from collapsing into many near-zero effects. This makes rankings more informative when comparing models.
  • I5. Hide the study's purpose and shuffle everything. Annotators were not told the study concerned semantic change. Uses within pairs and pairs across groups were randomized, which reduces expectation bias. This is a cheap safeguard for any annotation project measuring differences between conditions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-2027

@inproceedings{schlechtweg2018diachronic,
  title = {{Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change}},
  author = {Schlechtweg, Dominik and Walde, Sabine Schulte im and Eckmann, Stefanie},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2027},
  url = {https://doi.org/10.18653/v1/n18-2027}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bamman and G. Crane. 2011. Measuring historical word sense variation. In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries. ACM, New York, NY, USA, pages 1–10. find
  2. J. H. Lau, P. Cook, D. McCarthy, D. Newman, and T. Baldwin. 2012. Word sense induction for novel sense detection. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Stroudsburg, PA, USA, pages 591–601. find
  3. A. Blank. 1997. Prinzipien des lexikalischen Bedeutungswandels am Beispiel der romanischen Sprachen. Niemeyer, Tübingen. find
  4. S. W. Brown. 2008. Choosing sense distinctions for WSD: Psycholinguistic evidence. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers. Stroudsburg, PA, USA, pages 249–252. find
  5. N. Osman. 1971. Kleines Lexikon untergegangener Wörter: Wortuntergang seit dem Ende des 18. Jahrhunderts. Beck, München. find
  6. H. Paul. 2002. Deutsches Wörterbuch: Bedeu-tungsgeschichte und Aufbau unseres Wortschatzes. Niemeyer, Tübingen, 10. edition. find
  7. P. Cook, J. H. Lau, D. McCarthy, and T. Baldwin. 2014. Novel word-sense identification. In 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland. pages 1624–1635. find
  8. E. Rosch and C.B. Mervis. 1975. Family resemblances: Studies in the internal structure of categories. Cognitive Psychology 7:573–605. find
  9. Deutsches Textarchiv. 2017. Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache. Herausgegeben von der Berlin-Brandenburgischen Akademie der Wissenschaften. http://www.deutschestextarchiv.de/. ↗
  10. D. Schlechtweg, S. Eckmann, E. Santus, S. Schulte im Walde, and D. Hole. 2017. German in flux: Detecting metaphoric change via word entropy. In Proceedings of the 21st Conference on Computational Natural Language Learning. Vancouver, Canada, pages 354–367. find
  11. Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017. Outta control: Laws of semantic change and inherent biases in word representation models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark, pages 1147–1156. find
  12. A. Soares da Silva. 1992. Homonı́mia e polissemia: Análise sémica e teoria do campoléxico. In Actas do XIX Congreso Internacional de Lingüı́stica e Filoloxı́a Románicas. Fundación Pedro Barrié de la Maza, La Coruña, volume 2 of Lexicoloxı́a e Met-alexicografı́a, pages 257–287. find
  13. K. Erk, D. McCarthy, and N. Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1. Stroudsburg, PA, USA, pages 10–18. find
  14. David Tuggy. 1993. Ambiguity, polysemy, and vagueness. Cognitive Linguistics 4(3):273–290. find
  15. J. Zlatev. 2003. Polysemy or generality? Mu, Mouton de Gruyter, volume 23 of Cognitive Approaches to Lexical Semantics, pages 447–494. find
  16. K. Erk, D. McCarthy, and N. Gaylord. 2013. Measuring word meaning in context. Computational Linguistics 39(3):511–554. find
  17. L. Frermann and M. Lapata. 2016. A Bayesian model of diachronic meaning change. Transactions of the Association for Computational Linguistics 4:31–45. find
  18. K. Gulordava and M. Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics. Stroudsburg, PA, USA, pages 67–71. find our card
  19. W. L. Hamilton, J. Leskovec, and D. Jurafsky. 2016. Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas, pages 2116–2121. find our card
  20. N. Ide and J. Pustejovsky, editors. 2017. Handbook of Linguistic Annotation. Springer, Dordrecht. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schlechtweg-2018-diachronic-usage-relatedness-durel-framework/