Elamur Research⌕
☰
home / papers / 2012 / Word Sense Induction for Novel Sense Detection

Word Sense Induction for Novel Sense Detection

Summary

Lau et al. tackle word sense induction (WSI), the task of discovering a word's senses from unlabelled usage rather than from a fixed inventory. They also show that WSI can serve a practical lexicographic purpose: detecting words that have acquired new senses. Each target word gets its own topic model. Every context (a sentence or short fragment) is treated as a "document", and the dominant topic of that context is taken as its sense. Features are lemmatised bag-of-words, plus optional positional words and dependency relations involving the target. A standard LDA model with a preset number of topics is compared against a Hierarchical Dirichlet Process (HDP), which decides how many senses each word needs. On SemEval-2010, HDP beats LDA even when LDA is given the gold number of senses, and it surpasses the best system from the original shared task. It also outperforms other topic-model approaches on SemEval-2007. In the novel sense experiments, BNC serves as the reference corpus and a ukWaC sample as the newer one. The HDP model separates words that gained a sense from distractor words and often isolates the tokens of the new sense.

Key points

  • SemEval-2010 WSI data: 100 target words (50 nouns, 50 verbs), about 163K web training instances and about 9K news test instances. Evaluation is WSD F-score after mapping induced senses to gold senses, using 80%/20% mapping/evaluation splits averaged over 5 random runs.
  • Overall F-scores:
    • HDP+Position: 0.68
    • LDA with oracle per-word T: 0.64
    • LDA with fixed T (7 for nouns, 3 for verbs): 0.63
    • UoY benchmark: 0.62
    • MFS baseline: 0.59
    • Adding dependency features to HDP brought no gain.
  • A novelty score computed from the induced senses ranks lemmas with a new sense above distractors (one-sided Wilcoxon rank sum test, p < .05). A baseline based on the ratio of word frequencies between the two corpora was not significant.
  • Token-level detection of the new sense (Nov method) reached F-scores of 0.95 for export and 0.92 for worm, but only 0.45 for domain and 0.00 for poster.
  • Oracle topic selection lifted poster to 0.62. Allowing the oracle to pick several topics raised domain to 0.92, because its new "Internet domain" sense was split across two topics.

Interesting ideas

  • I1. One model per target word, with context windows as documents. Training a separate topic model for each word over its short contexts turns a generic document-clustering tool into a sense clusterer. Each sense then comes with an interpretable weighted word list, which helps when inspecting results for any language.
  • I2. Let a non-parametric model choose the granularity. HDP beat LDA even when LDA was handed the true sense count. This suggests that fixing the number of clusters can hurt even when the "correct" number is known, which is worth considering for any clustering task where the categories are uncertain.
  • I3. Corpus comparison through induced senses rather than raw frequency. Inducing senses on pooled data from two corpora and checking which senses are concentrated in the newer one exposed semantic change that frequency ratios missed. The same approach could track meaning shifts across time periods, regions or domains.
  • I4. Oracle upper bounds to locate the bottleneck. Comparing the automatic sense selection against oracle single-topic and oracle multi-topic choices showed that most errors came from the selection heuristic, not from the induced clusters. This kind of diagnostic separates pipeline stages cleanly in many evaluation setups.
  • I5. Watch for genre mismatch in corpus comparison. The top false positive, guess, came from forum-style usage that was common in ukWaC but rare in BNC. The lesson for any diachronic or cross-corpus study is that the corpora need to be comparable in genre, or register differences will look like new meanings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Institutions

🎓University of California, IrvineIrvine · United States22 works · 35 120 citations🎓The University of MelbourneMelbourne · Australia47 works · 19 957 citations🔬Data61Sydney · Australia9 works · 2 187 citations🏢Lexical ComputingBrighton3 works · 472 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lau2012word,
  title = {{Word Sense Induction for Novel Sense Detection}},
  author = {Lau, Jey Han and Cook, Paul and McCarthy, Diana and Newman, David and Baldwin, Timothy},
  booktitle = {Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2012},
  url = {https://aclanthology.org/E12-1060/}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre and Aitor Soroa. 2007. SemEval-2007 Task 02: Evaluating word sense induction and discrimination systems. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 7–12, Prague, Czech Republic. find
  2. John Ayto. 2006. Movers and Shakers: A Chronology of Words that Shaped our Age. Oxford University Press, Oxford. find
  3. David Bamman and Gregory Crane. 2011. Measuring historical word sense variation. In Proceedings of the 2011 Joint International Conference on Digital Libraries (JCDL 2011), pages 1–10, Ottawa, Canada. find
  4. D. Blei, A. Ng, and M. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  5. S. Brody and M. Lapata. 2009. Bayesian word sense induction. pages 103–111, Athens, Greece. find our card
  6. Lou Burnard. 2000. The British National Corpus Users Reference Guide. Oxford University Computing Services. find
  7. Paul Cook and Suzanne Stevenson. 2010. Automatically identifying changes in the semantic orientation of words. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), pages 28–34, Valletta, Malta. find
  8. Marie-Catherine De Marneffe, Bill Maccartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. Genoa, Italy. find our card
  9. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. find
  10. Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008. Introducing and evaluating ukwac, a very large web-derived corpus of english. In Proceedings of the 4th Web as Corpus Workshop: Can we beat Google, pages 47–54, Marrakech, Morocco. find
  11. William A. Gale, Kenneth W. Church, and David Yarowsky. 1992. One sense per discourse. pages 233–237. find
  12. Kristina Gulordava and Marco Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 67–71, Edinburgh, Scotland. find our card
  13. Adam Kilgarriff and David Tugwell. 2002. Sketching words. In Marie-Hélène Corréard, editor, Lexicography and Natural Language Processing: A Festschrift in Honour of B. T. S. Atkins, pages 125– 137. Euralex, Grenoble, France. find
  14. Dan Klein and Christopher D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15 (NIPS 2002), pages 3– 10, Whistler, Canada. find
  15. Ioannis Korkontzelos and Suresh Manandhar. 2010. Uoy: Graphs of unambiguous vertices for word sense induction and disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 355–358, Uppsala, Sweden. find
  16. Suresh Manandhar, Ioannis Klapaftis, Dmitriy Dligach, and Sameer Pradhan. 2010. SemEval-2010 Task 14: Word sense induction & disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 63–68, Uppsala, Sweden. find our card
  17. Roberto Navigli and Giuseppe Crisafulli. 2010. Inducing word senses to improve web search result clustering. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 116–126, Cambridge, USA. find
  18. Zheng-Yu Niu, Dong-Hong Ji, and Chew-Lim Tan. 2007. I2R: Three systems for word sense discrimination, chinese word sense disambiguation, and english word sense disambiguation. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 177–182, Prague, Czech Rep… find
  19. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Comput. Linguist., 33:161–199. find our card
  20. Yves Peirsman, Dirk Geeraerts, and Dirk Speelman. 2010. The automatic identification of lexical variation between language varieties. Natural Language Engineering, 16(4):469–491. find
  21. Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2009. Semantic density analysis: Comparing word meaning across time and space. In Proceedings of the EACL 2009 Workshop on GEMS: GEometrical Models of Natural Language Semantics, pages 104– 111, Athens, Greece. find
  22. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of the International Conference on New Methods in Language Processing, pages 44–49, Manchester, UK. find
  23. Hinrich Schutze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97– 123. find our card
  24. Catherine Soanes and Angus Stevenson, editors. 2008. The Concise Oxford English Dictionary. Oxford University Press, eleventh (revised) edition. Oxford Reference Online. find
  25. Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. 2006. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101:1566– 1581. find
  26. Della Thompson, editor. 1995. The Concise Oxford Dictionary of Current English. Oxford University Press, Oxford, ninth edition. find
  27. Xuchen Yao and Benjamin Van Durme. 2011. Non-parametric bayesian word sense induction. In Proceedings of TextGraphs-6: Graph-based Methods for Natural Language Processing, pages 10–14, Portland, Oregon. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lau-2012-word-sense-induction-novel-sense/