Elamur Research⌕
☰
home / papers / 2009 / Bayesian Word Sense Induction

Bayesian Word Sense Induction

Summary

The paper addresses word sense induction: discovering a word's senses directly from raw text rather than from a fixed inventory such as a dictionary. Fixed inventories are hard to port to new domains or low-resource languages, and their granularity may not suit a given application. Brody and Lapata recast the task as Bayesian generative modelling inspired by LDA. Instead of one global topic model over whole documents, they train a separate small model for each ambiguous word. A short context window around the target is treated as a mixture over a few latent senses, and each sense is a distribution over context features. The model's key extension is that several feature "layers" share one sense mixture per instance while keeping separate sense-feature distributions. The layers are word windows of different sizes, word n-grams, part-of-speech features and dependencies. On the Semeval-2007 sense induction benchmark, with models trained on WSJ or BNC text, the best configuration outperforms the leading competition systems. The framework also provides a clean way to measure which kinds of linguistic information actually help.

Key points

  • Model design. Each target word gets its own model with typically fewer than ten senses. This contrasts with the tens or hundreds of topics common in document-level LDA.
  • Priors and evaluation mapping.
    • Sense mixtures have a Dirichlet prior α.
    • Sense-feature distributions have a symmetric Dirichlet prior β, with one per layer in the extended model.
    • Induced clusters are mapped to gold senses only for evaluation.
  • Best result on WSJ. The two-layer model combining a 10-word window and a 5-word window (10w+5w) reaches an F-score of 87.3. This compares with:
    • 86.8 for I2R, an Information Bottleneck system (called IR2 in the text);
    • 84.5 for UMND2, which uses k-means on second-order co-occurrence vectors;
    • 80.9 for the most-frequent-sense baseline.
  • Statistical significance. The best model is significantly better than UMND2 and MFS (p < 0.01, χ² test). Its edge over I2R is not significant.
  • BNC results.
    • Single layers range from 81.1 (dependencies) to 84.6 (10w).
    • All layers together score 84.1, and dropping dependencies raises this to 84.7.
    • 10w+5w is again best, at 85.5.
    • The sparse local layers (n-grams, dependencies) suffer most from the domain shift.

Interesting ideas

  • I1. Separate distributions per feature type. Different feature types share a single latent mixture but each keeps its own emission distribution. Simply merging all features into one bag lets dense features like POS tags swamp sparse ones. This pattern could fit any multi-view clustering problem, such as text plus metadata, or captions plus images, as the authors themselves suggest.
  • I2. Per-word local topic models. Running LDA-style inference on small windows around one target, rather than on whole documents, yields fine-grained, task-specific clusters. The same approach could be applied to induce senses of Ukrainian polysemous words or domain-specific trade terms without any lexicon.
  • I3. Ablation by adding and removing layers. The authors compare each layer alone, all layers minus one, and selected pairs. This cheaply shows which information sources help and which hurt; for example, the very sparse dependency layer gives near-uniform sense estimates that confuse the combined model.
  • I4. Combining complementary but equally strong signals. Pairing a wide topical window with a narrower local one worked better than piling on every layer. This is a useful heuristic for feature combination in general.
  • I5. Open-ended future extensions. The authors propose learning hyperparameters and layer weights inside Gibbs sampling. They also suggest a nonparametric (infinite Dirichlet) prior so the number of senses is inferred rather than fixed in advance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Word sense disambiguation

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom202 works · 55,753 citations🎓Columbia UniversityNew York · United States118 works · 44,744 citations

Cite

Cite as Samuel Brody and Mirella Lapata. 2009. Bayesian Word Sense Induction. Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{brody2009bayesian,
  title = {{Bayesian Word Sense Induction}},
  author = {Samuel Brody and Mirella Lapata},
  booktitle = {Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)},
  year = {2009},
  doi = {10.3115/1609067.1609078},
  url = {https://doi.org/10.3115/1609067.1609078}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, Eneko, Lluı́s Màrquez, and Richard Wicentowski, editors. 2007. Proceedings of the SemEval-2007. Prague, Czech Republic. find
  2. Agirre, Eneko and Aitor Soroa. 2007. Semeval-2007 task 02: Evaluating word sense induction and discrimination systems. In Proceedings of SemEval-2007. Prague, Czech Republic, pages 7–12. find
  3. Barnard, K., P. Duygulu, D. Forsyth, N. De Freitas, D. M. Blei, and M. I. Jordan. 2003. Matching words and pictures. J. of Machine Learning Research 3(6):1107–1135. find
  4. Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research 3:993–1022. our card
  5. Bordag, Stefan. 2006. Word sense induction: Triplet-based clustering and automatic evaluation. In Proceedings of the 11th EACL. Trento, Italy, pages 137–144. find
  6. Boyd-Graber, Jordan and David Blei. 2007. Putop: Turning predominant senses into a topic model for word sense disambiguation. In Proceedings of SemEval-2007. Prague, Czech Republic, pages 277–281. find
  7. Boyd-Graber, Jordan, David Blei, and Xiaojin Zhu. 2007. A topic model for word sense disambiguation. In Proceedings of the EMNLP-CoNLL. Prague, Czech Republic, pages 1024–1033. find
  8. Briscoe, Ted and John Carroll. 2002. Robust accurate statistical annotation of general text. In Proceedings of the 3rd LREC. Las Palmas, Gran Canaria, pages 1499–1504. our card
  9. Cai, J. F., W. S. Lee, and Y. W. Teh. 2007. Improving word sense disambiguation using topic features. In Proceedings of the EMNLP-CoNLL. Prague, Czech Republic, pages 1015–1023. find
  10. Carpuat, Marine and Dekai Wu. 2005. Word sense disambiguation vs. statistical machine translation. In Proceedings of the 43rd ACL. Ann Arbor, MI, pages 387–394. find
  11. Dorow, Beate and Dominic Widdows. 2003. Discovering corpus-specific word senses. In Proceedings of the 10th EACL. Budapest, Hungary, pages 79–82. find
  12. Firth, J. R. 1957. A Synopsis of Linguistic Theory 1930-1955. Oxford: Philological Society. find
  13. Gauch, Susan and Robert P. Futrelle. 1993. Experiments in automatic word class and word sense identification for information retrieval. In Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval. Las Vegas, NV, pages 425–434. find
  14. Geman, S. and D. Geman. 1984. Stochastic relaxation, Gibbs distribution, and Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6(6):721–741. find
  15. Goldwater, Sharon and Tom Griffiths. 2007. A fully Bayesian approach to unsupervised part-of-speech tagging. In Proceedings of the 45th ACL. Prague, Czech Republic, pages 744–751. our card
  16. Griffiths, Thomas L., Mark Steyvers, David M. Blei, and Joshua B. Tenenbaum. 2005. Integrating topics and syntax. In Lawrence K. Saul, Yair Weiss, and Léon Bottou, editors, Advances in Neural Information Processing Systems 17, MIT Press, Cambridge, MA, pages 537–544. find
  17. Griffiths, Tom L. and Mark Steyvers. 2002. A probabilistic approach to semantic representation. In Proeedings of the 24th Annual Conference of the Cognitive Science Society. Fairfax, VA, pages 381–386. find
  18. Hovy, Eduard, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of the HLT, Companion Volume: Short Papers. Association for Computational Linguistics, New York City, USA, pages 57–60. find
  19. Jansen, B. J., A. Spink, and A. Pfaff. 2000. Linguistic aspects of web queries. find
  20. Lee, Yoong Keok and Hwee Tou Ng. 2002. An empirical evaluation of knowledge sources and learning algorithms for word sense disambiguation. In Proceedings of the EMNLP. Morristown, NJ, USA, pages 41–48. our card
  21. McCarthy, Diana, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant senses in untagged text. In Proceedings of the 42nd ACL. Barcelona, Spain, pages 280–287. our card
  22. Mihalcea, Rada and Phil Edmonds, editors. 2004. Proceedings of the SENSEVAL-3. Barcelona. find
  23. Niu, Zheng-Yu, Dong-Hong Ji, and Chew-Lim Tan. 2007. I2r: Three systems for word sense discrimination, chinese word sense disambiguation, and english word sense disambiguation. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Association for Computational L… find
  24. Pantel, Patrick and Dekang Lin. 2002. Discovering word senses from text. In Proceedings of the 8th KDD. New York, NY, pages 613–619. find
  25. Pedersen, Ted. 2007. Umnd2 : Senseclusters applied to the sense induction task of senseval-4. In Proceedings of SemEval-2007. Prague, Czech Republic, pages 394–397. find
  26. Preiss, Judita and David Yarowsky, editors. 2001. Proceedings of the 2nd International Workshop on Evaluating Word Sense Disambiguation Systems. Toulouse, France. find
  27. Purandare, Amruta and Ted Pedersen. 2004. Word sense discrimination by clustering contexts in vector and similarity spaces. In Proceedings of the CoNLL. Boston, MA, pages 41–48. find
  28. Schütze, Hinrich. 1998. Automatic word sense discrimination. Computational Linguistics 24(1):97–123. our card
  29. Teh, Y. W., M. I. Jordan, M. J. Beal, and D. M. Blei. 2006. Hierarchical Dirichlet processes. Journal of the American Statistical Association 101(476):1566–1581. find
  30. Véronis, Jean. 2004. Hyperlex: lexical cartography for information retrieval. Computer Speech & Language 18(3):223–252. find
  31. Vickrey, David, Luke Biewald, Marc Teyssier, and Daphne Koller. 2005. Word-sense disambiguation for machine translation. In Proceedings of the HLT/EMNLP. Vancouver, pages 771–778. our card
  32. Voorhees, Ellen M. 1993. Using wordnet to disambiguate word senses for text retrieval. In Proceedings of the 16th SIGIR. New York, NY, pages 171–180. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/brody-2009-bayesian-word-sense-induction/