Elamur Research
☰
home / papers / 2006 / Evaluating WordNet-based Measures of Lexical Semantic Relatedness

Evaluating WordNet-based Measures of Lexical Semantic Relatedness

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J06-1003.pdf

Summary

Many NLP tasks need a numeric estimate of how closely two word meanings are connected. These include word sense disambiguation, summarization, information retrieval and automatic error correction. Many such measures exist, but it is hard to tell which one is best. The authors stress that relatedness is wider than similarity. It also covers meronymy, antonymy and simple functional or frequent associations.

They first survey dictionary-based (LDOCE), Roget-thesaurus-based and WordNet-based approaches. They then test five WordNet-based measures, by Hirst–St-Onge, Leacock–Chodorow, Resnik, Jiang–Conrath and Lin, in a practical task: detecting and correcting real-word spelling errors (malapropisms). A word that fits poorly with its context is flagged, and a similarly spelled word that fits better is proposed.

The information-content measure of Jiang and Conrath comes out on top. The paper also argues that distributional similarity is not a good substitute for semantic relatedness. Its value lies in giving an application-driven way to choose between competing measures and in pointing to non-classical relations as future work.

Key points

  • Five WordNet-based measures were compared on malapropism detection and correction. Jiang–Conrath outperformed Hirst–St-Onge, Leacock–Chodorow, Lin and Resnik.
  • All experiments used WordNet 1.5 to keep results strictly comparable. The noun hierarchy has 11 unique beginners and a maximum depth of 16 nodes, and hyponymy/hypernymy makes up close to 80% of its relations.
  • A distributional-similarity test (Weeds, British National Corpus) was limited to about 2,000 words. This severely restricted which errors could be handled and gave generally poor results.
  • In Morris's experiments with human readers, about 60% of perceived lexical relations in text were "non-classical." No current resource catalogues such relations.
  • Some relatedness is formed ad hoc in context, for example sex, drinking and drag racing grouped as dangerous behaviors. This kind of relatedness cannot be fully derived from a fixed resource like WordNet.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 158 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.2006.32.1.13

@article{budanitsky2006evaluating,
  title = {{Evaluating WordNet-based Measures of Lexical Semantic Relatedness}},
  author = {Budanitsky, Alexander and Hirst, Graeme},
  journal = {Computational Linguistics},
  year = {2006},
  doi = {10.1162/coli.2006.32.1.13},
  url = {https://doi.org/10.1162/coli.2006.32.1.13}
}

References

57 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agresti, Alan and Barbara Finlay. 1997. Statistical Methods for the Social Sciences. Prentice Hall, Upper Saddle River, NJ, 3rd edition. find
  2. Banerjee, Satanjeev and Ted Pedersen. 2003. Extended gloss overlaps as a measure of semantic relatedness. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, Acapulco, Mexico, pages 805–810, August. find
  3. Barsalou, Lawrence W. 1983. Ad hoc categories. Memory and Cognition, 11:211–227. find
  4. Barsalou, Lawrence W. 1989. Intra-concept similarity and its implications for inter-25 Indeed Murphy (2003) has suggested that semantic relations (of all types) are best conceived of as metalexical: derived from a (pre-existing) lexicon, but not part of it. concept similarity. In Stella Vosniadou an… find
  5. Bernard, J. R. L., editor. 1986. The Macquarie Thesaurus. Macquarie Library, Sydney, Australia. find
  6. Budanitsky, Alexander. 1999. Lexical semantic relatedness and its application in natural language processing. Technical Report CSRG-390, Computer Systems Research Group, University of Toronto, August. find
  7. Budanitsky, Alexander and Graeme Hirst. 2001. Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures. In Workshop on WordNet and Other Lexical Resources, Second Meeting of the North American Chapter of the Association for Computational Linguistics, Pittsburgh… find
  8. Corley, Courtney and Rada Mihalcea. 2005. Measuring the semantic similarity of texts. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, MI, June, pages 13–18. find
  9. Dagan, Ido. 2000. Contextual word similarity. In Robert Dale, Hermann Moisl, and Harold Somers, editors, Handbook of Natural Language Processing. Marcel Dekker, pages 459–475. find
  10. Dagan, Ido, Lillian Lee, and Fernando C. N. Pereira. 1999. Similarity-based models of word cooccurrence probabilities. Machine Learning, 34(1–3):43–69. find
  11. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA. find
  12. Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20:116–131. find
  13. Francis, Winthrop Nelson and Henry Kučera. 1982. Frequency Analysis of English Usage: Lexicon and Grammar. Houghton Mifflin, Boston. find
  14. Fromkin, Victoria A. 1980. Errors in Linguistic performance: Slips of the Tongue, Ear, Pen, and Hand. Academic Press, San Francisco. find
  15. Grefenstette, Gregory. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer Academic Publishers, Norwell, MA. find
  16. Halliday, M. A. K. and Ruqaiya Hasan. 1976. Cohesion in English. English Language Series. Longman, New York. find
  17. Hirst, Graeme and Alexander Budanitsky. 2005. Correcting real-word spelling errors by restoring lexical cohesion. Natural Language Engineering, 11:87–111. find
  18. Hirst, Graeme and David St-Onge. 1998. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332. find
  19. Hoey, Michael. 1991. Patterns of Lexis in Text. Describing English Language. Oxford University Press. find
  20. Jarmasz, Mario and Stan Szpakowicz. 2003. Roget’s Thesaurus and semantic similarity. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-2003), Borovetz, Bulgaria, pages 212–219, September. find
  21. Jiang, Jay J. and David W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33. find
  22. Kohomban, Upali Sathyajith and Wee Sun Lee. 2005. Learning semantic classes for word sense disambiguation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 34–41, Ann Arbor, MI, June. find
  23. Kozima, Hideki and Teiji Furugori. 1993. Similarity between words computed by spreading activation on an English dictionary. In Proceedings of 6th Conference of the European Chapter of the Association for Computational Linguistics (EACL-93), pages 232–239, Utrecht. find
  24. Kozima, Hideki and Akira Ito. 1997. Context-sensitive word distance by adaptive scaling of a semantic space. In Ruslan Mitkov and Nicolas Nicolov, editors, Recent Advances in Natural Language Processing: Selected Papers from RANLP’95, volume 136 of Amsterdam Studies in the Theory and History of Ling… find
  25. Lakoff, George. 1987. Women, Fire, and Dangerous Things: What Categories Reveal About the Mind. University of Chicago Press, Chicago. find
  26. Leacock, Claudia and Martin Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283. find
  27. Lee, Joon Ho, Myong Ho Kim, and Yoon Joon Lee. 1993. Information retrieval based on conceptual distance in IS-A hierarchies. Journal of Documentation, 49(2):188–207. find
  28. Lee, Lillian. 1999. Measures of distributional similarity. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), College Park, MD, pages 25–31. find
  29. Lesk, Michael. 1986. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings, Fifth International Conference on Systems Documentation (SIGDOC ’86), Toronto, Canada, pages 24–26. find
  30. Lin, Dekang. 1998a. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING–ACL ’98), Montreal, Canada, pages 768–774, August. find
  31. Lin, Dekang. 1998b. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison, WI, pages 296–304, July. find
  32. McCarthy, Diana, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in untagged text. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, pages 280–287. find our card
  33. Miller, George A. and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
  34. Mohammad, Saif and Graeme Hirst. 2005. Distributional measures as proxies for semantic relatedness. Submitted for publication. find
  35. Morris, Jane. 2006. Readers’ Perceptions of Lexical Cohesion and Lexical Semantic Relations in Text. Ph.D. thesis, University of Toronto. find
  36. Morris, Jane and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics, 17(1):21–48. find our card
  37. Morris, Jane and Graeme Hirst. 2004. Non-classical lexical semantic relations. In Workshop on Computational Lexical Semantics, Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Boston, MA, pages 46–51, May. find
  38. Morris, Jane and Graeme Hirst. 2005. The subjectivity of lexical cohesion in text. In James G. Shanahan, Yan Qu, and Janyce Wiebe, editors, Computing Attitude and Affect in Text. Springer, New York, pages 41–48. find
  39. Murphy, M. Lynne. 2003. Semantic Relations and the Lexicon. Cambridge University Press. find
  40. Patwardhan, Siddharth, Satanjeev Banerjee, and Ted Pedersen. 2003. Using measures of semantic relatedness for word sense disambiguation. In Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, Mexico City, Mexico, pages 241–257, February. find
  41. Pedersen, Ted, Siddharth Patwardhan, and Jason Michelizzi. 2004. Wordnet:: similarity — measuring the relatedness of concepts. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI-04). AAAI Press, Cambridge, MA, pages 1024–1025. find
  42. Procter, Paul, editor. 1978. Longman Dictionary of Contemporary English. Longman. find
  43. Rada, Roy and Ellen Bicknell. 1989. Ranking documents with a thesaurus. JASIS, 40(5):304–310. find
  44. Rada, Roy, Hafedh Mili, Ellen Bicknell, and Maria Blettner. 1989. Development and application of a metric on semantic nets. IEEE Transactions on Systems, Man, and Cybernetics, 19(1):17–30. find
  45. Resnik, Philip. 1995. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, August. find
  46. Resnik, Philip and Mona Diab. 2000. Measuring verb similarity. In Proceedings of the 22nd Annual Meeting of the Cognitive Science Society, Philadelphia, PA, pages 399–404. find
  47. Rubenstein, Herbert and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  48. Spellman, Barbara A., Keith J. Holyoak, and Robert G. Morrison. 2001. Analogical priming via semantic relations. Memory and Cognition, 29(3):383–393. find
  49. St-Onge, David. 1995. Detecting and correcting malapropisms with lexical chains. Master’s thesis, University of Toronto. Published as Technical Report CSRI-319. find
  50. Stevenson, Mark and Mark Greenwood. 2005. A semantic approach to IE pattern induction. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 379–386, Ann Arbor, MI, June. find
  51. Sussna, Michael. 1993. Word sense disambiguation for free-text indexing using a massive semantic network. In Proceedings of the Second International Conference on Information and Knowledge Management (CIKM-93), pages 67–74, Arlington, VA. find
  52. Sussna, Michael John. 1997. Text Retrieval Using Inference in Semantic Metanetworks. Ph.D. thesis, University of California, San Diego. find
  53. Weeds, Julie and David Weir. 2005. Co-occurrence retrieval: A flexible framework for lexical distributional similarity. Computational Linguistics, 31(4):433–475. find
  54. Weeds, Julie E. 2003. Measures and applications of lexical distributional similarity. Ph.D. thesis, University of Sussex, September. find
  55. Wei, Mei. 1993. An analysis of word relatedness correlation measures. Master’s thesis, University of Western Ontario, London, Ontario. find
  56. Whitten, William B., W. Newton Suter, and Michael L. Frank. 1979. Bidirectional synonym ratings of 464 noun pairs. Journal of Learning and Verbal Behavior, 18:109–127. find
  57. Wu, Zhibiao and Martha Palmer. 1994. Verb semantics and lexical selection. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, pages 133–138, Las Cruces, NM, June. find