Elamur Research⌕
☰
home / papers / 1997 / A Perspective on Word Sense Disambiguation Methods and Their Evaluation

A Perspective on Word Sense Disambiguation Methods and Their Evaluation

Summary

This position paper argues that word sense disambiguation (WSD) remains a largely unsolved lexical problem, unlike POS tagging or morphology. One major reason is that the field lacks shared evaluation practice: studies use different target words, corpora and metrics. The authors make four observations:

  • evaluation is unstandardized;
  • the value of WSD differs by application, with machine translation looking more promising than IR or speech recognition;
  • sense-annotated data are scarce;
  • methods have barely converged, since WSD draws on many knowledge sources and is highly word-specific.

They then propose remedies:

  • Score systems with a cross-entropy-style measure so that well-calibrated probabilities earn partial credit, rather than relying only on exact-match accuracy.
  • Weight errors by semantic distance between senses in hierarchical inventories.
  • Build a common evaluation setup that combines the traditional focus on selected words with broad coverage.
  • Limit sense inventories to distinctions that languages commonly express with different words. Aligned parallel corpora could then supply tagged data without manual annotation.

The paper matters because it shaped the evaluation agenda later adopted in shared WSD tasks.

Key points

  • Earlier shared test data were rare. One exception is the 2094-instance line dataset of Leacock et al. (1993), reused by Lehman (1994) and Mooney (1996). The WordNet semantic concordance (Miller et al., 1994) is the only all-words annotation, but it typically yields fewer than 10 examples per polysemous word.
  • In the interest example with four hypothetical systems, all of them choose the wrong sense. Exact match scores them identically. Under cross-entropy, System 1, which gives probability .42 to the correct sense, costs 1.25 bits. A system that gives the correct sense zero probability receives an infinite penalty.
  • For systems without reliable probabilities, the authors suggest a variant without the log term. It reduces to simple % correct when outputs are 0/1.
  • A hierarchical inventory for bank (financial vs. geographical, with subsenses) illustrates distance-sensitive penalties.
  • Translation distinctions can serve as sense labels, for example duty → French devoir/droit, German Tisch vs. Tischrunde, and Spanish dedo (finger/toe). The authors propose mapping WordNet, LDOCE and COBUILD senses to such distinctions, for instance via EuroWordNet (Spanish, Italian, Dutch, English).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Word sense disambiguation Evaluation & benchmarks

Cite

Cite as Philip Resnik and David Yarowsky. 1997. A Perspective on Word Sense Disambiguation Methods and Their Evaluation. Tagging Text with Lexical Semantics: Why, What, and How?.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{resnik1997perspective,
  title = {{A Perspective on Word Sense Disambiguation Methods and Their Evaluation}},
  author = {Philip Resnik and David Yarowsky},
  booktitle = {Tagging Text with Lexical Semantics: Why, What, and How?},
  year = {1997},
  url = {https://aclanthology.org/W97-0213/}
}

References

28 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Bahl and R. Mercer. 1976. Part-of-speech as-sigument by a statistical decision algorithm. In International Symposium on Information Theory, Ronneby, Sweden. find
  2. L. Bald, F. Jelinek, and R. Mercer. 1983. A maxi m u m likelihood approach to continuous speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-5(2):179-190. find
  3. L. Bloksma, P. Dfez-Orzas and P. Vossen. 1996. User Requirements and Functional Specification of the EuroWordNet Project. http:/lw~m, let. uva. nll-er, m. find
  4. E. Brill. 1993. A Corpus-Based Approach to Language Learning. Ph.D. thesis, Computer and Information Science, University of Pennsylvania. find
  5. P. Brown, V. Della Pietra, P. deSouza, J. Lai, and R. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467-480. find
  6. R. Bruce and J. Wiebe. 1994. Word-sense disambiguation using decomposable models. In Proceedings of the 3~nd Annual Meeting of the Association .for Computational Linguistics, Las Cruces. find
  7. K. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted texts. In Proceedings of the Second Conference on Applied Natural Language Processing, Austin, Texas. find
  8. B. Dorr and D. Jones. 1996a. Acquisition of semantic lexicons: Using word sense disarnbiguation to improve precision. In Proceedings of the SIGLEX Workshop on Breadth and Depth of Semantic Lexicons, Santa Cruz, CA. find
  9. B. Dorr and D. Jones. 1996b. Role of word sense disambiguation in lexical acquisition: Predicting semantics from syntactic cues. In Proceedings of the International Conference on Computational Linguistics, Copenhagen, Denmark. find
  10. W. Gale, K. Church, and D. Yarowsky. 1992. One sense per discourse. Proceedings of the 4th DARPA Speech and Natural Language Workshop. find
  11. W. Gale, K. Church, and D. Yarowsky. A method for disambiguating word senses in a large corpus. Computers and the Humanities, 26:415-439, 1992. find
  12. F. Jelinek. 1985. Markov source modeling of text generation. In J. Skwirzinski, editor, Impact of Processing Techniques on Communication. Dordrecht. find
  13. S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-35(3):400-401. find
  14. It. Krovetz and W. B. Croft. 1992. Lexical ambiguity and information retrieval. ACM Transactions on Information Systems, 10(2):115-141. find
  15. C. Leaeock, G. Towell and E. Voorhees. 1993. Corpus-based statistical sense resolution. In Proceedings, ARPA Human Language Technology Workshop, pp. 260-265, Plainsboro, NJ. find
  16. J. Lehman. 1994. Toward the essential nature of statistical knowledge in sense resolution. In Proceedings of the TvJelfth National Conference on Artificial Intelligence, pp. 734-471. find
  17. R. Mooney. 1996. Comparative experiments on disam~i~mating word senses: An illustration of the role of bias in machine learning. In Proceedings of the Con]erence on Empirical Methods in Natural Language Processing, Philadelphia. find
  18. S. McRoy. 1992. Using multiple knowledge sources for word sense discrimination. Computational Linguistics, 18(1):1-30. find
  19. G. Miller and W. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1-28. find
  20. G. Miller, M. Chodorow, S. Landes, C. Leacock, and R. Thomas. 1994. Using a semantic concordance for sense identification. In Proceedings of the ARPA Human Language Technology Workshop, San Francisco. Morgan Kaufmarm. our card
  21. H. Ng and H. Lee. 1996. Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach. In Proceedings o.f the 34th Annual Meeting o.f the Society for Computational Linguistics, pp. 40-47, Santa Cruz, CA. 1993. Selection and ln]ormation: A Class-Based Approach to Le~ca… our card
  22. P. Resnik. Ph.D. thesis, University of Pennsylvania. ( f t p : / / ftp. cis. upenn, edu/pub/ircs/tr/93-42, ps. Z). find
  23. P. Resnik. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of the I~th International Joint Conference on Artificial Intelligence (IJCAI-95). (cmp-lg/9511007). our card
  24. R. Richardson, A. Smeaton, and J. Murphy. 1994. Using WordNet as a knowledge base for measuring semantic similarity between words. Working Paper CA-1294, Dublin City University, School of Computer Applications, Dublin, Ireland. f t p : / / f t p , compapp, dcu. ie/pub/w-papers/ 1994/CA1294. ps. Z. find
  25. P. Tapanainen and A. Voutilainen. 1994. Tagging accurately - don't guess if you know. In Proceedings of ANLP '94. find
  26. Y. Wilks and M. Stevenson. 1996. The grammar of sense: Is word-sense tagging much more than part-of-speech tagging? cmp-lg/9607028. find
  27. D. Yarowsky. 1992. Word-sense disambiguation using statistical models of Roget's categories trained on large corpora. In Proceedings of COLING-9~, pp. 454-460, Nantes, Prance. our card
  28. D. Yarowsky. 1993. One sense per collocation. Proccedings of the ARIA Human Language Technology Workshop, Morgan Kaufmann, pp. 266-271. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/resnik-1997-perspective-word-sense-disambiguation-methods/