Elamur Research⌕
☰
home / papers / 2002 / An Empirical Evaluation of Knowledge Sources and Learning Algorithms for Word Sense Disambiguation

An Empirical Evaluation of Knowledge Sources and Learning Algorithms for Word Sense Disambiguation

Summary

The paper investigates how supervised word sense disambiguation (WSD) accuracy depends on two factors: the features used to describe a word's context and the classifier trained on them. Earlier studies usually varied only one of these, or tested on just a few words. The authors evaluate both together on the English lexical sample tasks of SENSEVAL-2 and SENSEVAL-1. They define four knowledge sources: POS tags of neighbouring tokens, bag-of-words context unigrams, local collocations, and syntactic relations taken from Charniak's parser output converted to dependencies. These are combined with four WEKA learners: SVM, AdaBoost with decision stumps, Naive Bayes, and decision trees. With all four knowledge sources, no feature selection, and a single linear SVM, the system outperforms the best official participants in both competitions, and the difference is statistically significant. The study shows that a well-engineered feature set paired with one strong learner can beat ensemble systems. It also shows that the value of a feature type depends on which algorithm uses it.

Key points

  • Best micro-averaged recall: 65.4% on SENSEVAL-2 and 79.2% on SENSEVAL-1, using SVM with all knowledge sources and no feature selection. Both scores exceed the top official systems (JHU on SENSEVAL-2, hopkins on SENSEVAL-1) at the 0.05 significance level under a paired t test.
  • Removing knowledge sources one at a time on SENSEVAL-2 lowers SVM recall step by step:
    • 65.4% with all four sources;
    • 64.8% without syntactic relations;
    • 61.8% after also removing POS;
    • 60.5% with collocations only.
  • The features are:
    • 7 POS features in a window around the target word, kept within the sentence;
    • binary unigram features over the lemmatized, stop-word-filtered context;
    • 11 local collocation features, each taking string values;
    • POS-specific syntactic relation features: 4 for nouns, 6 for verbs, 2 for adjectives.
  • Learner and feature interactions:
    • Collocations help SVM the most, while POS helps Naive Bayes the most.
    • Naive Bayes beats SVM when POS is the only feature type.
    • SVM works best without feature selection, whereas Naive Bayes gains from some frequency-based selection.
    • Higher-order polynomial SVM kernels performed worse than the linear kernel.
  • The SENSEVAL-2 system UMD-SST also used SVM and reported 56.8% recall. The authors' re-implementation of it reached 58.6%, while their own SVM with the same two feature types reached 61.8%. They suggest the gap may come from the different choice of collocations.

Interesting ideas

  • I1. Evaluate features and learners as a grid. Crossing every knowledge source with every classifier exposes interactions that single-axis studies miss. It is a cheap protocol for any feature-engineering task, such as tagging or classification for Ukrainian.
  • I2. Encode collocations as multi-valued features. Each collocation position (offset span) becomes one nominal feature whose values are the observed strings, instead of creating one binary feature per n-gram. This keeps the feature space compact while preserving word order around the target.
  • I3. Report both micro and macro averages, and test significance per unit. Micro-averaging lets frequent words dominate the score, while macro-averaging weights each word task equally. Running paired t tests per instance or per task makes claims of beating a baseline defensible.
  • I4. Re-implement competitors before attributing gains. Rebuilding UMD-SST showed that the improvement came from feature design rather than from SVM itself. This is a useful sanity check whenever "same algorithm, different result" appears.
  • I5. Tune feature selection separately for each learner. Frequency-threshold selection helped Naive Bayes but hurt SVM. Preprocessing choices should therefore be tuned per model, not fixed globally.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Word sense disambiguation Evaluation & benchmarks

Institutions

🎓National University of SingaporeSingapore · Singapore61 works · 28,316 citations

Cite

Cite as Yoong Keok Lee and Hwee Tou Ng. 2002. An Empirical Evaluation of Knowledge Sources and Learning Algorithms for Word Sense Disambiguation. Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1118693.1118699

@inproceedings{lee2002empirical,
  title = {{An Empirical Evaluation of Knowledge Sources and Learning Algorithms for Word Sense Disambiguation}},
  author = {Yoong Keok Lee and Hwee Tou Ng},
  booktitle = {Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)},
  year = {2002},
  doi = {10.3115/1118693.1118699},
  url = {https://doi.org/10.3115/1118693.1118699}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Clara Cabezas, Philip Resnik, and Jessica Stevens. 2001. Supervised sense tagging using support vector machines. In Proceedings of the Second International Workshop on Evaluating Word Sense Disambiguation Systems (SENSEVAL-2), pages 59–62. find
  2. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics, pages 132–139. our card
  3. Martin Chodorow, Claudia Leacock, and George A. Miller. 2000. A topical/local classifier for word sense identification. Computers and the Humanities, 34(1– 2):115–120. find
  4. Richard O. Duda and Peter E. Hart. 1973. Pattern Classification and Scene Analysis. Wiley, New York. find
  5. Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of the Second International Workshop on Evaluating Word Sense Disambiguation Systems (SENSEVAL-2), pages 1–5. find
  6. Gerard Escudero, Llu ís Màrquez, and German Rigau. 2000. An empirical study of the domain dependence of supervised word sense disambiguation systems. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 172–180. find
  7. Yoav Freund and Robert E. Schapire. 1996. Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 148–156. find
  8. Adam Kilgarriff and Martha Palmer. 2000. Introduction to the special issue on SENSEVAL. Computers and the Humanities, 34(1–2):1–13. find
  9. Rada F. Mihalcea and Dan I. Moldovan. 2001. Pattern learning and active feature selection for word sense disambiguation. In Proceedings of the Second International Workshop on Evaluating Word Sense Disambiguation Systems (SENSEVAL-2), pages 127–130. find
  10. Raymond J. Mooney. 1996. Comparative experiments on disambiguating word senses: An illustration of the role of bias in machine learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 82–91. find
  11. Hwee Tou Ng and Hian Beng Lee. 1996. Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 40–47. our card
  12. Hwee Tou Ng. 1997. Exemplar-based word sense disambiguation: Some recent improvements. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, pages 208–213. find
  13. Ted Pedersen and Rebecca Bruce. 1997. A new supervised learning algorithm for word sense disambiguation. In Proceedings of the 14th National Conference on Artificial Intelligence, pages 604–609. find
  14. Ted Pedersen. 2001a. A decision tree of bigrams is an accurate predictor of word sense. In Proceedings of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics, pages 79– 86. find
  15. Ted Pedersen. 2001b. Machine learning with lexical features: The Duluth approach to Senseval-2. In Proceedings of the Second International Workshop on Evaluating Word Sense Disambiguation Systems (SENSEVAL-2), pages 139–142. find
  16. J. Ross Quinlan. 1993. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco. our card
  17. Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 133–142. our card
  18. Jeffrey C. Reynar and Adwait Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing, pages 16–19. our card
  19. Hee-Cheol Seo, Sang-Zoo Lee, Hae-Chang Rim, and Ho Lee. 2001. KUNLP system using classification information model at SENSEVAL-2. In Proceedings of the Second International Workshop on Evaluating Word Sense Disambiguation Systems (SENSEVAL-2), pages 147–150. find
  20. Mark Stevenson and Yorick Wilks. 2001. The interaction of knowledge sources in word sense disambiguation. Computational Linguistics, 27(3):321–349. find
  21. Vladimir N. Vapnik. 1995. The Nature of Statistical Learning Theory. Springer-Verlag, New York. find
  22. Jorn Veenstra, Antal van den Bosch, Sabine Buchholz, Walter Daelemans, and Jakub Zavrel. 2000. Memory-based word sense disambiguation. Computers and the Humanities, 34(1–2):171–177. find
  23. Ian H. Witten and Eibe Frank. 2000. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco. find
  24. David Yarowsky, Silviu Cucerzan, Radu Florian, Charles Schafer, and Richard Wicentowski. 2001. The Johns Hopkins SENSEVAL2 system descriptions. In Proceedings of the Second International Workshop on Evaluating Word Sense Disambiguation Systems (SENSEVAL-2), pages 163–166. find
  25. David Yarowsky. 2000. Hierarchical decision lists for word sense disambiguation. Computers and the Humanities, 34(1–2):179–186. find
  26. Jakub Zavrel, Sven Degroeve, Anne Kool, Walter Daelemans, and Kristiina Jokinen. 2000. Diverse classifiers for NLP disambiguation tasks: Comparison, optimization, combination, and evolution. In TWLT 18. Learning to Behave, pages 201–221. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2002-empirical-evaluation-knowledge-sources-learning/