📄 SemEval-2007 Task 02: Evaluating Word Sense Induction and Discrimination Systems

Summary

The paper describes SemEval-2007 Task 02, a shared evaluation for word sense induction and discrimination (WSID) systems. These are unsupervised methods that cluster occurrences of a word into "uses" rather than picking from a fixed dictionary. Such systems had mostly been tested in-house and were hard to compare, both with each other and with supervised or knowledge-based WSD. The organizers reused the English lexical sample data from task 17: Wall Street Journal text for 100 target words, tagged with coarse-grained OntoNotes senses. Participants clustered the unlabeled corpus, and their output was scored in two ways:

• Unsupervised: clusters were compared with gold classes using FScore, entropy and purity.
• Supervised: clusters were mapped to senses using the training portion, then scored with precision and recall on the test portion.

Six systems took part, one of them not an induction system. No system beat the one-cluster-per-word baseline under FScore. In the supervised setting, the best induction system came close to semi-supervised results. Because system rankings changed with the evaluation method, the authors argue the two views should be read together.

Key points

• Data: 100 target words (65 verbs, 35 nouns) from the task 17 lexical sample. The corpus has 27,132 occurrences, split into 22,281 train and 4,851 test.
• Induction systems: I2R, UBC-AS, UMND2, upv si and UOY. UoFL was a knowledge-based WSD system and was reported only for completeness.
• Supervised evaluation, official split:
  • Best induction system: 81.6 recall.
  • MFS baseline: 78.7.
  • Best task 17 systems: 88.7 supervised, 85.1 semi-supervised.
  • Only three participants beat MFS.
• Random 82/18 re-split: every system beat MFS (78.4), with I2R best at 82.2. UOY was hit hardest by the original non-random split.
• Random-cluster baselines under mapping: random2 scored 78.6, close to MFS. Scores fell as the number of clusters grew, reaching 31.8 for random1000.

Interesting ideas

• I1. Mapping matrix: estimate P(sense | cluster) from a labeled slice of data, then multiply each instance's cluster-score vector by this matrix to get sense scores. This makes any unsupervised clusterer comparable with standard classifiers, which is useful wherever induced categories must be checked against an existing label set.
• I2. Two complementary metrics: report both a clustering metric and a mapped, supervised metric. Each has its own bias: FScore here favoured systems with few clusters, while the mapped score was more neutral about cluster count.
• I3. Trivial baselines under the same protocol: run random clusterings and a one-cluster-per-word solution through the evaluation. This exposes how much of a score comes from the mapping step or the label distribution rather than from what the system actually learned.
• I4. Check the train/test split: a non-random split changed which systems beat MFS. Re-running with a random split of the same proportion is a cheap robustness check for any benchmark.
• I5. Inventory granularity matters: coarse OntoNotes senses, plus many single-sense words, made trivial baselines strong. Gold-standard granularity should be considered when designing sense or category benchmarks.