📄 SemEval-2010 Task 14: Word Sense Induction &Disambiguation

Summary

This SemEval-2010 shared task offers a common benchmark for unsupervised word sense induction (WSI), where senses are learned from raw text instead of taken from a hand-built lexicon. The organisers chose 100 target words (50 nouns, 50 verbs). Systems first built sense clusters from a large training corpus collected from the web. They then labelled new instances drawn from OntoNotes with those clusters. Unlike the 2007 edition, training and test data were kept strictly apart, so the test checks how well a learned clustering model generalises.

Scoring used two methods:

• Unsupervised: V-Measure and paired F-Score.
• Supervised: clusters were mapped to gold senses on part of the test data, then scored as WSD on the rest.

Of the 26 submitted systems, the metrics disagreed sharply. V-Measure favoured systems producing many clusters. Paired F-Score was topped by the most-frequent-sense (MFS) baseline. In the supervised setting, UoY ranked first. Overall, the task shows how strongly WSI conclusions depend on the evaluation design.

Key points

• Data sizes:
  • Training: 879,807 instances (716,945 nouns, 162,862 verbs).
  • Test: 8,915 instances (5,285 nouns, 3,630 verbs).
  • Gold senses per word: 3.79 on average (4.46 for nouns, 3.12 for verbs).
• Training corpus construction:
  • For each WordNet sense, a query combined the target word with related lemmas (hypernyms, hyponyms, synonyms, meronyms, holonyms).
  • Queries went to the Yahoo! search API, with up to 1,000 pages per query.
  • Paragraph text containing the target was kept, then filtered by POS with the Genia tagger.
  • Only morphology and syntax tools were allowed as extra resources.
• Test data: OntoNotes sense-tagged sentences from news sources such as CNN and ABC.
• V-Measure results: best scores were Hermit 16.2%, UoY 15.7% and KSU KDD 15.7%. These systems produced roughly 11–17.5 clusters per word. No system beat the MFS baseline on paired F-Score (63.5%).
• Supervised results (80/20 mapping/evaluation split, averaged over 5 random splits):
  • UoY led with 62.4%, and 14 systems beat MFS.
  • UoY was strongest on nouns, while Duluth-Mix-Narrow-Gap was strongest on verbs.
  • With a 60/40 split, UoY stayed about 3.7 points above MFS, while KSU KDD fell well below it.

Interesting ideas

• I1. Keep induction and test data separate. The test data is used only for tagging, never for inducing senses. This turns WSI into a check of whether a clustering model generalises, not just whether it fits one sample. The same principle applies to any unsupervised clustering benchmark.
• I2. Pair metrics with opposite biases. V-Measure rewards many small clusters, while paired F-Score penalises having either too many or too few clusters. Reporting both, along with the cluster count, exposes cases where a system games one metric. This is useful whenever a clustering is scored against gold classes.
• I3. Average over several random splits. The organisers averaged the supervised scores over five random mapping/evaluation splits because single splits in 2007 had produced unstable rankings. This is a cheap way to make leaderboards more trustworthy on small test sets.
• I4. Shrink the mapping data to test robustness. Reducing the mapping portion (80% down to 60%) reveals whether a system's score relies on having plenty of labelled data for mapping. Systems with many clusters suffer most, which makes this a useful diagnostic for any evaluation that maps clusters to labels.
• I5. Collect sense-specific training text from the web. Searching for a word together with lexicon neighbours of one particular sense, then filtering results by POS, yields sense-oriented text semi-automatically. This could be adapted to bootstrap corpora for languages with a WordNet-like resource but little annotated text.