📄 SemEval-2007 Task 10: English Lexical Substitution Task

Summary

This paper introduces the English Lexical Substitution task at SemEval-2007. It is a way to test word sense disambiguation without fixing a sense inventory in advance. For each target word in a sentence, systems must propose words or phrases that could replace it while keeping the meaning. Systems may draw substitutes from any lexical resource, so the task tests both the resource and the context-based disambiguation.

The organisers sampled sentences from Sharoff's English Internet Corpus for 201 polysemous nouns, verbs, adjectives and adverbs. Five native British English speakers each annotated the whole set with up to three substitutes per item. There are three scored subtasks: a single "best" answer, an "out of ten" (oot) list, and multiword detection and identification (mw). Baselines came from WordNet and from several distributional similarity measures.

Eight teams submitted ten systems, and most beat the baselines. A post hoc human judgement showed that system substitutes were still much worse than human ones. The task gave the field an application-oriented, inventory-independent benchmark for lexical meaning.

Key points

• Data:
  • 201 target words with 10 sentences each, 2010 in total, split into 300 trial and 1710 test sentences.
  • After filtering, 298 trial and 1696 test items remained.
  • Some words had hand-picked sentences (MAN); the others had randomly sampled ones (RAND).
• Inter-annotator agreement:
  • Pairwise agreement was only 27.75%.
  • Agreement with the mode (majority answer) was 50.67%.
  • Dropping one annotator who gave many substitutes raised these to 31.13% and 64.7%.
• best subtask:
  • KU led on recall with 12.90.
  • UNT was best at hitting the mode (20.73).
  • The WordNet baseline scored 9.95 and beat all distributional baselines; lin was the strongest of those.
• oot subtask: IRST 2 reached recall of 68.90, and UNT achieved the highest mode score (66.26).
• Other results:
  • Only HIT entered the mw subtask, and it outperformed the WordNet baseline there.
  • In the post hoc evaluation, judges rated 71.85% of system substitutes as bad, against 21.63% for the original annotators' substitutes.

Interesting ideas

• I1. Open answer space. Not giving participants a candidate list means the evaluation also tests where substitutes come from. This lets hand-built and automatically induced lexicons compete on equal terms, which suits any benchmark comparing resources.
• I2. Weighted, hedging-proof scoring. Credit for a guess is weighted by how many annotators gave it and divided by the number of guesses. This penalises listing many answers and rewards items where humans agree, so it is a reusable recipe for scoring open-ended outputs with multiple valid references.
• I3. Two scoring regimes side by side. The strict "best" score is paired with a lenient top-10 score in which credit is not split across the guesses submitted. This separates precision-oriented systems from recall-oriented ones and makes up for the small number of annotators.
• I4. Blind post hoc judging. Human and system answers were mixed in random order and rated by fresh judges. This checks whether low scores reflect bad outputs or simply valid answers the gold annotators missed, and fits any task whose gold set is incomplete.
• I5. Majority threshold for multiwords. An item counts as part of a multiword only if at least two annotators name the same expression. This is a cheap way to turn noisy, untrained judgements into usable gold data.