📄 English Lexical Sample Task Description

Summary

The paper documents how the English lexical sample track of SENSEVAL-2 was built. It covered adjectives and nouns; verbs were handled with the all-words task. The organisers chose a fresh set of words spanning low, medium and high frequency. Contexts came mostly from the new edition of the British National Corpus, plus a small amount of Wall Street Journal text. The key change from SENSEVAL-1 was the sense inventory: the HECTOR lexicon was replaced by WordNet. Before use, a lexicographer revised the WordNet entries and merged senses that were too close to tell apart, and these edits were folded into WordNet 1.7. Recruited taggers, screened by a web-based test, annotated the data. Each item went to more annotators only when the first ones disagreed. Overall fine-grained inter-tagger agreement reached 85.5%. Even so, the best systems scored roughly 14% below their SENSEVAL-1 counterparts. The author attributes this drop to WordNet's less clear-cut sense distinctions. His conclusion is that WSD research should now focus on sense inventories rather than on algorithms.

Key points

• The sample had 15 adjectives and 29 nouns, giving 7,567 tagged instances (2,301 adjectives, 5,266 nouns), split 2:1 into training and test data in SENSEVAL XML format.
• The target number of instances per word was 75 + 15n + 6m, where n is the number of senses and m the number of multiword expressions the word belongs to.
• The annotation team had 3 professional lexicographers and 14 students or others. They were recruited through an online aptitude test whose gold answers came from SENSEVAL-1 data.
• 66.5% of items needed only two taggings and 32.3% needed three; just 90 items needed four or five. Fine-grained ITA was 83.4% for adjectives and 86.3% for nouns.
• Kappa was deliberately not reported. Results were published in two sets: the official submissions and later bug-fix submissions marked (R).

Interesting ideas

• I1. Annotation effort scales with difficulty: start with two annotators and add one at a time until at least two agree on a tag, so effort concentrates on genuinely hard items.
• I2. Annotators worked with short human-readable mnemonics for each sense instead of bare sense numbers, which are easy to mix up.
• I3. Submitted answers were checked automatically against the permitted labels for that word, and invalid ones were sent back for correction. Taggers could also give several tags, or mark an item as Unassignable or Proper-name, rather than being forced into one choice.
• I4. The paper argues that the true ceiling for a task is replicability, meaning how well two independent teams using the same method agree, not agreement within a single team.
• I5. Sense inventories were cleaned by merging distinctions a lexicographer could not justify, on the view that disambiguation quality depends more on the inventory than on the algorithm.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Our silver-data pipeline could call an extra witness parser or a costlier LLM teacher tier only when the cheaper witnesses disagree, saving compute on easy sentences.
• I2 (world): WordNet-derived sense IDs in our ontology could carry curated mnemonic labels, making frame and entity explanations readable for people debugging the world model.
• I3 (learn): LLM teacher outputs could be validated against the allowed VerbNet or sense inventory and re-queried when out of range, with explicit "none fits" and "several fit" options kept in the label schema.
• I4 (learn): When estimating upper bounds for silver data, we could compare outputs from independently configured annotation pipelines instead of witnesses that share training data.
• I5 (world): Before building frames on WordNet senses, we could collapse poorly motivated sense splits so that downstream readers are not penalised for distinctions nobody can reliably make.