📄 English Senseval: Report and Results

Summary

This paper describes the English part of SENSEVAL, the first open, shared evaluation of word sense disambiguation (WSD), held in 1998 and modelled on MUC-style campaigns. The organisers chose a lexical-sample design: systems tagged only selected target words. The main reason was that many systems needed per-word training data or hand-built input. Both the sense inventory and the corpus came from the HECTOR database. Target words were picked through a sampling frame based on BNC frequency and WordNet polysemy, so the choice did not depend on taste. Professional lexicographers built the gold standard, with multiple taggers and an arbitration step, and a repeat of the process showed it was highly replicable. Seventeen systems took part, split into supervised and nonsupervised groups, and were compared with Lesk-style baselines. With training data, fine-grained WSD reached about 75–80%, and a carefully built simple Lesk was hard to beat. The paper also shows that sense-distribution entropy predicts difficulty better than raw polysemy. The exercise set a template for later rigorous WSD evaluation.

Key points

• Scale: 35 words formed 41 tasks (15 noun, 13 verb, 8 adjective, 5 POS-indeterminate). The evaluation set had 8448 instances, and training data ranged from none (5 words) to 26–2008 instances per word.
• Gold-standard quality: individual lexicographers scored 88–100% against the arbitrated tags, and only 5 of 122 ⟨lexicographer, word⟩ pairs fell below 95%. Fresh annotators re-tagged 1057 instances of four words and agreed with the earlier tags 95% of the time.
• Results: the best systems scored over 80% on nouns and around 70% on verbs. Most systems lost to the Lesk baseline for their category, and on the verb tasks no system cut the best baseline's error rate by more than 2%.
• Difficulty: system scores correlated with entropy at −0.510, compared with −0.258 for polysemy. For supervised systems the figures were −0.699 and −0.247. Generous and slight both have 6 senses, yet their entropies are 2.30 and 1.28.
• Systems that tagged with a different inventory (e.g. WordNet) and then mapped to HECTOR senses were heavily penalised by gaps and many-to-many mappings.

Interesting ideas

• I1. Instead of choosing test items by hand, the authors stratified them by frequency and ambiguity, which makes a benchmark more representative and harder to game.
• I2. They tested the replicability of the whole annotation pipeline, not only pairwise agreement: unseen annotators re-ran tagging plus arbitration, and the two outputs were compared.
• I3. A Lesk overlap baseline, weighted by IDF over dictionary definitions and examples, worked as a strong, knowledge-free control that most systems failed to beat.
• I4. Entropy of the observed sense distribution predicted task difficulty much better than the number of senses.
• I5. Systems that tagged only some items could not be fairly compared, because the items carried no difficulty grades.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Build evaluation sets by sampling lemmas or constructions across frequency × ambiguity bins, so scores do not just reflect the common head of the distribution.
• I2 (learn): Check silver data the same way by rebuilding a slice with a different set of witness parsers or LLM teachers and measuring how well the two silver versions agree.
• I3 (frames): Report every sense or frame chooser against an IDF-weighted gloss-overlap control from WordNet/FrameNet, so real gains from explainable structure can be told apart from lexical overlap.
• I4 (world): Store per-lemma sense entropy in the compiled knowledge layer and use it to decide where to spend disambiguation effort or to flag low-confidence readings.
• I5 (answer): Give test questions difficulty ratings so that readers which abstain on some items can still be compared on equally hard subsets.