📄 Classifier Optimization and Combination in the English All Words Task

Summary

The paper describes the CNTS-Antwerp entry to the SENSEVAL-2 English all words task. The goal was to test whether supervised word sense disambiguation, which had worked well on small lexical samples, could cover every content word in running text. The system uses a "word expert" design. Each word form-POS pair with enough SemCor training examples gets its own small ensemble: two TiMBL memory-based learners (one on local context, one on keyword features) and a Ripper rule learner. Words with a single WordNet1.7 sense, or with too few examples, get the WordNet sense or the most frequent sense. Every expert's parameters were tuned separately by cross-validation, and majority and weighted voting were compared with the single classifiers. On the 2,473 test words the system reached 63.61% fine-grained and 64.54% coarse-grained accuracy. Nouns scored best and verbs worst. Tuned memory-based learning usually beat both Ripper and the voting schemes. This suggests that per-word, cross-validated model choice is practical even with limited data.

Key points

• Training data is the SemCor corpus shipped with WordNet1.6: 409,990 word forms, of which 190,481 carry sense tags. The test set is three articles with 2,473 words to tag. Tokenization, lemmatization and tagging were done in-house with MBT, which was trained on the Wall Street Journal.
• Routing:
  • About 20% of test words have only one WordNet1.7 sense and get it directly.
  • Ambiguous words with fewer than 11 SemCor occurrences get the most frequent sense.
  • The rest, over 60% of test words, are handled by 568 word experts.
• Features and tuning:
  • The first TiMBL learner sees a ±3 word window with POS tags, plus the lemma and POS of the focus word.
  • The second TiMBL learner uses binary keyword features. These come from a three-sentence context, filtered by count and probability thresholds (set to 3 and 0.001), and from the WordNet example sentences.
  • Ripper gets the local context plus a set-valued bag of context content words.
• Optimization:
  • The first learner used backward sequential selection, four feature-weighting options and k from 1 to 19, evaluated by leave-one-out.
  • The keyword learner used 10-fold CV over k, weighting schemes and distance metrics.
  • Ripper kept its default settings.
• Results by POS, fine-grained: nouns 74.51%, verbs 47.83%, adjectives 62.58%, adverbs 73.42%. All 86 "unknown" items, which had no applicable WordNet sense, were wrong.

Interesting ideas

• I1. Instead of one global model, each ambiguous lemma-POS pair gets its own classifier and its own tuned settings, so hard words and easy words are not forced into the same configuration.
• I2. A frequency threshold sends rare words to a safe default (the most frequent sense) and only trains models where data supports it; this makes coverage complete and the fallback explicit.
• I3. Gloss example sentences from the lexicon are mined as extra keyword evidence, turning dictionary content into training signal without new annotation.
• I4. Voting was evaluated against single classifiers per word rather than assumed to help, and often a single tuned learner won.
• I5. Separating results by POS and by "unknown/no applicable sense" items shows where errors really come from (verbs, out-of-inventory items).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): Sense choice for VerbNet/FrameNet frame assignment could use small per-verb experts, each with its own inspectable features, instead of one opaque disambiguator.
• I2 (world): When mapping words to WordNet/ontology nodes, Mova could record whether a sense came from a trained expert or a frequency default, keeping the explanation honest.
• I3 (learn): WordNet/FrameNet example sentences could seed silver training data for sense or frame classifiers before any LLM teacher is consulted.
• I4 (learn): When combining witness parsers for silver data, Mova should measure per-construction whether agreement voting actually beats the best single witness instead of always voting.
• I5 (answer): Evaluation reports could split accuracy by category and count items outside the inventory separately, acting as a built-in negative control.