📄 The “Meaning” system on the English all-words task

Summary

The paper describes the "Meaning" system, built within the Meaning European project, for the Senseval-3 English all-words word sense disambiguation (WSD) task. It joins seven supervised classifiers with five knowledge-based modules. The supervised learners are SVM (two variants), AdaBoost, cosVSM, ME, NB and DL. They were trained only on SemCor-1.6 and share one rich feature set: local patterns, wider-context words, syntactic relations, and semantic features from the Multilingual Central Repository (MCR). The knowledge-based modules are two Domain Driven Disambiguation variants, DOM, autoPS and MFS. They cover words that have too few training examples, and they run as a cascade ordered by precision. All combination settings were tuned on the Senseval-2 all-words data. The best tuned ensemble uses weighted voting over three classifiers ranked per frequency band, and it reached 67.5% precision on Senseval-2. On the Senseval-3 test set the system dropped to about 62.5% precision, which the authors attribute mostly to differences between the training and test corpora. The work shows both the practical value and the limits of combining many WSD modules when training data is sparse.

Key points

• A separate classifier was trained for each polysemous word with at least 10 SemCor-1.6 examples, giving 2,440 learning problems. WordNet-1.6 senses were then mapped automatically to WordNet-1.7 and 1.7.1.
• On covered polysemous Senseval-2 words, the individual supervised modules fell between 53.4% (DL) and 59.5% (SVM), with AdaBoost close behind at 59.1%.
• Among the unsupervised modules, the precision-oriented DDD reached 61.9% precision at only 48.8% coverage. The coverage-oriented DDD variant reached 50.2% at 99.6% coverage. autoPS (45.2%) was more precise than WordNet MFS (32.5%).
• On Senseval-2, the baselines scored 64.8%, 67.0% and 67.4%, and weighted voting with three classifiers scored 67.5% at 98% coverage. That result would have ranked second in the Senseval-2 all-words task.
• On Senseval-3, Meaning-c (cascade baseline) scored 61.1% precision and 61.0% recall. Meaning-wv (weighted voting) scored 62.5% precision and 62.3% recall, so the vote was more robust on unseen data.

Interesting ideas

• I1. Back-off modules are applied in a fixed order of measured precision, and each one may abstain so the next one decides, instead of mixing strong and weak modules in one vote. Mixing them hurt here, because the weak modules were far worse than the best one.
• I2. The same method is offered in two tunings, one maximizing precision and one maximizing coverage, and they are stacked so the precise variant answers first and the broad variant fills the gaps.
• I3. Classifiers are ranked separately for groups of words with similar amounts of training data, rather than by one global accuracy. This balances specialization against overfitting to a small tuning set.
• I4. Pairwise agreement between systems is measured before ensembling, to estimate how independent their errors are and whether a vote can help at all.
• I5. Two configurations that look equally good on the tuning set behaved differently on new data, and both lost about 5 points because the training and test corpora differ.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): When choosing a WordNet sense or ontology node for an entity, Mova could try high-precision evidence first and fall back step by step, recording which tier made the decision so the choice stays explainable.
• I2 (frames): Semantic role or frame assignment could pair a strict, high-precision rule set with a looser fallback, keeping the strict answers apart as more trustworthy.
• I3 (learn): Witness parsers that build silver data could be weighted per frequency band of lemma or construction, because a parser that is best overall may not be best on rare items.
• I4 (learn): Before trusting agreement among witness parsers as a silver label, Mova could check how often each pair agrees; witnesses that almost always agree add little independent evidence.
• I5 (learn): Evaluations should keep a held-out set from a different domain alongside the tuning set, so that ties on the tuning set are broken by robustness rather than chance.