📄 SW-AG: Local Context Matching for English Lexical Substitution

Summary

The paper describes two unsupervised, knowledge-light systems for the SemEval-2007 English lexical substitution task. Each system proposes ten suitable replacements for a marked word in a sentence. The authors use a "direct" approach and avoid choosing a word sense first. They build a substitution lexicon from Roget's New Millennium Thesaurus (RT), keeping only entries that match the target exactly and removing rare candidates using Web 1T 5-gram unigram counts. The first system, SWAG1, ranks candidates by how often the trigrams around the target, with the candidate swapped in, occur in Web 1T. Each trigram's contribution is normalized. The second system, SWAG2, adds a cosine nearest-neighbour model built on British National Corpus sentences. It uses thesaurus "senses" only to filter candidates and picks the best model per part of speech. SWAG2 scored higher on the out-of-ten metric, with precision 37.80 against 35.53. The results suggest that substitution can work well without explicit sense disambiguation.

Key points

• On trial data, the oracle maximum recall was 53% with a WordNet-synset lexicon and 85% with RT. Dropping the least frequent third of candidates lowered this ceiling to 75% but clearly improved precision.
• LCM draws on as many as three trigrams from the test sentence that include the target word. It discards those without another content word, using the NLTK-Lite tagger, and matches the rest against more than 967 million unique Web 1T trigrams after lemmatization.
• Filtering by sense raised nearest-neighbour accuracy from .41 to .44. Keeping only the top-ranked sense for nouns raised LCM noun accuracy from .51 to .57, counting only instances with a mode.
• The combined system was assembled per part of speech. Filtered LCM was used for nouns and adjectives, filtered nearest-neighbour for adverbs, and the combined voter for verbs.
• OOT results were P 35.53 / R 32.83 / Mode P 47.41 for SWAG1 and P 37.80 / R 34.66 / Mode P 50.18 for SWAG2. Both systems did better on instances where annotators agreed on a mode.

Interesting ideas

• I1. Before tuning a ranker, measure the oracle ceiling of the candidate generator, meaning the share of gold answers that are reachable at all, because that limit caps every later stage.
• I2. Test a candidate by swapping it into the actual local word context and checking whether the result is attested in a large corpus. Normalize per context so that one very frequent trigram does not dominate the decision.
• I3. Discard context windows that contain only function words besides the target, since they carry almost no evidence and only add noise.
• I4. Use sense information as a filter on directly generated candidates rather than as a required first decision, which avoids problems when the sense inventory is too fine or too coarse.
• I5. Prune rare, obscure alternatives by corpus frequency, accepting a lower recall ceiling in exchange for choices humans would actually make.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluations could report, for each stage, how much of the gold remains reachable from the previous stage's candidates, for example correct heads among the parser's options. Failures would then be split into generation misses and ranking misses.
• I2 (world): When choosing among WordNet synonyms or ontology labels for an entity, an explainable swap-and-check score against n-gram counts could justify the choice with concrete attested contexts.
• I3 (frames): When scoring VerbNet class or role candidates from nearby words, the evidence could be limited to windows containing a content word, so that weak cues do not outweigh informative ones.
• I4 (frames): VerbNet or FrameNet classes could prune frame candidates proposed directly from the UD tree, without forcing one sense choice up front.
• I5 (back): When generating text from the meaning graph, the system could prefer frequent lexicalizations of a concept so that paraphrases read naturally.