📄 HIT: Web based Scoring Method for English Lexical Substitution

Summary

The paper presents HIT, a system for the SemEval-2007 English Lexical Substitution task. Given a target word in a sentence, the system must propose words that could replace it in that context. HIT works in two stages. First, it gathers candidates from WordNet. It takes synonyms from every synset of the given POS, uses the nearest hypernym's synonyms when a synset has none, and swaps between adjective and adverb when nothing is found. Second, it ranks the candidates by web evidence. It puts each candidate into short fragments cut from the sentence around the target, sends them to Google as queries, and averages the snippet counts. The fragments are built by rules rather than a fixed window, so each one is just long enough to be informative. Multiword expressions are detected by looking up WordNet, trying the longest strings first. HIT was the only entry in all three subtasks (best, oot, mw). It placed fourth in best and seventh in oot, and it beat the WordNet and LIN baselines. Error analysis shows that candidate coverage, not ranking alone, is the main bottleneck.

Key points

• Results: best P = R = 11.35 (ModeP 18.86) and oot P = R = 33.88 (ModeP 46.91). The WORDNET baseline scored 9.95 on best and 29.70 on oot.
• The mw subtask scored 45.34 P / 56.15 R for detection and 41.61 P / 51.54 R for identification. Multiwords were searched in a window of 2 words on each side of the target.
• Coverage was the main weakness: only 2,168 of the 6,873 gold substitutes (31.54%, across 1,710 items) appeared among the WordNet candidates. Of those found, 1,388 (64.02%) were ranked in the top 10.
• If fewer than 10 candidates had a nonzero web score, the remaining oot slots were filled using a word frequency list. The best answer was simply the top oot candidate.
• The authors name web querying as slow and plan to use Google's Web 1T 5-gram (LDC2006T13) instead. They also plan to try other thesauri, such as Oxford American Writer Thesaurus and Encarta.

Interesting ideas

• I1. Fragment length adapts to the sentence. Each fragment is the shortest span that has at least two other words, one of them a content word, and that does not cross a comma. This avoids windows that are too strict or too loose.
• I2. Each candidate is scored with up to three fragments, placing it at the start, middle and end. Any fragment that contains another one is dropped. Averaging over several local views makes the score more robust than relying on one context.
• I3. Missing synonyms are filled with the nearest hypernym's synonyms, and adjective/adverb POS is swapped as a fallback. This is a clear, ordered backoff over a lexical resource.
• I4. Multiword lookup tries the longest string first and skips its substrings. For example, finding "get along with" prevents a separate match for "get along".
• I5. The error analysis separates recall of the candidate generator from the quality of the ranker. This shows clearly which stage limits the system.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Checks on compiled knowledge, such as verb–object plausibility, could query n-gram counts with such minimal, clause-bounded fragments instead of fixed windows.
• I2 (learn): A silver-data witness could judge a lexical choice from several short local contexts and accept it only when the views agree, with a visible count behind each one.
• I3 (world): Lookups in the WordNet-based ontology could use the same documented backoff chain, logging which step produced each answer so the result stays explainable.
• I4 (tokens): The lemmatizer could apply longest-match-first multiword grouping from WordNet before tagging, so phrasal verbs reach the UD tree as one unit.
• I5 (learn): Evaluation could report the coverage of each stage's proposal set separately from its ranking accuracy, to find where answers are lost.