📄 Word-Sense Disambiguation Using Statistical Methods

Summary

Statistical MT systems of this period relied on local models: a trigram language model and a word-alignment translation model. As a result, they mistranslated ambiguous words whenever the deciding context lay more than a couple of words away. For example, French prendre came out as "take" instead of "make" when décision was not adjacent. The authors add a step to the analysis phase that tags each such word with one of two senses before translation. They do not use dictionary senses: a distinction counts only if it predicts a different translation. The training data come from Viterbi alignments of roughly one million short French–English sentence pairs from the Canadian Hansard. For each word, the method checks several candidate context positions and picks the binary question that carries the most mutual information with the word's translation, using the flip-flop algorithm. Applied to frequent words in both languages, the sense labels lowered the translation system's error rate by 13%. The work shows that useful word senses can be learned from parallel text without any lexicon and can directly benefit a downstream application.

Key points

• Training data: 1,002,165 short sentence pairs from the Canadian Hansard. Their Viterbi alignments produced 12,028,485 word connections, which were used to estimate the joint distribution p(e, f).
• The question for each word is built with the flip-flop algorithm of Nadas et al. This algorithm uses the splitting theorem of Breiman et al., which finds the best binary split of a vocabulary of size V in linear time instead of searching all 2^V options.
• Pilot experiment: each of the 500 highest-frequency English words and each of the 200 highest-frequency French words was given two senses.
  • French words could use seven informants: adjacent words, nearest nouns and verbs on either side, and verb tense.
  • English words were limited to the one or two preceding words, because the decoder builds English output left to right.
• Example information gains:
  • prendre: noun to the right, .381 bits.
  • vouloir: its tense, .349 bits.
  • depuis: next word, .738 bits; it becomes "since" with probability .772 before an article, versus .016 otherwise.
  • English cent: previous word, .378 bits; this separates currency use from "per cent".
• Integrating the senses into the statistical MT system reduced its error rate by thirteen percent.

Interesting ideas

• I1. Define senses by translation. A sense distinction matters only if it changes the translation, so senses the target language does not separate are merged. One example is French intérêt, which needs two senses for German but one for English. This task-driven notion of ambiguity could suit Ukrainian–English or Ukrainian–Russian work, where dictionary senses are often too fine or too coarse.
• I2. Let the data choose the context feature. The method builds one question for each candidate context site (word to the right, nearest noun, verb tense, and so on) and keeps the most informative one. This is a cheap way to discover which piece of context actually resolves a given word, and it transfers to other feature-selection problems.
• I3. Alternate between two clusterings. The flip-flop procedure switches between splitting the context vocabulary and splitting the word's translations. Each step can only raise the mutual information, so the process converges. This alternating scheme is reusable for any setting where two sets of categories must be grouped to explain each other.
• I4. Match features to the decoder. English-side questions were restricted to previous words so they would fit a left-to-right generator. A placeholder token (TERM_WORD) handles cases where an informant is missing. Both are practical tricks for adding context features to incremental systems.
• I5. Measure the gain in bits and in end-task errors. Each question is scored by the entropy reduction it gives on translations, and the whole component is judged by the MT error rate. This makes it clear how much a sense split helps before and after integration.