📄 A Phrase-Based,Joint Probability Model for Statistical Machine Translation

Summary

Marcu and Wong address a limitation of the noisy-channel translation models of the time, such as the IBM models: their lexicons map single words to single words, so they cannot capture cases where a multi-word phrase corresponds to one word or another phrase. The authors propose a joint probability model in which abstract "concepts" generate a source phrase and a target phrase at the same time. The joint model can then be marginalized into conditional models for either translation direction. Model 1 uses uniform phrase ordering. Model 2 adds an absolute position-based distortion component. Because exhaustive EM is infeasible, training proceeds in four steps:

1. Restrict phrases to frequent n-grams and unigrams.
2. Initialize the t-table with combinatorial counts based on Stirling numbers.
3. Run EM over approximate Viterbi alignments, with smoothing.
4. Derive the conditional probabilities.

A greedy hill-climbing decoder uses the learned phrase table. On French-English Hansard data, the system beats IBM Model 4 in both the share of perfect translations and BLEU. This shows that learning phrases and words together inside one model improves translation quality.

Key points

• Training data was 100,000 French-English sentence pairs from the Hansard corpus, none longer than 20 words. In English there were 1,073,480 tokens (21,484 types), and in French 1,177,143 tokens (28,132 types).
• The test set had 500 unseen sentences, 100 each at lengths 6, 8, 10, 15 and 20. Output was compared with IBM Model 4 (trained with Giza) decoded by the Germann et al. (2001) decoder.
• Perfect translations, judged manually, averaged 28% for the phrase-based joint model versus 22% for IBM Model 4. Average IBM-Bleu was 0.2325 versus 0.2158.
• Practical limits were applied to keep memory and training cost manageable:
  • Phrases were capped at six words per side.
  • A phrase needed at least five corpus occurrences to enter the initial table.
  • The language model was word-level, built with the CMU Language Modeling Toolkit.
• Only contiguous phrases are modeled, so a discontinuous pair like French "ne … pas" for English "not" cannot be learned directly. The model compensates by memorizing longer phrase pairs such as ("ne est pas", "is not").

Interesting ideas

• I1. Joint rather than conditional modeling. The model generates both sides of a sentence pair together and marginalizes afterwards. This yields translation tables for both directions from a single training run, which helps whenever symmetric resources are needed, such as bilingual lexicons for a low-resource language pair.
• I2. Initialization by counting. Before any training, the fraction of all possible alignments that link a given pair of phrases is estimated using Stirling numbers of the second kind. This gives EM a principled, data-driven starting point instead of uniform guesses, and the same trick could seed other latent-alignment models.
• I3. Cheap candidate pruning plus smoothing. Explicit entries are kept only for frequent n-grams and unigrams, and smoothing provides coverage for rarer phrases. This general pattern keeps huge parameter tables tractable in many count-based models.
• I4. One model for words and phrases. Words and phrases are treated identically and learned together, rather than extracting phrases separately and bolting them onto a word model. The authors argue this helps EM reach better optima and stops the decoder from discarding good phrasal translations. It is a useful design caution for any pipeline that mixes separately learned components.
• I5. A length-neutral distortion design. The distortion model is built so that the number of probability factors does not depend on target phrase length. This avoids a built-in bias toward short outputs, a property worth checking in any generative scoring function.