📄 Improved Statistical Alignment Models

Summary

Och and Ney address two problems in statistical word alignment for machine translation: there was no established way to evaluate alignment models directly, and the models themselves needed improvement. They propose evaluating a model by comparing its Viterbi alignment against a manually produced reference alignment. Annotators mark both certain ("sure") and ambiguous ("possible") links, and the comparison yields precision, recall and an alignment error rate (AER). Using this measure, they compare IBM-1 through IBM-5 and the HMM alignment model on German–English (Verbmobil) and French–English (Hansards) data. They also test several training refinements:

• smoothing of alignment and fertility parameters,
• word-class dependencies,
• different alignment subsets for EM count collection,
• bootstrapping schemes.

Fertility-based models with a first-order alignment dependency clearly beat the simple IBM-1 and IBM-2. Starting IBM-3 training from the HMM instead of IBM-2 further improves alignment quality. Combining the Viterbi alignments from both translation directions can raise precision or recall, depending on the merging method. The work matters because it provides a reproducible, automatic way to judge design decisions in alignment modelling.

Key points

• Six models are compared:

  • IBM-1 and IBM-2 use zero-order alignment.
  • The HMM uses first-order alignment.
  • IBM-3 and IBM-4 add explicit fertility.
  • IBM-5 is a non-deficient reformulation of IBM-4.

  An "empty word" absorbs source words that have no counterpart.

• Evaluation compares the Viterbi alignment with a hand-made reference containing sure and possible links, producing precision, recall and AER.

• Training uses EM over a parallel corpus.

  • IBM-1, IBM-2 and the HMM allow exact count collection; the HMM uses the Baum–Welch algorithm.
  • For IBM-3 to IBM-5, counts are gathered only over a small set of promising alignments. Three options are compared: the Viterbi alignment alone, its neighbours (one move or swap away), and an even larger set that includes "pegged" alignments.

• Using the HMM rather than IBM-2 as the starting point for IBM-3 to IBM-5 significantly improves alignment quality.

• Word classes are learned automatically with a separate statistical clustering procedure (Kneser & Ney). They condition the alignment probabilities in the HMM, IBM-4 and IBM-5.

Interesting ideas

• I1. Intrinsic evaluation against a gold standard with graded confidence. Separating sure from possible links lets the reference reflect genuine annotator uncertainty without penalising either choice. The same scheme could work for evaluating other ambiguous structure-prediction tasks, such as aligning Ukrainian–English parallel data or matching entities across datasets.
• I2. Length-aware fertility smoothing. Fertility estimates are interpolated with a general distribution that depends only on word length in letters, since longer (e.g. German compound) words tend to align to more words. The specific estimate dominates for frequent words and the general one for rare words. This is a cheap backoff trick for any morphologically rich language with sparse vocabulary, Ukrainian included.
• I3. Interpolation with a uniform distribution, tuned on held-out data. Alignment probabilities are mixed with a constant distribution, and the mixing weights are chosen to maximise alignment quality on a validation set. Tuning smoothing for the downstream metric rather than likelihood is broadly reusable.
• I4. Bootstrapping complex models from better simple ones. Initialising an expensive model from a stronger intermediate model (HMM instead of IBM-2) improves the final result. This is a reminder that training pipelines and initialisation order deserve explicit experiments.
• I5. Symmetrising directional outputs. Merging the alignments from both translation directions by intersection, union, or a refined in-between rule lets you trade precision against recall. The same applies to combining outputs of any two asymmetric predictors.