📄 A Comparison of Alignment Models for Statistical Machine Translation

Summary

The paper asks how to judge statistical word alignment models directly, rather than only through downstream translation output. Och and Ney compare IBM Models 1–4, a diagonal-oriented Model 2 and an extended Hidden-Markov (HMM) alignment model on the German–English Verbmobil Task, which covers appointment scheduling, travel and hotel dialogues. Their HMM extensions add an empty word, smoothing, and alignment probabilities conditioned on automatically trained word classes. They also inject a bilingual dictionary into training. Quality is measured by comparing each model's Viterbi alignment against a hand-made reference built with a sure/possible annotation scheme, scored by a new alignment error rate (AER). Model 4 aligns best, and the first-order HMM beats the fertility-based Model 3. Better alignments also improve subjective translation quality in an alignment template system. The work matters because it supplies a reusable, automatic and fine-grained benchmark for alignment design choices, and its AER metric became a standard in the field.

Key points

• About 1.4% of the training corpus (34,446 sentence pairs) was aligned by hand by two annotators. The reference keeps only the sure (S) links both annotators agree on, plus every possible (P) link either one marked. AER is computed from the overlaps of a predicted alignment with S and with P.

• In the English→German direction with the dictionary and empty word, the AER results were:

  • Model 4: 6.5%
  • HMM: 8.0%
  • Model 3: 8.5%
  • Model 1: 16.0%

• The dictionary helps much less than switching to a better model.

• Conditioning on 50 word classes cut AER further, by 0.9 points for HMM (to 7.1%) and 0.5 points for Model 4 (to 6.0%). Alignment quality differs strongly by direction, because German compounds cannot map to several English words when German is the source.

• Translation used the alignment template system, trained on 58,332 sentences:

  Alignments from	WER	SSER
  Model 1	49.8%	22.2%
  HMM	47.7%	19.3%
  Model 4	48.6%	16.8%

  WER barely changed, while human-judged SSER improved clearly.

• Model 3 and Model 4 were trained with an extension of the Giza toolkit, each model initialised from the previous one. The empty word's alignment probability was fixed to a constant because of model deficiency.

Interesting ideas

• I1. Two-tier gold annotation. Annotators mark links as either certain or merely plausible, and the metric does not penalise a system for matching plausible links. This reduces disagreement on idioms and free translations, and the same idea would suit any ambiguous annotation task, such as word alignment for Ukrainian or entity linking.
• I2. Combining annotators asymmetrically. Intersecting the sure sets and uniting the possible sets gives a strict core and a permissive outer layer. It is a cheap way to merge two annotators without adjudication.
• I3. Dictionaries as weighted pseudo-sentences. The bilingual lexicon is simply appended to the parallel corpus, with entries that actually co-occur in the data upweighted tenfold. This plugs prior knowledge into EM with no model changes, and it could help low-resource language pairs.
• I4. Position-free jump modelling with doubled empty-word states. Alignment probabilities depend only on jump width. Each target word gets its own empty-word state that remembers the previous position, so the HMM can produce unaligned words without breaking distance modelling. The trick generalises to other sequence models that need an empty or null state.
• I5. Checking automatic metrics against human judgment. WER stayed flat while human SSER separated the models. This is a useful reminder to validate intrinsic gains with human evaluation before concluding that a change has no downstream effect.