📄 Experiments in Domain Adaptation for Statistical Machine Translation

Summary

The WMT 2007 shared task centered on domain adaptation: building statistical MT systems for news commentary when nearly all available parallel text comes from European Parliament proceedings. Koehn and Schroeder used the phrase-based Moses decoder to compare several ways of combining a large out-of-domain corpus (Europarl, about 40M words) with a small in-domain one (NewsCommentary, about 1M words), working on French–English. They tested training on each corpus alone, simple concatenation, an LM trained only on in-domain data, a perplexity-tuned interpolated LM, two separate LMs weighted by minimum error rate training, and two separate translation tables used as alternative decoding paths. Every adaptation method beat the three baselines. The two-translation-model setup scored highest (27.64 %BLEU versus 26.69 for plain concatenation). The paper also documents Edinburgh's 14 submitted systems, which ranked at or near the top in most tasks. The work matters because it shows that cheap biasing of the language and translation models toward in-domain data reliably improves output when in-domain resources are scarce.

Key points

• Data: Europarl has about 1.26M sentence pairs (≈37.5M French and 33.8M English words). NewsCommentary has 42,884 pairs (≈1.2M French and 1.0M English words).
• French–English results (%BLEU on nc-devtest-2007):
  • Europarl only: 25.11
  • NewsCommentary only: 25.88
  • Concatenated: 26.69
  • In-domain LM: 27.46
  • Interpolated LM: 27.12
  • Two LMs: 27.30
  • Two translation tables: 27.64
• Submitted systems: These used the interpolated LM (built with SRILM). The tuned Europarl LM weight ranged from 0.40 to 0.51 depending on the language pair.
• Training and decoding settings: Raising the GIZA++ training sentence length cap from 40 to about 80 helped. Enlarging the stack size (200→1000) and the ttable-limit (20→100) changed BLEU by under 0.05.
• Practical issues:
  • Old MERT weights were reused for most systems, and the word-count weight was fixed by hand once, when the output/reference length ratio fell to 0.88.
  • A wrong detokenizer cost English–French 2–3 BLEU points.
  • German input was reordered with Collins et al. (2005) rules.

Interesting ideas

• I1. Tune LM interpolation by perplexity: The mixing weight between in-domain and out-of-domain LMs is chosen by minimizing perplexity on a target-domain dev set. The resulting curve is smooth, so a coarse grid search suffices. This is a cheap way to adapt any n-gram or mixture model to a niche domain.
• I2. Keep domain models as separate log-linear features: Rather than merging data, each domain's LM or phrase table can enter the log-linear model as its own feature, with weights set by translation-quality tuning. This lets the optimizer decide how much to trust each source, which is useful whenever corpora of very different sizes must be combined.
• I3. Use alternative decoding paths for domains: Moses's factored framework allows two phrase tables, one in-domain and one out-of-domain, to be offered as competing paths during decoding. This avoids the small corpus being swamped by the large one and gave the best score in the study.
• I4. Separate translation knowledge from target-side style: Training the translation model on all data while using an in-domain-only LM already captured most of the gain. Broad data can supply coverage while a small target-domain model steers word choice.
• I5. Check output length as a quick sanity test: Monitoring the output/reference length ratio caught a mistuned word-penalty weight. This simple diagnostic is worth adding to any BLEU-driven pipeline where tuning is reused or skipped.