Edinburgh Neural Machine Translation Systems for WMT 16

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W16-2323.pdf

Summary

This paper describes the University of Edinburgh's entries in the WMT16 news translation shared task. The systems cover English paired with Czech, German, Romanian and Russian, each in both directions. The core model is an attention-based encoder-decoder built on the dl4mt-tutorial code, which was later released as Nematus. Byte-pair encoding (BPE) splits words into subword units, so a fixed vocabulary can still represent rare and unseen words.

On top of this baseline, the authors test three additions:

• synthetic parallel data, made by back-translating in-domain News Crawl monolingual text;
• dropout on every layer, including the recurrent ones, used for the low-resource Romanian pair;
• re-scoring of n-best lists with models that generate the target sentence right-to-left.

Ensembles of training checkpoints add further gains. Together, these techniques raise BLEU by 4.3–11.2 points over the baselines. In the human evaluation, the systems were the best or tied-best constrained entry in 7 of 8 directions. The work showed that neural MT with inexpensive data augmentation could lead a major evaluation campaign. This recipe transfers directly to languages with limited parallel data, such as Ukrainian.

Key points

• Back-translation: 2–10M synthetic sentences per direction were generated, mostly from News Crawl 2015. For CS→EN, growing the synthetic set from 2.5M to 10M sentences helped steadily, though the first 2.5M brought most of the gain.
• BPE setup: 89,500 merge operations were learned jointly on source and target text. For EN↔RU, Russian was first transliterated to Latin script (ISO-9) to learn shared merges, which were then mapped back to Cyrillic.
• EN↔RO:
  • Pervasive dropout gave +4–5 BLEU. It reuses one mask across time steps, following Gal (2015), with word dropout at 0.1 and 0.2 on other layers.
  • Removing diacritics from the Romanian source gave +1.3–1.4 BLEU.
  • Synthetic data gave +4.1–5.1 BLEU.
• EN↔DE:
  • Synthetic data: +3.4–5.7 BLEU.
  • Ensemble of the last 4 checkpoints: +1.3–1.7 BLEU.
  • Right-to-left reranking of 50-best lists: +0.6–1.1 BLEU.
  • Final newstest2016 scores: 34.2 (EN→DE) and 38.6 (DE→EN).
• Released resources: the Nematus toolkit, the wmt16-scripts repository (configs, synthetic data, trained models), and subword-nmt for BPE. The AmuNMT C++ decoder was used for faster decoding.