📄 Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation

Summary

The paper presents Microsoft's English-German entries for the WMT19 news translation task, with a focus on translating whole documents rather than isolated sentences. The author first builds strong sentence-level baselines in the Marian toolkit. These combine dual cross-entropy data filtering with large-scale noisy back-translation of German News-Crawl. Evaluation uses a custom dev set made from test2016–2018, divided by the original language of each segment. This split shows that back-translation mostly helps when the input is translationese. Fine-tuning on filtered parallel data recovers quality on genuinely English input. Deeper 12-layer transformers and weighted ensembles add further gains.

For document-level translation, document boundaries in authentic and synthetic data are used to build training sequences of up to 1000 subword units. The author also tests data augmentation, a BERT-style multi-task objective on the encoder, and second-pass post-editing models. Very long sequences cause no quality loss. In preliminary human evaluation, the document-level systems were clearly preferred over the sentence-level one and even scored above the human reference. This makes the work significant evidence that standard transformers can handle full documents.

Key points

• Tools and setup: Marian toolkit with SentencePiece and fp16 training on 8 Volta GPUs.

  • 12-layer models use residual initialization scaled by 1/√i, where i is the layer index.
  • Even deeper stacks (18 and 24 blocks) brought no improvement.

• Back-translation data: about 550M back-translated sentences were generated by output sampling, implemented with the Gumbel softmax trick.

• Dev set and fine-tuning: each half of the origin-split dev set holds about 4500 sentences. Fine-tuning reversed the loss that back-translation caused on originally-English text.

• Automatic results: cased BLEU on newstest2019 was:

  • WMT18 system: 41.9
  • sentence-level ensemble: 43.0
  • document-level ensemble: 43.9
  • hybrid with second-pass models: 43.9

• Human evaluation: in the preliminary source-based direct assessment, average raw scores were:

  • hybrid: 93.0
  • document-level: 92.6
  • HUMAN reference: 90.3
  • sentence-level: 86.6

  The author questions the official ranking based on z-scores.