Facebook FAIR's WMT19 News Translation Task Submission

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5333.pdf

Summary

This paper presents Facebook AI Research's systems for the WMT19 news translation shared task in four directions: English↔German and English↔Russian. All systems work at the sentence level. They are built with the FAIRSEQ toolkit on big Transformer models, with a wider feed-forward layer (FFN 8192) and BPE subword units.

The pipeline has five main steps:

• Bitext cleaning: langid language identification plus length and length-ratio rules.
• Back-translation: large-scale back-translation of Newscrawl, done by sampling from an ensemble of reverse-direction models.
• Extra Russian data: because Russian Newscrawl is small, an in-domain slice of Russian Commoncrawl is added, chosen with Moore–Lewis cross-entropy difference scoring using KenLM n-gram models.
• Adaptation: models are fine-tuned on earlier newstest sets and News-Commentary, then ensembled.
• Decoding: n-best lists are reranked with a noisy channel model that combines forward, channel (reverse-direction) and Transformer language model scores.

Each step adds measurable gains. Reranking still helps even on top of strong ensembles. The En→De system beats the team's WMT18 entry by about 4.5 BLEU, and the En→Ru submission achieved the top case-sensitive BLEU in the competition. The work shows that monolingual data and reranking remain valuable even for high-resource language pairs.

Key points

• Bitext preprocessing: filtering removed roughly 30% of the bitext. It dropped sentences over 250 tokens and pairs whose source/target length ratio exceeded 1.5. En–De used joint BPE with 32K merges. En–Ru used separate 24K BPE per language, which worked clearly better than joint BPE.
• Back-translation setup: synthetic data was sampled from an ensemble of three target-to-source models and mixed 1:1 with bitext. Moore–Lewis selection kept about 5% of Russian Commoncrawl (1.2B → 60M sentences).
• Noisy channel reranking: the forward model generates a beam-50 n-best list. Weights and length penalty were tuned by random search on newstest2014+2016. Reranking added 1.0 BLEU (En→Ru) to 1.5 BLEU (De→En).
• WMT19 test BLEU: En→De 42.7, De→En 40.8, En→Ru 36.3 (best case-sensitive score), Ru→En 40.0.
• Training scale: models were trained on 128 Volta GPUs. The weaker Russian language model, trained on less data, likely limited reranking gains for En→Ru.