Iterative Back-Translation for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-2703.pdf

Summary

Neural machine translation (NMT) can use target-language monolingual text by back-translating it into the source language. This creates synthetic sentence pairs. The authors first test whether the quality of the reverse-direction system matters. On WMT 2017 German↔English, they compare back-translation systems trained for different lengths of time and find that stronger reverse models produce more useful synthetic corpora.

Building on this, they propose iterative back-translation. Systems in both directions take turns regenerating synthetic data for each other, so each round can draw on monolingual text from both languages. This resembles co-training. Experiments use the Marian toolkit for the high-resource setting and Sockeye for low-resource settings. The low-resource settings are a simulated English–French setup with 100K and 1M sentence pairs and a more realistic English–Farsi setup.

Repeating the process gives consistent gains over a single round of back-translation. Deep ensembles trained this way exceeded the best WMT 2017 German↔English submissions. This makes the method a simple and practical way to get more value from abundant monolingual data.

Key points

• Reverse-model quality matters. A reverse model stopped after 10k iterations added nothing. Training for 100k iterations gave +1.5 BLEU in both directions, and a converged model gave +2.9 (de→en) and +2.2 (en→de) on newstest2017.
• High-resource gains. One extra round (re-back-translation) improved final systems by 0.5–1.1 BLEU across shallow, deep and ensemble architectures. The best results were 36.5 BLEU for de→en and 29.3 for en→de, about 1 BLEU above the top WMT 2017 entries (35.1 and 28.3).
• Low-resource English–French. With 100K pairs, plain back-translation added +5.4 BLEU for fr→en and +3.5 for en→fr. Extra iterations helped further, with 1 and 2 additional rounds performing about the same.
• English–Farsi. Gains were small (+0.3 to +0.4 BLEU), possibly because of the language's difficulty and a domain mismatch between the parallel and monolingual data.
• Extra findings. Raising the ratio of synthetic to real data up to 1:3 helped slightly. Using Moses phrase-based SMT for back-translation gave mixed results, but scores stayed close to those from neural back-translation.