📄 Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection

Summary

Web-crawled and automatically mined parallel corpora often contain misaligned, partially translated or otherwise flawed sentence pairs, and NMT is more sensitive to such noise than SMT. Domain data selection is well studied, but its techniques had not been adapted for denoising. The authors (Google Research) assume a huge noisy corpus and only a small trusted set of a few thousand sentence pairs. They first fine-tune the noisy NMT model on the trusted data to get a slightly cleaner model. A sentence pair's noise is then scored by how differently the two seq2seq models treat it, which generalizes the bilingual cross-entropy difference (CED) of Axelrod et al. During online training, batches are chosen so the data the model sees becomes progressively cleaner, a curriculum-like schedule. On Paracrawl data, this gives large BLEU gains over random ordering, while a neural LM selector built the same way does not help. The work shows that a small amount of trusted data can make NMT training robust to heavy corpus noise.

Key points

• Trusted data: WMT newstest 2010–2011 served as the trusted set, and newstest 2012–2013 was used as the dev set. The approach needs only a tiny trusted fraction of the full corpus.

• Online denoising (P3) vs random order (P1) on Paracrawl:

  Test set	P1 (random order)	P3 (online denoising)
  n2014	31.6	35.2
  d2015	30.7	35.6
  patent	37.2	46.9

• Data order matters:

  • Reversing the schedule so the model sees clean data first (P4) yields only 32.6 / 31.1 / 40.9 BLEU on n2014 / d2015 / patent.
  • Successive fine-tuning on nested subsets (S 80% ⊃ S 40% ⊃ S 20%) improves the score at each step, ending at 34.4 / 34.6 / 45.6.

• NNLM selection fails at denoising:

  • Its scores do not track human quality ratings.
  • Fine-tuning on the top 20% chosen by NNLM (P9) gives 31.8 / 30.5 / 35.4, with patent dropping below the P1 baseline.
  • Fine-tuning on the top 20% chosen by the NMT-based method (P8) gives 34.3 / 34.7 / 45.8.

• Strong baseline: the WMT baseline W1, a standard LSTM RNN, reaches 36.2 detokenized BLEU on n2014 (40.2 tokenized).