📄 Semi-Supervised Learning for Neural Machine Translation

Summary

Standard NMT systems learn only from parallel text. Parallel data is scarce, narrow in domain and uneven in quality, while monolingual text is plentiful. The authors propose a semi-supervised scheme that trains a source-to-target model and a target-to-source model together. The training objective adds autoencoder-style reconstruction terms for monolingual sentences to the usual likelihood on parallel data. A monolingual sentence is "encoded" by translating it into the other language, which acts as a latent sentence. It is then "decoded" back with the reverse model, and the reconstruction probability is maximized.

Summing over all possible latent translations is intractable, so the method approximates it with a top-k candidate list, where k=10 is enough. The approach does not depend on the network architecture and can use monolingual corpora in both the source and target languages.

On Chinese-English NIST data, it significantly outperforms strong SMT and NMT baselines in both directions. It also beats the back-translation method of Sennrich et al. (2015). This makes it a simple, architecture-agnostic way to make use of unlabeled text.

Key points

• Training objective. The joint objective has four terms:

  • source-to-target likelihood,
  • target-to-source likelihood,
  • a target autoencoder,
  • a source autoencoder.

  The autoencoder terms are weighted by hyper-parameters λ1 and λ2.

• Data. The parallel Chinese-English corpus has 2.56M sentence pairs (67.54M Chinese and 74.82M English words). The monolingual corpora contain 18.75M Chinese and 22.32M English sentences.

• Approximation and batching. The intractable sum over latent translations is replaced by a top-k list of candidates, with k=10. Each SGD step combines a parallel mini-batch with one mini-batch from each monolingual corpus.

• Which monolingual data helps. Target-side monolingual data helps source-to-target translation more than source-side data. Using both together brought no further significant gain.

• Comparison with back-translation. The method gains up to +1.8 BLEU (Chinese-to-English) and +1.0 BLEU (English-to-Chinese) over Sennrich et al. (2015) using the same corpora.

Interesting ideas

• I1. Translations as a discrete latent layer. Treating a translation as the hidden layer of an autoencoder lets two opposite-direction models supervise each other on unlabeled text. The same idea could apply to any task with a natural inverse, such as transliteration or text normalization.
• I2. Iterative back-translation. The method can be read as back-translation that keeps refreshing its synthetic data. Re-translations of monolingual sentences improve as training proceeds, whereas a one-shot pseudo-parallel corpus stays fixed. This argues for regenerating synthetic data during training rather than freezing it.
• I3. Top-k instead of the full sum. Approximating an exponential sum over output sequences with a small n-best list (k≈10) is a cheap, practical trick. It is reusable wherever a loss requires marginalizing over generated sequences.
• I4. Architecture-agnostic design. Monolingual data enters only through the loss, not through changes to the network. The scheme can therefore be dropped into other end-to-end models without redesign, which is useful for low-resource pairs such as Ukrainian with limited parallel data.
• I5. Tracking pseudo-label quality over time. Inspecting the best reconstructions of monolingual sentences at several training checkpoints is a simple qualitative diagnostic. It shows whether the pseudo-labels are actually getting better.