Tagged Back-Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5206.pdf

Summary

Back-translation (BT) creates synthetic parallel data by translating target-language monolingual text into the source language. Edunov et al. (2018) found that adding noise to beam-search BT output (NoisedBT) improves NMT, and attributed this to greater source-side diversity. Caswell, Chelba and Grangier challenge that explanation. They argue that noise mainly lets the model recognize which training sources are synthetic. This allows it to learn from the useful target-side signal while discounting the artifacts of machine translation.

To test this, they replace noise with a single reserved <BT> token placed before each back-translated source sentence (TaggedBT). They train Transformer models in lingvo on WMT EnDe and WMT16 EnRo bitext plus large Newscrawl back-translations, and evaluate with SacreBLEU. TaggedBT equals or slightly beats NoisedBT on EnDe. On EnRo it beats both standard and noised BT, and it makes iterative BT work where untagged BT degraded. Attention and tagged-decoding analyses support the domain-marker interpretation. The method is simpler and more robust than noising and gives a new state of the art on EnRo.

Key points

• Data: about 5.0M filtered EnDe bitext pairs with 216.5M back-translated sentences, and about 0.6M EnRo pairs. EnFr was also explored. Models used transformer-base/big, 32k subwords and EMA weights.
• EnRo results: iterative TaggedBT reached 33.4 SacreBLEU on WMT16 EnRo, about +3.5 over the best previously published figure the authors knew of. Plain TaggedBT gained about +0.4 over vanilla BT.
• EnDe results: TaggedBT was on par with or above the best Edunov et al. (2018) model on five of seven test sets. The largest gain was +1.4 on Newstest2018.
• Evidence against the diversity explanation: noising bitext sources barely hurt quality. Training only on NoisedBT data scored about 1 BLEU below training on clean BT alone.
• Tagged decoding: feeding the <BT> tag at test time raised the source-target copy rate from 8.9% to 10.7%, close to the 11.4% seen in BT data. This suggests the tag acts as a domain marker. Tagging forward-translation brought no benefit.