📄 Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Summary

The authors from Google address the task of translating between many languages with a single neural model (NMT). Previously, a separate system was usually trained for each language pair. They left the GNMT architecture unchanged and modified only the input data: an artificial token indicating the target language, for example <2es>, is added to the beginning of the sentence. The model identifies the source language on its own. The encoder, decoder, attention mechanism, and wordpiece vocabulary are shared across all languages. The model was trained on mixed multilingual data from the WMT benchmarks and on Google's production corpora. A single such model performs on par with or better than the best systems of its time on WMT'14/WMT'15, and low-resource languages benefit from joint training with high-resource languages. Most importantly, the model can translate between language pairs it did not see during training (zero-shot translation). This approach simplifies the deployment of hundreds of language directions. It also shows that cross-lingual knowledge transfer is possible in neural translation and that a shared cross-lingual representation (interlingua) is likely formed.

Key points

• Method: the only change is a target-language token at the input. The model uses a shared wordpiece vocabulary, typically of 32,000 units (similar to BPE). Implemented in TensorFlow on top of GNMT.
• Results on WMT'14: for English→French the quality is comparable to the best systems, and for English→German it is better. For French→English (WMT'14) and German→English (WMT'15), the multilingual model also surpasses the then state of the art.
• On Google's production corpora, the models cover up to 12 language pairs in a single network. Many individual pairs are translated better as a result.
• Zero-shot: a model trained on Portuguese→English and English→Spanish produces acceptable Portuguese→Spanish translation. A small amount of data for this pair noticeably improves quality.
• t-SNE analysis showed that sentences with the same meaning in different languages group into shared clusters, which supports the interlingua hypothesis. The farther a zero-shot translation is from the "learned" representation, the lower the BLEU (Pearson correlation −0.42). The model also handles language switching within an input sentence (Japanese + Korean). When target-language tokens are mixed, the output usually switches from one language to the other at a weight of about 0.5, rather than producing a "hybrid" language.