Elamur Research
☰
home / papers / 2018 / A neural interlingua for multilingual machine translation

A neural interlingua for multilingual machine translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-6309.pdf

Summary

Multilingual NMT usually takes one of two routes. One trains a separate model for every language pair, so the number of models grows quadratically. The other shares a single encoder-decoder and a joint vocabulary, which forces many languages into one fixed-size output layer. Lu et al. (Amazon) propose a middle path. Each language keeps its own BiLSTM encoder and attentional LSTM decoder. Between them sits a shared attentional LSTM layer, the neural interlingua, which converts any encoder's output into a fixed-length representation that all decoders consume. No language indicator tokens are used.

The model was trained on WMT15 data pairing English with French, German, Czech, Finnish and Russian, using a rotating schedule of language pairs. Despite having fewer parameters, it reaches BLEU comparable to separate bilingual baselines. It also performs direct zero-shot translation on the UN Parallel Corpus, outperforming a universal encoder-decoder baseline. Its sentence embeddings let a Yelp classifier trained only on English also handle French and German reviews. The work matters because components scale linearly with the number of languages, and a language-neutral representation supports crosslingual transfer.

Key points

  • Architecture: language-specific BiLSTM encoders and attentional LSTM decoders are connected through a shared attentional LSTM interlingua with a fixed output length of 50. Encoders, decoders and attention mechanisms all scale linearly with the number of languages.
  • Hyperparameters: vocabulary of 30,000 per language, embeddings of 256, hidden size 512, Adam with learning rate 0.0002, batch size 400. Importance sampling was used to speed up training.
  • Training schedule: each minibatch contains a single source-target pair. A naive schedule of only X→En and En→X failed to yield a shared representation. The final schedule (Algorithm 1) adds same-language pairs (S, S) for every language.
  • Zero-shot translation: direct translation without pivoting was tested on Fr↔Ru, Zh↔Es and Es↔Fr from the updated UN Parallel Corpus. Results improve on the universal encoder-decoder approach (Johnson et al., 2017) but still lag behind pivot translation.
  • Crosslingual transfer: an English Yelp review classifier built on interlingua embeddings also classifies French and German reviews. In a PCA projection to 2D, parallel sentences in different languages cluster together.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

50 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 44 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6309

@inproceedings{lu2018neural,
  title = {{A neural interlingua for multilingual machine translation}},
  author = {Lu, Yichao and Keung, Phillip and Ladhak, Faisal and Bhardwaj, Vikas and Zhang, Shaonan and Sun, Jason},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6309},
  url = {https://doi.org/10.18653/v1/w18-6309}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  3. Akiva Miura, Graham Neubig, Katsuhito Sudoh, and Satoshi Nakamura. 2017. Tree as a pivot: Syntactic matching methods in pivot translation. In WMT. find
  4. Christian Boitet. 1988. Pros and cons of the pivot and transfer approaches in multilingual machine translation. In Readings in machine translation, pages 273–279. find
  5. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. find
  6. Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. ACL. find
  7. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In NAACL-HLT. find
  8. Trevor Cohn and Mirella Lapata. 2007. Machine translation by triangulation: Making effective use of multi-parallel corpora. In ACL. find
  9. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  10. Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In WMT. find
  11. Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In LREC. find our card
  12. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In ACL. find our card
  13. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In NAACL-HLT. find
  14. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL-HLT. find our card
  15. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In EMNLP. find
  16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In ICML. find our card
  17. Thanh-Le Ha, Jan Niehues, and Alexander Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. In IWSLT. find our card
  18. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In ACL. find our card
  19. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: enabling zero-shot translation. In TACL. find
  20. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find