Elamur Research
☰
home / papers / 2016 / Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2016.iwslt-1.6.pdf

Summary

Attention-based NMT is usually trained for one language pair. Earlier multilingual extensions used several encoders or decoders and custom ways of sharing or merging attention, which adds parameters and complicates training. Ha, Niehues and Waibel propose a simpler design: a single shared encoder and a single shared decoder handle every language, and the standard attention model of Bahdanau et al. is kept unchanged. Their technique, Language-specific Coding, tags each word with its language, so the same surface form in two languages becomes two separate entries in a mixed vocabulary. The network can still place words with similar meanings close together in a common embedding space. Because the architecture is not modified, the model stays about the size of one bilingual system and trains in the usual way. In a simulated low-resource setting, adding data from other languages raised quality by up to 2.6 BLEU. The same framework also gave a new way to use monolingual text, adding 3.15 BLEU on IWSLT'16 English→German. This is useful for languages with little parallel data.

Key points

  • The approach uses one universal encoder and one universal decoder for all languages, supports many-to-many translation, and needs no network changes to use attention.
  • Language-specific Coding keeps words from different languages apart in the shared vocabulary. The free-parameter count stays at the scale of a single NMT system.
  • Results: up to +2.6 BLEU in a simulated under-resourced scenario, and +3.15 BLEU on the IWSLT'16 English→German task when monolingual data is used through the same framework.
  • The paper compares its method with multi-encoder/decoder systems by Luong et al., Dong et al., Firat et al. (shared attention) and Zoph & Knight (multi-source). Its setup draws on subword units (Byte-Pair Encoding), WIT3 data and ADADELTA.
  • Venue: IWSLT 2016, the 13th edition of the international workshop series devoted to translating speech. Future work proposed by the authors includes real (not simulated) low-resource tests and zero-resource translation without direct parallel data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 154 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ha2016toward,
  title = {{Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder}},
  author = {Ha, Thanh-Le and Niehues, Jan and Waibel, Alex},
  booktitle = {Proceedings of the 13th International Conference on Spoken Language Translation},
  year = {2016},
  url = {https://aclanthology.org/2016.iwslt-1.6/}
}

References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, R. Cattoni, and M. Federico, “The IWSLT 2015 Evaluation Campaign,” in Proceedings of the 12th International Workshop on Spoken Language Translation (IWSLT 2015), Danang, Vietnam, 2015. find
  2. O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, et al., “Findings of the 2016 Conference on Machine Translation (WMT16),” in Proceedings of the First Conference on Machine Translation (WMT16). Berlin, Germany: Association for Comp… find our card
  3. D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” CoRR, vol. abs/1409.0473, 2014. [Online]. Available: http://arxiv.org/abs/1409.0473 ↗ our card
  4. S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997. [Online]. Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735 ↗
  5. K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation,” in Proceedings of Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8. Baltimore, ML,… find
  6. M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 15. Lisbon, Portugal: Association for Computational Linguistics, September 2015, pp. 1412–14… find
  7. M.-T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, and W. Zaremba, “Addressing the rare word problem in neural machine translation,” in Proceedings of ACL-IJNLP 2015. Beijing, China: Association for Computational Linguistics, July 2015, pp. 11–19. [Online]. Available: http://www.aclweb.org/anthology/… ↗ our card
  8. R. Sennrich, B. Haddow, and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” in Association for Computational Linguistics (ACL 2016), Berlin, Germany, August 2016. find
  9. M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser, “Multi-task sequence to sequence learning,” in International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016. find our card
  10. D. Dong, H. Wu, W. He, D. Yu, and H. Wang, “Multi-Task Learning for Multiple Language Translation,” in Proceedings of ACL-IJNLP 2015. Beijing, China: Association for Computational Linguistics, July 2015, pp. 1723–1732. [Online]. Available: http://www. aclweb.org/anthology/P15-1166 find our card
  11. O. Firat, K. Cho, and Y. Bengio, “Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism,” CoRR, vol. abs/1601.01073, 2016. [Online]. Available: http://arxiv.org/abs/1601. 01073 ↗ our card
  12. B. Zoph and K. Knight, “Multi-Source Neural Translation,” in The North American Chapter of the Association for Computational Linguistics (NAACL 2016), San Diego, CA, USA, June 2016. find
  13. M. Cettolo, C. Girardi, and M. Federico, “Wit 3 : Web inventory of transcribed and translated talks,” in Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), Trento, Italy, May 2012, pp. 261–268. find
  14. M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” CoRR, vol. abs/1212.5701, 2012. [Online]. Available: http://arxiv.org/abs/1212.5701 ↗ our card
  15. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL 2002). Association for Computational Linguistics, 2002, pp. 311–318. find our card
  16. R. Sennrich, B. Haddow, and A. Birch, “Improving Neural Machine Translation Models with Monolingual Data,” in Association for Computational Linguistics (ACL 2016), Berlin, Germany, August 2016. find
  17. Ç. Gülçehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On Using Monolingual Corpora in Neural Machine Translation,” CoRR, vol. abs/1503.03535, 2015. [Online]. Available: http://arxiv.org/abs/1503.03535 ↗ our card
  18. E. Cho, J. Niehues, T.-L. Ha, M. Sperber, M. Mediani, and A. Waibel, “Adaptation and Combination of NMT Systems: The KIT Translation Systems for IWSLT 2016,” in Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT 2016) -To be appeared. find