Elamur Research⌕
☰
home / papers / 2016 / Zero-Resource Translation with Multi-Lingual Neural Machine Translation

Zero-Resource Translation with Multi-Lingual Neural Machine Translation

Summary

The paper asks whether a multi-way, multilingual NMT model (Firat et al., 2016) can translate between two languages that have no direct parallel data, such as Spanish→French when only En-Es and En-Fr corpora exist. The authors first add many-to-one decoding strategies to the model. These merge several source-language "views" of a sentence, either by averaging attention context vectors (early), by averaging output distributions (late), or both. They then demonstrate that the unmodified model fails at translating between language pairs it has no direct data for. To fix this, they finetune only a duplicated copy of the shared attention parameters on a pseudo-parallel corpus built with the model's own translations. Experiments on large English, Spanish and French corpora found that this approach matches or beats a single-pair model trained on 1M real Es-Fr sentence pairs. It also outperforms pivot translation through English. This matters because a usable direct system can be built from bilingual data that pass through a hub language, with only a small parameter overhead per new pair.

Key points

  • Training data: 34.71m En-Es pairs and 65.77m En-Fr pairs, drawn from UN, Europarl-v7, news-commentary-v7, LDC, internal technical-domain data and, for En-Fr, ReutersUN and Gigaword. A three-way En-Es-Fr subset contains 11.32m sentences.
  • Evaluation and preprocessing:
    • newstest-2012 is the dev set and newstest-2013 is the test set.
    • Text is tokenized with Moses, and numbers, dates and URLs are replaced by placeholders.
    • BPE vocabularies are 37k (En), 43k (Es) and 45k (Fr) tokens.
    • Training keeps only sentences of at most 50 symbols.
  • Zero-resource Es→Fr results (test scores, Table 5):
    • Pivot through English: 20.4.
    • Single-pair model trained on 1M true pairs: 20.35.
    • Early-average finetuning with 1M pseudo pairs: 21.46.
    • The same setup with 1M true pairs: 21.7.
  • Effect of pseudo vs. true pairs: With few examples, pseudo pairs worked better than true ones. Early averaging scored 19.43 with 1k pseudo pairs versus 9.77 with 1k true pairs. Real pairs only paid off at around 1M or more.
  • Early+late averaging: Combining both strategies helped in multi-source translation but brought no gain over early averaging alone in the zero-resource setting.

Interesting ideas

  • I1. Finetune only a copy of the attention module. Freezing encoders and decoders while duplicating and retraining only the attention parameters adapts a model to a new language pair cheaply. This pattern could suit any modular multilingual system where adding a full model per pair is too costly. The authors note one drawback: the number of parameters still grows linearly with the number of target pairs.
  • I2. Use the model's own outputs to realign components. Pseudo-parallel data generated by the model repaired the mismatch between an encoder and a decoder that were never trained together. It even beat small amounts of real data, which the authors attribute to a better fit with the model's inductive bias and to a possible regularizing effect of noise. This is a useful trick whenever gold data for a direction is tiny.
  • I3. Ensemble-like decoding from bilingual-only training. Early averaging of context vectors and late averaging of output probabilities make multi-source translation possible without multi-way parallel corpora. Late averaging even works across separately trained models. This makes it easy to combine sources or systems at inference time.
  • I4. Create a pseudo source on the fly. At test time, the system can first generate a hub-language version of the input (e.g., English from Spanish) and then decode from both views at once. This turns a single-source problem into a multi-source one and could help low-resource pairs connected through a high-resource language like English.
  • I5. Benchmark against real data budgets. Comparing the zero-resource system with single-pair models trained on 1k–1M true pairs shows how much real data the method is "worth." This evaluation design transfers well to any low-resource method.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

89 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1026

@inproceedings{firat2016zero,
  title = {{Zero-Resource Translation with Multi-Lingual Neural Machine Translation}},
  author = {Firat, Orhan and Sankaran, Baskaran and Al-onaizan, Yaser and Vural, Fatos T. Yarman and Cho, Kyunghyun},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1026},
  url = {https://doi.org/10.18653/v1/d16-1026}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015. Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈c Barrault, and Joost van de Weijer. 2016. Does multim… find our card
  2. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In ACL. find our card
  3. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. find our card
  4. ACL. Orhan Firat and Kyunghyun Cho. 2016. DL4MT-Tutorial: Conditional gated recurrent unit with attention mechanism. find
  5. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL. Mikel L Forcada and Ramón P Ñeco. 1997. Recursive hetero-associative memories for translation. In Biological and Artificial Computation: From Neurosc… find our card
  6. Springer. Nizar Habash and Jun Hu. 2009. Improving arabic-chinese statistical machine translation using english as pivot language. In Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT ’09, pages 173–181. Association for Computational Linguistics. Sébastien Jean, Orhan Fi… find
  7. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709. find our card
  8. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. The International Conference on Learning Representations (ICLR). find
  9. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… find
  10. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015. Character-based neural machine translation. arXiv:1511.04586. find our card
  11. Minh-Thang Luong and Christopher D Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. arXiv:1604.00788. find our card
  12. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114. find our card
  13. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model… find
  14. Language Transfer. Cambridge University Press. Cambridge Books Online. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. Håkan Ringbom. 2007. Cross-linguistic similarity in foreign language learning, volu… find
  15. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. find
  16. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  18. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. find
  19. Masao Utiyama and Hitoshi Isahara. 2007. A comparison of pivot methods for phrase-based statistical machine translation. In HLT-NAACL, pages 484–491. find our card
  20. Hua Wu and Haifeng Wang. 2007. Pivot language approach for phrase-based statistical machine translation. Machine Translation, 21(3):165–181. find our card
  21. Bing Zhao and Shengyuan Chen. 2009. A simplex armijo downhill algorithm for optimizing statistical machine translation decoding parameters. In HLT-NAACL, pages 21–24. find
  22. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In NAACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/