Elamur Research
☰
home / papers / 2017 / Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/I17-2050.pdf

Summary

The paper addresses neural machine translation (NMT) for language pairs with very little parallel data. It asks whether a parent model trained on a related and also low-resource pair can help. The authors extend the transfer learning approach introduced by Zoph et al. (2016), where a child model starts from the weights of a trained parent model. That method maps parent source embeddings to child words arbitrarily. Here, the parent and child instead share a source vocabulary, so a subword present in both languages keeps the embedding it learned in the parent.

To create this overlap, they apply BPE to the combined source and target data of both pairs, producing one joint vocabulary. Uyghur is first transliterated from Arabic to Latin script. Experiments use LORELEI data, with Uzbek-English as the parent and Turkish-English and Uyghur-English as children, and a Luong-style attentional model. Word-level transfer gave small or insignificant gains. BPE-based transfer beat a strong BPE baseline by 0.8 BLEU for Turkish and 4.3 BLEU for Uyghur. The approach looks promising for agglutinative language families with scarce data.

Key points

  • Setup: Uzbek-English parent model, with Turkish-English and Uyghur-English child models. All corpora come from the DARPA LORELEI program and were tokenized with Moses. The model is a 2-layer, 512-unit global attentional NMT (Luong et al., 2015).
  • Test BLEU for Turkish-English: BPE baseline 12.4 vs. BPE+transfer 13.2 (p < 0.05). Word-based transfer gave 8.1 → 8.5/8.6, which was not significant.
  • Test BLEU for Uyghur-English: BPE baseline 11.1 vs. BPE+transfer 15.4 (p < 0.01). Word-based results were 8.5 → 10.6 with fine-tuned target embeddings and 8.8 with frozen ones.
  • Shared BPE sharply increases overlap: the share of child training source types also found in the parent rises from 3.9% to 58.8% for Turkish and from 0.5% to 57.2% for Uyghur. The authors credit this overlap for most of the gain.
  • Training details: BPE merges were tuned between 5k and 60k operations. The training data was duplicated, with rare words replaced by UNK in one copy. Optimization used Adadelta, and decoding used beam search with length normalization (α = 0.8).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

127 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1708.09803

@inproceedings{nguyen2017transfer,
  title = {{Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation}},
  author = {Nguyen, Toan Q. and Chiang, David},
  booktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2017},
  eprint = {1708.09803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.09803}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. find our card
  2. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proc. Workshop on Syntax, Semantics and Structure in Statistical Translation. find our card
  3. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. ACL. find
  4. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proc. First Workshop on Neural Machine Translation. Association for Computational Linguistics. find
  5. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP. find
  6. Preslav Nakov and Hwee Tou Ng. 2009. Improved statistical machine translation for resource-poor languages using related resource-rich languages. In Proc. EMNLP. find
  7. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. find
  8. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27. find
  9. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. find
  10. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv:1212.5701v1. find our card
  11. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proc. EMNLP. find