Elamur Research
☰
home / papers / 2018 / Rapid Adaptation of Neural Machine Translation to New Languages

Rapid Adaptation of Neural Machine Translation to New Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1103.pdf

Summary

Neubig and Hu study how to build neural MT from a newly needed low-resource source language (LRL) into English that is both accurate and quick to train. The motivation is disaster response, where waiting days for a system is not acceptable. Their approach combines multilingual training with cross-lingual transfer: a seed model is trained in advance on many languages and then fine-tuned on the LRL. They compare single-source, bi-source and all-source seed models. They also contrast warm-start, where the LRL is included in seed training, with cold-start, where LRL data arrives only later. Their new technique, similar-language regularization (SLR), adds data from a closely related high-resource language during fine-tuning to limit over-fitting. This is done either by simple corpus concatenation or by balanced mini-batch sampling. Experiments use the 58-language TED corpus and an attentional LSTM model built in xnmt. A universal model already gives usable output with no LRL data at all, and fine-tuning it with SLR works best in both start settings. Pre-trained massively multilingual models are therefore a practical basis for rapid emergency MT.

Key points

  • Language pairs (LRL/helper): Azerbaijani/Turkish (aze/tur), Belarusian/Russian (bel/rus), Galician/Portuguese (glg/por) and Slovak/Czech (slk/ces).
    • LRL training sets range from about 4.5k sentences (bel) to 61.5k (slk).
    • Helper sets range from about 103k to 208k sentences.
  • Multilingual seed models: Bi-source or all-source training beats single-source training by 7–13 BLEU. All multilingual variants clearly outperform the Moses phrase-based baseline.
  • Cold-start results: With zero LRL training data, the all-languages-but-LRL model (All−) reaches up to 15.5 BLEU on glg-eng and beats the Bi− model. Unsupervised NMT (undreamt) scores near 0 BLEU.
  • SLR gains: Similar-language regularization improves on fine-tuning with only LRL data by 1.7 BLEU on average across the 4 settings. More regularization data trades training speed for accuracy.
  • Setup:
    • Subword vocabularies are learned separately per language with sentencepiece (8,000 units each) and then merged.
    • This allows LRL subwords to be built only when that language's data arrives.
    • Code is released at github.com/neubig/rapid-adaptation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1103

@inproceedings{neubig2018rapid,
  title = {{Rapid Adaptation of Neural Machine Translation to New Languages}},
  author = {Neubig, Graham and Hu, Junjie},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1103},
  url = {https://doi.org/10.18653/v1/d18-1103}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Graham Neubig, Yuichiroh Matsubayashi, Masato Hagiwara, and Koji Murakami. 2011. Safety information mining - what can NLP do in a disaster -. In Proc. IJCNLP, pages 965–973. find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041. find our card
  3. Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, Sarguna Padmanabhan, Ye Qi, Devendra Singh Sachan, Philip Arthur, Pierre Godard, John Hewitt, Rachid Riad, and Liming Wang. 2018. XNMT: The extensible neural machine translation toolkit. In Proc. AMTA, Boston. find
  4. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. pages 296–301. find our card
  5. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proc. NAACL, New Orleans, USA. find our card
  6. Tanja Schultz and Alan W Black. 2006. Challenges with rapid adaptation of speech translation systems to new language pairs. In Proc. ICASSP, volume 5. IEEE. find
  7. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL, pages 1715–1725. find
  8. Kate Starbird, Grace Muzny, and Leysia Palen. 2012. Learning from the crowd: Collaborative filtering techniques for identifying on-the-ground Twitterers during mass disruptions. In Proc. ISCRAM. find
  9. Jörg Tiedemann. 2018. Emerging language spaces learned from massively multilingual corpora. arXiv preprint arXiv:1802.00273. — [arXiv:1802.00273](https://arxiv.org/abs/1802.00273) find
  10. Sarah Vieweg, Amanda L Hughes, Kate Starbird, and Leysia Palen. 2010. Microblogging during two natural hazards events: what Twitter may contribute to situational awareness. In Proc. CHI, pages 1079– 1088. find
  11. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proc. EMNLP, pages 1568–1575. find