Elamur Research⌕
☰
home / papers / 2020 / Tigrinya Neural Machine Translation with Transfer Learning for Humanitarian Response

Tigrinya Neural Machine Translation with Transfer Learning for Humanitarian Response

Summary

The paper builds what the authors present as the first neural Tigrinya-to-English translator. Tigrinya is an Ethiopic language with about 7.9 million speakers. No commercial or public MT model existed for it, which left Tigrinya-speaking refugees arriving in Europe facing communication barriers. Translators without Borders (TWB) trained an OpenNMT-py Transformer in three stages. First it learned from a pooled mix of Ge'ez-script parallel data, around 1.45M sentence pairs covering Amharic, Ge'ez and Tigrinya. It was then narrowed to Tigrinya only and finally adapted to a small set of humanitarian-domain sentences from TWB's translation memories. The comparison baseline skips the multilingual stage but receives the same in-domain fine-tuning. The full pipeline beats this baseline on all three metrics, gaining 1.3 BLEU, 3.1 ChrF and 0.9 Meteor. The authors release the pipeline as open source (mt-tools) and launched a public demo. The work shows that pooling data from languages sharing a script can lift a low-resource language with complex morphology.

Key points

  • Training data:
    • JW300 supplies 722K Amharic and 400K Tigrinya pairs.
    • The Ethiopian corpus adds 66K Amharic, 11K Ge'ez and 36K Tigrinya pairs.
    • Bible-uedin, Global Voices, GNOME and Tanzil contribute further Amharic.
    • TWB contributes 2.5K in-domain Tigrinya pairs.
  • Stages:
    • The mixed multilingual stage takes 73,500 steps.
    • Tigrinya-only fine-tuning uses 438K sentences and runs to 85,000 steps.
    • In-domain fine-tuning uses 2,300 sentences and runs to 85,240 steps.
    • Each stage stops when dev perplexity shows no gain over 5 validations.
  • Model: a Transformer with 6 layers, 512 units and 8 heads, trained with Adam and 4000 warm-up steps. Token batch sizes are 4096, 2048 and 10 for the three stages.
  • Preprocessing:
    • Separate BPE models are trained for Latin and Ge'ez script.
    • English is lowercased and tokenized with the Moses tokenizer.
    • Ge'ez text is processed with a punctuation-splitting script.
  • Results: tested on 200 random TWB sentences.
    • Final model: BLEU 23.6, ChrF 49.59, Meteor 27.04.
    • Baseline: 22.28, 46.51 and 26.1.
    • Intermediate stages score lower (e.g. 15.84 and 17.8 BLEU) before domain adaptation.

Interesting ideas

  • I1. Grouping related languages by shared writing system gives a cheap pretraining pool, because subword units overlap even when vocabulary differs. — For Mova (tree): gparse could pretrain on a pool of English-adjacent UD treebanks (different genres and dialects) before narrowing to the target English treebank.
  • I2. A broad → narrow → tiny-domain curriculum lets a few thousand in-domain examples steer a model shaped by over a million general ones. — For Mova (learn): silver data from witness parsers could form the broad stage, with a small hand-checked set of school-test sentences as the final adaptation stage.
  • I3. Intermediate checkpoints can score poorly on the target domain and still produce the best final model. Each stage should therefore be judged by what it yields after the final step, not in isolation. — For Mova (learn): stop judging pretraining runs on raw dev scores and always measure them after the downstream fine-tune.
  • I4. Using a baseline that differs from the full pipeline by exactly one removed stage isolates that stage's contribution cleanly. — For Mova (learn): ablate one step at a time, such as repair rules or LLM-teacher tiers, while keeping every other step identical, to explain where gains come from.
  • I5. Several metrics that agree in direction (BLEU, ChrF, Meteor) make a small gain on a small test set more credible than a single number. — For Mova (back): judge the text → graph → text compressor with several independent surface and character-level metrics, and trust only improvements on which they all agree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

9 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.11523 · arXiv 2003.11523

@misc{plitt2020tigrinya,
  title = {{Tigrinya Neural Machine Translation with Transfer Learning for Humanitarian Response}},
  author = {Plitt;, Alp Oktem; Mirko and Tang, Grace},
  year = {2020},
  doi = {10.48550/arxiv.2003.11523},
  eprint = {2003.11523},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.11523}
}

References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić and Ivan Vulić. JW300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3204–3210, Florence, Italy, July 2019. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. find our card
  3. Christos Christodouloupoulos and Mark Steedman. A massively parallel corpus: the bible in 100 languages. Language Resources and Evaluation, 49(2):375–395, Jun 2015. find our card
  4. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1723–1732, Beijing, China, july 2015. find our card
  5. Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1700–1709, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. find our card
  6. Diederik P. Kingma and Jimmy Ba. abs/1412.6980, 2014. ↗
  7. Adam: A method for stochastic optimization. Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander Rush. OpenNMT: Neural machine translation toolkit. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pp. 177–184, Boston, MA, March 2018. find
  8. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp. 177–180, Prague, Czech Republic, June 2007. find
  9. Alon Lavie and Abhaya Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT 07, pp. 228231, USA, 2007. find our card
  10. Graham Neubig and Junjie Hu. Rapid adaptation of neural machine translation to new languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 875–880, Brussels, Belgium, October-November 2018. find our card
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia, Pennsylvania, USA, July 2002. find our card
  12. Maja Popović. chrF: Character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392–395, Lisbon, Portugal, September 2015. http://www.masakhane.io/ ↗ our card
  13. CoRR, Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016. find
  14. Yemane Tedla and Kazuhide Yamamoto. The effect of shallow segmentation on english-tigrinya statistical machine translation. 2016 International Conference on Asian Language Processing (IALP), pp. 79–82, 2016. find
  15. Solomon Teferra Abate, Michael Melese, Martha Yifiru Tachbelie, Million Meshesha, Solomon Atinafu, Wondwossen Mulugeta, Yaregal Assabie, Hafte Abera, Binyam Ephrem, Tewodros Abebe, Wondimagegnhue Tsegaye, Amanuel Lemma, Tsegaye Andargie, and Seifedin Shifaw. Parallel corpora for bi-directional statistical machine translation for seven Ethiopian language pairs. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pp. 83–90, Santa Fe, New Mexico, USA, August 2018. find
  16. Jrg Tiedemann. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, May 2012. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998–6008, 2017. find
  18. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1568–1575, Austin, Texas, November 2016. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oktem-2020-tigrinya-neural-machine-translation-transfer/