Elamur Research⌕
☰
home / papers / 2023 / Machine Translation for Ge'ez Language

Machine Translation for Ge'ez Language

Summary

The paper looks at how to build machine translation for Ge'ez. Ge'ez is the classical Semitic language of Ethiopia and Eritrea. No community speaks it natively today, though it survives in church liturgy. Parallel data for it is scarce and almost entirely biblical. The author trains Transformer models with OpenNMT. Bilingual systems for English↔Ge'ez and Amharic↔Ge'ez serve as baselines. They are compared with one multilingual model covering Ge'ez, English, Amharic and Tigrinya. These partner languages were chosen because they share geography, morphology and, notably, the Ge'ez script. The multilingual model uses a shared BPE vocabulary and a target-language token, and it beats the bilingual baselines by roughly 4 BLEU on average. Two other routes do worse. Fine-tuning a large NLLB-200 model on a few thousand Ge'ez pairs gives poor results. Prompting GPT-3.5 with retrieved similar examples translates surprisingly well for a model with no prior Ge'ez, but still trails the multilingual baseline. The work offers a first benchmark for Ge'ez and practical guidance for very low-resource, historical languages.

Key points

  • Data sources: Opus supplied Amharic and Tigrinya texts aligned with English (Bible, Tanzil, TED and others). The Ge'ez material is a Bible corpus from the AAU Ethiopian Languages corpus, because Opus has no Ge'ez.
  • Multilingual model setup:
    • OpenNMT Transformer with 6 encoder-decoder layers and 1024 hidden units.
    • Trained for 300k steps.
    • Each source sentence starts with a target-language tag (Johnson et al. style).
    • Source and target share one BPE vocabulary.
  • Multilingual vs. bilingual: the multilingual model gains about 4 BLEU on average over the bilingual ones. In one example it correctly renders "right" and "left", which plausibly comes from Tigrinya, where those words match Ge'ez. The bilingual model instead outputs "north and south".
  • NLLB-200 fine-tuning: the 1.3B variant, fine-tuned on about 4k Ge'ez pairs, reaches only 0.2 BLEU for en→gez and 3.8 BLEU for gez→en.
  • GPT-3.5 few-shot:
    • Model: text-davinci-003, temperature 0.3.
    • Each prompt holds up to 10 fuzzy-match examples, retrieved by embedding similarity.
    • On 50 random test pairs it scores 9.2 BLEU, against 15.2 for the MNMT baseline.

Interesting ideas

  • I1. Picking auxiliary training languages by shared script as well as kinship lets surface-identical tokens transfer meaning across corpora. — For Mova (tree): when mixing treebanks for gparse, choose companions by how closely their annotation conventions match our UD dialect, not just by genre.
  • I2. A single tag token at the start of the input tells one shared model which output variety to produce. This avoids training a separate model per direction. — For Mova (tree): a conversion-target tag could let one parser emit several UD dialects, replacing a stack of post-hoc converters.
  • I3. Few-shot prompts become far more useful when the examples are nearest neighbours of the input, retrieved by embedding similarity. Fixed generic examples are much weaker. — For Mova (learn): the LLM teacher tier could get silver-annotated sentences most similar to the query as demonstrations, which should make its labels more consistent.
  • I4. A billion-parameter pretrained model fine-tuned on a tiny set lost badly to a small model trained from scratch with related-language data. — For Mova (learn): always keep a cheap, simple baseline in evaluation, and don't assume that adapting a bigger model wins when the data is a few thousand examples.
  • I5. Inspecting outputs by hand exposed a characteristic out-of-vocabulary failure: the bilingual model swapped in a semantically nearby but wrong word ("north and south" for "right and left"). — For Mova (back): when evaluating graph→text generation or translation, add a check for near-synonym substitutions that BLEU-like scores barely penalise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

4 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.14530 · arXiv 2311.14530

@misc{wassie2023machine,
  title = {{Machine Translation for Ge'ez Language}},
  author = {Wassie, Aman Kassahun},
  year = {2023},
  doi = {10.48550/arxiv.2311.14530},
  eprint = {2311.14530},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.14530}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solomon Teferra Abate, Michael Melese, Martha Yifiru Tachbelie, Million Meshesha, Solomon Atinafu, Wondwossen Mulugeta, Yaregal Assabie, Hafte Abera, Binyam Ephrem, Tewodros Abebe, Wondimagegnhue Tsegaye, Amanuel Lemma, Tsegaye Andargie, and Seifedin Shifaw. Parallel corpora for bi-lingual english-ethiopian languages statistical machine translation. Santa Fe, New Mexico, USA, 2018. Association for Computational Linguistics. Proceedings of the 27th International Conference on Computational Linguistics. find
  2. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. ↗
  3. Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. Transfer learning for low-resource neural machine translation. pp. 282–286, Austin, Texas, 2017. The National University (Phillippines). doi: http://aclweb.org/anthology/Y17-1038. In Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation. ↗
  4. Asmelash Teka Hadgu, Abel Aregawi, and Adam Beaudoin. Lesan – machine translation for low resource languages, 12 2021. find
  5. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339– 351, 2017. doi: 10.1162/tacl_a_00065. URL https://aclanthology.org/Q17-1024. ↗
  6. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. ↗
  7. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. Opennmt: Open-source toolkit for neural machine translation. 2017. find our card
  8. Taku Kudo and John Richardson. A simple and language independent subword tokenizer and detokenizer for neural text processing. 2018. find our card
  9. Surafel M. Lakew, Matteo Negri, and Marco Turchi. Low resource neural machine translation: A benchmark for five african languages, 2020. find
  10. Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. Adaptive machine translation with large language models. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 227–237, Tampere, Finland, June 2023. European Association for Machine Translation. URL https://aclanthology.org/2023.eamt-1.22. ↗ our card
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002a. find our card
  12. Kishore Papineni, Todd Ward Salim Roukos, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. pp. 311–318. Association for Computational Linguistics, 2002b. In Proceedings of the 40th annual meeting on association for computational linguistics. find our card
  13. Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186–191, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6319. URL https://aclanthology.org/W18-6319. ↗ our card
  14. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. Chat-GPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation, pp. 392–418, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.wmt-1.40. URL https://aclanthology.org/2023.wmt-1.40. ↗ our card
  15. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. ↗
  16. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. arXiv preprint arXiv:2309.16575, 2023. ↗
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017a. find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017b. In Advances in Neural Information Processing Systems. find
  19. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neural machine translation. pp. 1568–1575, Austin, Texas, 2016. Association for Computational Linguistics. doi: https://doi.org/10.18653/v1/D16-1163. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wassie-2023-machine-translation-ge-ez-language/