Elamur Research⌕
☰
home / papers / 2020 / Evaluating Amharic Machine Translation

Evaluating Amharic Machine Translation

Summary

Amharic, the official language of Ethiopia, has no trustworthy shared benchmark for machine translation. In-domain test sets, such as religious text, can hide train–test overlap or domain mismatch. The authors build an evaluation-only benchmark for Amharic↔English that covers four use-case domains: news headlines, Wikipedia sentences, tweets and conversational phrases. The Amharic-side sources reflect local content: Ethiopian news outlets, Amharic Wikipedia, and tweets found by searching for Amharic stop-words. The English-side sources reflect global topics: the Wikipedia current events portal for 2013–2020, and tweets retrieved using entities and dates from those headlines. Translators were recruited through Telegram, screened in stages, and their output was post-edited. The authors then scored Google Translate and Yandex Translate with SacreBLEU, with tokenization switched off. Google beats Yandex in every domain. Both systems are much weaker from English into Amharic: Google's combined BLEU is 9.6 in that direction, against 23.2 the other way. The dataset is released on Zenodo, giving a reproducible multi-domain benchmark that shows how much room remains for improving low-resource MT.

Key points

  • The dataset has 997 Amharic-source bitext pairs (383 news, 257 Wikipedia, 203 Twitter, 154 conversational) and 1915 English-source pairs (371 / 499 / 641 / 404). It is hosted at Zenodo record 3734260.
  • Annotation pipeline:
    • 198 people answered a Telegram job post, and 124 received a 10-sentence trial.
    • Trials were graded on a five-level scale from bad to perfect, and anything below "good" was discarded.
    • 28 translators then got 50-line batches, which were post-edited.
  • Google Translate (neural MT) and Yandex Translate (hybrid neural + statistical) were queried through their APIs. The Am→En runs were on 14 Feb 2020 and the En→Am runs on 30 Mar 2020.
  • Combined BLEU:
    • Google scored 23.2 for Am→En and 9.6 for En→Am.
    • Yandex scored 4.8 and 1.3.
    • Google's best Am→En domains were news (30.8) and conversational (30.5).
    • Google's En→Am conversational score was only 4.8.
  • SacreBLEU's default 13a tokenizer splits URLs in tweets into many pieces and inflates the score, so tokenization "none" is the headline setting. The 13a differences are given separately for the non-Twitter domains.

Interesting ideas

  • I1. Each translation direction uses source text native to that language: local Ethiopian content for Amharic, global events for English. This tests systems on realistic input rather than on translationese. — For Mova (back): test sets for translation through the meta-world should start from text originally written in the source language, so the text → graph → text path is judged on natural input.
  • I2. Tweets were matched to news headlines by querying with the headline's named entities plus its date. This yields informal texts about the same event as a formal one. — For Mova (world): the same trick could collect headline/tweet pairs about one event, to check whether the world builder extracts the same entities and event order from both registers.
  • I3. A metric detail (how URLs are tokenized) can quietly inflate scores, so the authors fix the setting and publish the full SacreBLEU signature. — For Mova (learn): every evaluation run should log its exact metric configuration and check that noise tokens like URLs or @mentions are not dominating the score.
  • I4. Annotators had to pass a short graded trial with a hard quality cutoff before receiving larger batches, and their work was still post-edited. This is a cheap way to filter a noisy labor pool. — For Mova (learn): an LLM teacher tier or a new witness parser could be admitted to silver-data production only after passing a small graded trial against gold trees.
  • I5. Reporting scores per domain and per direction exposed a large weakness that a pooled number would blur. Conversational En→Am, for example, is far below news. — For Mova (answer): reader results should be broken down by genre (fables, school tests, maths word problems) rather than averaged, so weak spots stay visible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.14386 · arXiv 2003.14386

@misc{hadgu2020evaluating,
  title = {{Evaluating Amharic Machine Translation}},
  author = {Hadgu, Asmelash Teka and Beaudoin, Adam and Aregawi, Abel},
  year = {2020},
  doi = {10.48550/arxiv.2003.14386},
  eprint = {2003.14386},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.14386}
}

References

9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solomon Teferra Abate, Michael Melese, Martha Yifiru Tachbelie, Million Meshesha, Solomon Atinafu, Wondwossen Mulugeta, Yaregal Assabie, Hafte Abera, Binyam Ephrem Seyoum, Tewodros Abebe, et al. Parallel corpora for bi-directional statistical machine translation for seven ethiopian language pairs. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pp. 83–90, 2018. find
  2. Alexandra Balahur and Marco Turchi. Multilingual sentiment analysis using machine translation? In Proceedings of the 3rd workshop in computational approaches to subjectivity and sentiment analysis, pp. 52–60. Association for Computational Linguistics, 2012. find
  3. Erik Brynjolfsson, Xiang Hui, and Meng Liu. Does machine translation affect international trade? evidence from a large digital platform. Management Science, 65(12):5449–5460, 2019. find
  4. Jyoti Guha and Carmen Heger. Machine translation for global e-commerce on ebay. In Proceedings of the AMTA, volume 2, pp. 31–37, 2014. find
  5. Asmelash Teka Hadgu and Robert Jschke. asmelashteka/twitteradvancedsearch: First release, March 2019. URL https://doi.org/10.5281/zenodo.2605413. ↗
  6. Asmelash Teka Hadgu, Adam Beaudoin, and Abel Aregawi. Machine translation evaluation dataset for amharic, February 2020. URL https://doi.org/10.5281/zenodo.3734260. ↗
  7. Philipp Koehn and Barry Haddow. Interactive assistance to human translators using statistical machine translation methods. MT Summit XII, 2009. find
  8. Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186–191, Belgium, Brussels, October 2018. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ W18-6319. ↗ our card
  9. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144, 2016. URL http://arxiv.org/abs/1609.08144. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hadgu-2020-evaluating-amharic-machine-translation/