Elamur Research⌕
☰
home / papers / 2026 / Evaluating Native-Speaker Preferences on Machine Translation and Post-Edits for Five African Languages

Evaluating Native-Speaker Preferences on Machine Translation and Post-Edits for Five African Languages

Summary

Wikipedia's ContentTranslation tool lets editors start from machine translation and then revise it. MT metrics such as COMET-style scores usually assume these human post-edits (HE) are better than raw MT. The authors test that assumption for Hausa, Igbo, Swahili, Yoruba and Zulu. They first noticed that AfriCOMET quality estimation found almost no average change between MT and HE segments in the Wikipedia dump.

From that dump they filtered candidate English–MT–HE triplets and had three native speakers per language, recruited through Masakhane, judge them blind in a customized Appraise interface. Annotators picked the better version and marked spans with Fluency, Adequacy or Explicitation reasons. The result is a released corpus of 4,335 annotated pairs.

Annotators favored the MT output for most segments in Igbo, Swahili and Zulu. Their choices depended mainly on fluency. Editors frequently left out source content or inserted material that was not in the source.

The work questions whether post-edits should serve as the gold standard. It also points to concrete changes for Wikipedia workflows and for building metrics that match human preferences.

Key points

  • Source data: parallel segments come from the 06/13/2025 Wikipedia ContentTranslation dump. Average AfriCOMET differences between HE and MT range from −0.01 to 0.02.
  • Curation filters: about 1,000 triplets per language were selected using four criteria:
    • non-zero AfriCOMET QE difference within ±0.5;
    • both QE scores above 0.5;
    • Levenshtein distance from 5 to 40;
    • English source of at least 20 words.
  • Inter-annotator agreement (Fleiss' κ): Hausa 0.83, Igbo 0.72, Zulu 0.70, Yoruba 0.57 and Swahili 0.37.
  • How often MT was preferred: Zulu 86%, Igbo 62%, Yoruba 27% and Hausa roughly 4%. Fluency labels such as Fluency_Natural were the most common reasons given.
  • Recommendations:
    • let editors tag their intent, for example correction versus cultural adaptation;
    • adopt language-specific quality strategies;
    • add lightweight automatic fluency checks on post-edited text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.africanlp-main.15

@inproceedings{oirghi2026evaluating,
  title = {{Evaluating Native-Speaker Preferences on Machine Translation and Post-Edits for Five African Languages}},
  author = {Oirghi, Hiba El and Gwadabe, Tajuddeen and Carpuat, Marine},
  booktitle = {Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.africanlp-main.15},
  url = {https://doi.org/10.18653/v1/2026.africanlp-main.15}
}

References

9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christian Federmann. 2018. Appraise evaluation framework for machine translation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 86–88, Santa Fe, New Mexico. Association for Computational Linguistics. find
  2. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382. ↗
  3. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗ our card
  4. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024. Error span annotation: A balanced approach for human evaluation of machine translation. In Proceedings of the Ninth Conference on Machine Translation, pages 1440–1453, Miami, Florida, USA. Association for Computational Linguistics. ↗ our card
  5. Vladimir I. Levenshtein. 1965. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady, 10:707–710. find
  6. Arle Lommel, Aljoscha Burchardt, Maja Popović, Kim Harris, Eleftherios Avramidis, and Hans Uszkoreit. 2014. Using a new analytic measure for the annotation and analysis of MT errors on real data. In Proceedings of the 17th Annual Conference of the European Association for Machine Translation, pages 165–172, Dubrovnik, Croatia. European Association for Machine Translation. find
  7. Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, Musie Meressa, Espoir Murhabazi, Orevaoghene Ahia, Elan van Biljon, Arshath Ramkilowan, Adewale Akinfaderin, Alp Öktem, Wole Akin, Ghollah Kioko, Kevin Degila, Herman Kamper, Bonaventure Dossou, Chris Emezue, Kelechi Ogueji, and Abdallah Bashir. 2020. Masakhane – machine translation for africa. Preprint, arXiv:2003.11529. ↗
  8. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  9. Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Hassan Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Abdi Mohamed, Hassan Ayinde, Oluwabusayo Olufunke Awoyomi, Lama Alkhaled, Sana Al-azzawi, Naome A. Etori, Millicent Ochieng, Clemencia Siro, Njoroge Kiragu, Eric Muchiri, Wangari Kimotho, Lyse Naomi Wamba Momo, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Nasir Iro, Saheed S. Abdullahi, Stephen E. Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Raphael Ogbu, Sam Ochieng’, Verrah Akinyi Otiende, Chinedu Emmanuel Mbonu, Sakayo Toadoum Sari, Yao Lu, and Pontus Stenetorp. 2024. AfriMTE and AfriCOMET: Enhancing COMET to embrace under-resourced African languages. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5997–6023, Mexico City, Mexico. Association for Computational Linguistics. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/oirghi-2026-evaluating-native-speaker-preferences-machine/