Elamur Research⌕
☰
home / papers / 2026 / KinyCOMET: Automatic Evaluation of Machine Translation Systems for Kinyarwanda–English

KinyCOMET: Automatic Evaluation of Machine Translation Systems for Kinyarwanda–English

Summary

Rwandan developers of Kinyarwanda–English MT mostly rely on BLEU and chrF, which agree poorly with human judgments. As a result, the community falls back on slow, costly manual evaluation. The authors build KinyCOMET, a learned COMET-style metric for this language pair.

They curated 450 clean source–reference pairs drawn from three corpora: the Digital Umuganda Dataset, Mbaza Tourism, and Mbaza Education. Six systems translated these pairs in both directions, and linguistics students rated the outputs with Direct Assessment scores from 1 to 100. After removing items with inconsistent ratings, the authors fine-tuned two reference-based variants. One starts from plain XLM-RoBERTa, which never saw Kinyarwanda during pretraining. The other starts from the Unbabel COMET checkpoint.

On a held-out test split, measured with Kendall τ and Spearman ρ, both variants track human scores closely. KinyCOMET Unbabel surpasses every baseline, including AfriCOMET, chrF and BLEU. The work shows that a modest DA dataset can turn a multilingual encoder into a usable evaluator for a low-resource language it was not pretrained on. Models and data are openly released.

Key points

  • Dataset size and split: the final DA dataset has 4,303 rated translations (2,225 eng2kin, 2,078 kin2eng), split 80/10/10 into 3,477 train, 404 validation and 422 test samples.
  • Translation systems: Claude 3.7-Sonnet, GPT-4o, GPT-4.1, Gemini Flash 2.0, and NLLB at 1.3B and 600M parameters.
  • Annotation and filtering: each translation received three DA ratings, and 410 samples (about 9.5%) were dropped because their rating standard deviation exceeded 20.
  • Training and baselines: training ran on a single Tesla T4 with MSE loss, keeping the checkpoint with the best validation Pearson correlation. Baselines were Unbabel COMET, AfriCOMET STL 1.1, BLEU and chrF.
  • Results: COMET-based metrics generally outperformed the lexical ones, with one exception. Unbabel COMET, despite never being trained on Kinyarwanda, beat AfriCOMET. All metrics correlated better for eng2kin than for kin2eng.
  • Release: models (chrismazii/kinycomet_unbabel) and the dataset (chrismazii/kinycomet_dataset) are on Hugging Face.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3nawgmkiq3mu

@inproceedings{mazimpaka2026kinycomet,
  title = {{KinyCOMET: Automatic Evaluation of Machine Translation Systems for Kinyarwanda–English}},
  author = {Mazimpaka, Prince Chris and Nehring, Jan and Rutunda, Samuel and España-Bonet, Cristina},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3nawgmkiq3mu},
  url = {https://doi.org/10.63317/3nawgmkiq3mu}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. ME-TEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. find
  2. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. E.L. Epstein and R. Kole. 1998. The Language of African Literature. Africa World Press. find
  6. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  7. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. Association for Computational Linguistics. find our card
  8. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191, Denver, Colorado. Association for Computational Linguistics. find our card
  9. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  10. Michael Hanna and Ondřej Bojar. 2021. A fine-grained analysis of BERTScore. In Proceedings of the Sixth Conference on Machine Translation, pages 507–517, Online. Association for Computational Linguistics. ↗ our card
  11. Bernd Heine and Derek Nurse. 2000. African Languages: An Introduction. Cambridge University Press, Cambridge. find
  12. Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. Lost in the source language: How large language models evaluate the quality of machine translation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3546– 3562, Bangkok, Thailand. Association for Computational Linguistics. find
  13. M. G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. ↗
  14. Senyu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? ↗ our card
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. find
  16. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Belgium, Brussels. Association for Computational Linguistics. find our card
  17. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. find our card
  18. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find our card
  19. NLLB Team. 2024. Scaling neural machine translation to 200 languages. Nature, 630:pages841–846. ↗ our card
  20. Patrick Nyongabo. 2017. Kinyarwanda–English Translator. A Statistical Machine Learning Approach. Bachelor’s thesis, Hendrix University, 1600 Washington Ave, Conway, AR 72032, United States. find
  21. Antoine Nzeyimana. 2024. Low-resource neural machine translation with morphological modeling. ArXiv:2404.02392 [cs]. ↗
  22. Chijioke I Okorie and Melissa Omino. 2025. Addressing inequitable openness in licences for sharing african data and datasets through the nwulite obodo open data licence. Law, Technology and Humans. ↗
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  24. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find
  25. Jenalea Rajab, Anuoluwapo Aremu, Everlyn Asiko Chimoto, Dale Dunbar, Graham Morrissey, Fadel Thior, Luandrie Potgieter, Jessica Ojo, Atnafu Lambebo Tonja, Wilhelmina Ndape-waOnyothi Nekoto, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, and Benjamin Rosman. 2025. The esethu framework: Reimagining sustainable dataset governance and curation for low-resource languages. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30763–30776, Vienna, Austria. Association for Computational Linguistics. ↗
  26. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  27. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  28. C. Spearman. 1904. The Proof and Measurement of Association Between two Things. American Journal of Psychology, 15:88–103. ↗
  29. Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Hassan Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Abdi Mohamed, Hassan Ayinde, Oluwabusayo Olufunke Awoyomi, Lama Alkhaled, Sana Al-azzawi, Naome A. Etori, Millicent Ochieng, Clemencia Siro, Njoroge Kiragu, Eric Muchiri, Wangari Kimotho, Lyse Naomi Wamba Momo, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Nasir Iro, Saheed S. Abdullahi, Stephen E. Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Raphael Ogbu, Sam Ochieng’, Verrah Akinyi Otiende, Chinedu Emmanuel Mbonu, Sakayo Toadoum Sari, Yao Lu, and Pontus Stenetorp. 2024. AfriMTE and AfriCOMET: Enhancing COMET to embrace under-resourced African languages. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5997–6023, Mexico City, Mexico. Association for Computational Linguistics. ↗
  30. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. 9. Language Resource References ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mazimpaka-2026-kinycomet-automatic-evaluation-machine-translation/