Elamur Research⌕
☰
home / papers / 2025 / SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?

SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?

Summary

Automatic MT evaluation works poorly for most African languages. The earlier AfriCOMET line of metrics had no public training data, small test sets, and only one MT system per language pair, and it was unstable on very low-resource languages such as Twi and Luo. The authors built SSA-MTE, a human-annotated evaluation corpus for 14 Sub-Saharan African language pairs. It draws on 1,500 English–French news sentences from Global Voices, with Portuguese used as the source for two Mozambican languages. Bilingual native speakers marked error spans with a simplified MQM scheme and gave each output a 0–100 direct-assessment score. The outputs came from six MT systems. On this data they trained SSA-COMET, which uses reference translations, and SSA-COMET-QE, which does not. They compared both with few-shot prompting of GPT-4o, Claude 3.7 and Gemini 2.5 Pro. SSA-COMET clearly outperforms AfriCOMET and roughly matches Gemini 2.5 Pro, while costing far less to run. This gives under-resourced African languages a practical, openly licensed evaluation tool.

Key points

  • SSA-MTE dataset: more than 73,000 sentence-level human judgments across 14 language pairs.
    • Covered languages include eng→amh/hau/kik/kin/luo/twi/yor/zul, fra→ewe/lin/wol, and por→vmw/nya.
    • About 6,600 items were annotated per pair, with 300 overlapping items used to check agreement between annotators.
  • MT systems: GPT-4o, Gemini-1.5, Claude-3.5, Google Translate, NLLB-200-distilled-600M and M2M-100-418M. M2M-100 was fine-tuned for languages it did not support.
  • Main result: SSA-COMET in its multi-task (MTL) variant has the highest average Spearman correlation of all regression-based metrics.
    • It beats GPT-4o and Claude-3.7 prompting.
    • It is competitive with Gemini 2.5 Pro, which used five demonstrations.
    • The largest gains are on Twi, Luo and Yorùbá.
  • Encoder and data ablations: AfroXLMR-114L generally outperforms AfroXLMR-76L, especially on por-vmw. Adding silver-filtered data from languages excluded for low annotator agreement raised the average Spearman score slightly (0.639 → 0.642).
  • Release: data, models (McGill-NLP/ssa-comet-*) and code are available under CC BY 4.0. The evaluation covers adequacy only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.656

@inproceedings{li2025ssa,
  title = {{SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?}},
  author = {Li, Senyu and Wang, Jiayi and Ali, Felermino D. M. A. and Cherry, Colin and Deutsch, Daniel and Briakou, Eleftheria and Sousa-Silva, Rui and Cardoso, Henrique Lopes and Stenetorp, Pontus and Adelani, David Ifeoluwa},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.656},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.656}
}

References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics. ↗ our card
  2. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 8 others. 2025. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757, Albuquerque, New Mexico. Association for Computational Linguistics. find
  3. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗ our card
  4. Felermino D. M. A. Ali, Henrique Lopes Cardoso, and Rui Sousa-Silva. 2024a. Building resources for emakhuwa: Machine translation and news classification benchmarks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14842–14857, Miami, Florida, USA. Association for Computational Linguistics. find
  5. Felermino Dario Mario Ali, Henrique Lopes Cardoso, and Rui Sousa-Silva. 2024b. Expanding FLO-RES+ benchmark for more low-resource settings: Portuguese-emakhuwa machine translation evaluation. In Proceedings of the Ninth Conference on Machine Translation, pages 579–592, Miami, Florida, USA. Association for Computational Linguistics. find
  6. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. ↗ our card
  7. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  9. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, and 1 others. 2021. Beyond english-centric multilingual machine translation. The Journal of Machine Learning Research, 22(1):4839–4886. find
  10. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics. find our card
  11. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. ↗
  12. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767, Singapore. Association for Computational Linguistics. find our card
  13. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. ↗
  14. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. In Tradumàtica. ↗
  15. NLLB-Team, Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. ArXiv, abs/2207.04672. ↗
  16. Chester Palen-Michel, June Kim, and Constantine Lignos. 2022. Multilingual open text release 1: Public domain news in 44 languages. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2080–2089, Marseille, France. European Language Resources Association. find
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  18. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  19. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  20. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗ our card
  21. Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, and Raj Dabre. 2023. IndicMT eval: A dataset to meta-evaluate machine translation metrics for Indian languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14210–14228, Toronto, Canada. Association for Computational Linguistics. ↗
  22. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek F Wong, and Lidia S Chao. 2022. Unite: Unified translation evaluation. arXiv preprint arXiv:2204.13346. ↗ our card
  23. Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Ola-toye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, and 39 others. 2024a. AfriMTE and AfriCOMET: Enhancing COMET to embrace under-resourced African languages. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5997–6023, Mexico City, Mexico. Association for Computational Linguistics. find
  24. Jiayi Wang, David Ifeoluwa Adelani, and Pontus Stenetorp. 2024b. find
  25. Evaluating WMT 2024 metrics shared task submissions on AfriMTE (the African challenge set). In Proceedings of the Ninth Conference on Machine Translation, pages 505–516, Miami, Florida, USA. Association for Computational Linguistics. find
  26. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗
  27. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. *Figure 4: Translation performance of MT systems used for por-nya.* the number of annotated errors increases, the adequacy score consistently decreases. These findings validate the reliability of error span annotations as strong indicators of perceived translation quality. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-ssa-comet-do-llms-outperform/