Elamur Research⌕
☰
home / papers / 2026 / SALT-31: A Machine Translation Benchmark Dataset for 31 Ugandan Languages

SALT-31: A Machine Translation Benchmark Dataset for 31 Ugandan Languages

Summary

Most MT benchmarks for African languages, such as FLORES, test isolated sentences. This hides discourse-level problems and leaves out the kinds of communication that matter locally. SALT-31, from Sunbird AI and Makerere University, is an evaluation set for 31 Ugandan languages built from connected text. Several LLMs drafted English five-sentence mini-sequences for 20 everyday Ugandan scenarios, including banking, clinic visits, markets, farming advice and government speeches. The authors manually chose one sequence per scenario. Native speakers translated the resulting 100 sentences, and linguists at Makerere checked every translation independently, settling disagreements by consensus. The authors benchmarked several systems with chrF. The regionally focused Sunflower models scored best in both directions and lost far less quality on Nilotic and Central Sudanic languages than general-purpose models like GPT-4o. Native speakers reviewing the outputs also found coreference, register and cultural errors that spanned whole sequences. For some of these languages, this is the first public parallel text outside Bible translations. Its small size, however, limits it to diagnostic use.

Key points

  • Scale: 20 scenarios × 5 sentences = 100 English sentences. Translations cover 31 languages: 17 Bantu, 11 Nilotic and 3 Central Sudanic. The dataset is released as Sunbird/salt-31 on Hugging Face.
  • Source text generation: Seven LLMs produced the English seed text: Claude Sonnet, Gemini 2 Flash, Mistral Large, LLaMA 3.3 70B, DeepSeek R1, GPT-4o and GPT-4.5. Humans selected the final sequences, and all translations were done by native speakers.
  • Average chrF across all languages:
    • Sunflower-32B led xx→en with 0.435.
    • Sunflower-14B led en→xx with 0.366.
    • Gemini 2.5 Pro scored 0.408 / 0.301 and GPT-4o scored 0.354 / 0.235 (xx→en / en→xx).
  • Gap across language families (en→xx):
    • GPT-4o dropped from 0.323 on Bantu to 0.139 on Nilotic and 0.094 on Central Sudanic.
    • Sunflower-14B held up better, going from 0.413 to 0.312 to 0.299.
  • Discourse-level failures found by native speakers:
    • inconsistent references to the patient in the medical dialogue;
    • mixed formal and colloquial register in government speeches;
    • urban-sounding terminology in agricultural extension text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.africanlp-main.21

@inproceedings{nsumba2026salt,
  title = {{SALT-31: A Machine Translation Benchmark Dataset for 31 Ugandan Languages}},
  author = {Nsumba, Solomon and Akera, Benjamin and Ouma, Evelyn Nafula and Ssentanda, Medadi E. and Kawalya, Deo and Bainomugisha, Engineer and Mwebaze, Ernest Tonny and Quinn, John},
  booktitle = {Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.africanlp-main.21},
  url = {https://doi.org/10.18653/v1/2026.africanlp-main.21}
}

References

5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Benjamin Akera, Jonathan Mukiibi, Lydia Sanyu Naggayi, Claire Babirye, Isaac Owomugisha, Solomon Nsumba, Joyce Nakatumba-Nabende, Engineer Bainomugisha, Ernest Mwebaze, and John Quinn. 2022. Machine translation for african languages: Community creation of datasets and models in uganda. In 3rd Workshop on African Natural Language Processing. find
  2. Benjamin Akera, Evelyn Nafula Ouma, Gilbert Yiga, Patrick Walukagga, Phionah Natukunda, Trevor Saaka, Solomon Nsumba, Lilian Teddy Nabukeera, Joel Muhanguzi, Imran Sekalala, and 1 others. 2025. Sunflower: A new approach to expanding coverage Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395. find our card
  3. SunbirdAI. 2023. Salt: Sunbird african language technology. https://github.com/SunbirdAI/salt. ↗
  4. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. arXiv preprint arXiv:1805.10163. ↗ our card
  5. Nekoto Wilhelmina, Marivate Vukosi, Matsila Tshinondiwa, Fasubaa Timi, Fagbohungbe Taiwo, Akinola Solomon Oluwole, Muhammad Shamsuddeen, Kabenamualu Salomon Kabongo, Osei Salomey, Sackey Freshia, and 1 others. 2020. Participatory research for low-resourced machine translation: A case study in african languages. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2144–2160. find our card

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nsumba-2026-salt-31-machine-translation-benchmark/