Elamur Research⌕
☰
home / papers / 2026 / Yeswa-Stories: A Three-Way Parallel Dataset of Female African Figures in Low-Web Data Languages

Yeswa-Stories: A Three-Way Parallel Dataset of Female African Figures in Low-Web Data Languages

Summary

Most benchmarks for gender bias in machine translation cover high-resource languages, use occupation-centred templates, or are translated from English without cultural adaptation. The authors address this gap for three Ethiopian low-web data languages with Yeswa-Stories, a three-way parallel resource of 1,300 sentences in Amharic, Afaan Oromo and Tigrinya, all centred on women. Local texts about women were scarce or not digitized, so the team began with English Wikipedia biographies of notable African women and had human translators render them. They then added Ethiopian news and cultural material about festivals, traditional roles and daily life, translating it into the other two languages.

The authors tested NLLB and Google Translate in all six directions. Both systems scored low on BLEU and chrF. Google Translate outperformed NLLB in every direction, and Afaan Oromo was the hardest source language. A manual review found hallucinations, erasure of women's agency, and word substitutions that changed facts. The dataset gives researchers natural, culturally grounded text for studying gender-inclusive MT in data-poor settings.

Key points

  • Dataset design: Yeswa-Stories contains 1,300 manually translated, aligned sentences.
    • Amharic and Tigrinya are written in Ge'ez script and are grammatically gendered.
    • Afaan Oromo uses the Latin-based Qubee alphabet and has a gender-neutral pronoun.
    • The data is released at github.com/hhnigatu/Yeswa-Stories.
  • Automatic scores: chrF is the primary metric.
    • Google Translate reached chrF 25.78–49.61 and BLEU 4.79–15.80.
    • NLLB fell as low as BLEU 0.87 for Afaan Oromo→Tigrinya.
  • Direction effects: With Afaan Oromo as the source, Google Translate's chrF advantage over NLLB was 22.0 points into Amharic and 13.6 points into Tigrinya. Amharic as the source gave the best results.
  • Error types: Manual analysis of Afaan Oromo→Amharic outputs, such as sentences about Mary Anning, found three recurring problems:
    • hallucinated content;
    • active constructions turned passive, removing the woman as actor;
    • factual lexical errors.
  • Setup and context: The authors ran NLLB (1.3B parameters) on a Colab CPU with beam size 4 and a 200-token limit. They interpret the findings alongside the 2,400-sentence occupational benchmark of Sewunetie et al. (2024).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mamo2026yeswa,
  title = {{Yeswa-Stories: A Three-Way Parallel Dataset of Female African Figures in Low-Web Data Languages}},
  author = {Mamo, Bethelhem and Nigatu, Hellina},
  booktitle = {Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies (GITT 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.gitt-1.7/}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Blodgett, Su Lin, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5454–5476. find our card
  2. Cho, Won Ik, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. Proceedings of the First Workshop on Gender Bias in Natural Language Processing, 173–181. find
  3. Joshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282–6293. find our card
  4. Nigatu, Hellina Hailu, Bethelhem Yemane Mamo, Bontu Fufa Balcha, Debora Taye Tesfaye, El-bethel Daniel Zewdie, Ikram Behiru Nesiru, Jitu Ewnetu Hailu, and Senait Mengesha Yayo. 2025. Evaluating machine translation datasets for low-web data languages: A gendered lens. Proceedings of the Findings of t… find our card
  5. Nigatu, Hellina Hailu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. 2024. The Zeno’s Paradox of Low-Resource Languages Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. find
  6. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, and Al Youngblood et al. 2022. No language left behind: Scaling human-centered machine trans… find
  7. Prates, Marcelo O. R., Pedro H. Avelar, and Luı́s C. Lamb. 2020. Assessing gender bias in machine translation: A case study with Google Translate. Neural Computing and Applications, 32:6363–6381. find
  8. Ramesh, Krithika, Sunayana Sitaram, and Monojit Choudhury. 2023. Fairness in language models beyond English: Gaps and challenges. Findings of the Association for Computational Linguistics: EACL 2023, 2106–2119. find
  9. Savoldi, Beatrice, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845–874. find our card
  10. Savoldi, Beatrice, Sara Papi, Matteo Negri, Ana Guerberof-Arenas, and Luisa Bentivogli. 2024. What the harm? Quantifying the tangible impact of gender bias in machine translation with a human-centered study. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 1804… find
  11. Sewunetie, Walelign, Atnafu Tonja, Tadesse Belay, Hellina Hailu Nigatu, Gashaw Gebremeskel, Zewdie Mossie, Hussien Seid, and Seid Yimam. 2024. Gender bias evaluation in machine translation for Amharic, Tigrigna, and Afaan Oromoo. Proceedings of the 2nd International Workshop on Gender-Inclusive Tran… find
  12. Stanovsky, Gabriel, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 1679–1684. find our card
  13. Talat, Zeerak, Aurelie Neveol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022. You reap what you sow: On the challe… find
  14. Wairagala, Eric Peter, Jonathan Mukiibi, Jeremy Francis Tusubira, Claire Babirye, Joyce Nakatumba-Nabende, Andrew Katumba, and Ivan Ssenkungu. 2022. Gender bias evaluation in Luganda-English machine translation. Proceedings of the 15th Biennial Conference of the Association for Machine Translation i… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mamo-2026-yeswa-stories-three-way-parallel/