Elamur Research⌕
☰
home / papers / 2026 / Uncertainty Quantification for Evaluating Gender Bias in Machine Translation

Uncertainty Quantification for Evaluating Gender Bias in Machine Translation

Summary

Standard gender-bias evaluation in machine translation relies on gender accuracy, which only works when the source sentence settles the referent's gender. When the English source is ambiguous (e.g., "they"), there is no correct answer to score, yet a fair system should not favour either gender. The authors treat bias as a question of how the model's predictive distribution behaves. An unbiased model should prefer the correct inflection when gender is marked, and it should be more uncertain when gender is genuinely underdetermined. They measure this with sampling-based semantic uncertainty metrics, Semantic Entropy (SE) and Similarity-sensitive Shannon Entropy (S3E). They apply these to OPUS-MT and debiased variants translating English into gendered languages, using WinoMT and MGenTe. First they validate the metrics against gender accuracy on unambiguous items. They then find that a model's behaviour on ambiguous inputs is not predicted by its translation quality or by its bias on unambiguous inputs. Debiasing effects also depend on ambiguity and target language. The result is a reference-free bias metric that could extend to other kinds of bias.

Key points

  • Language pairs: Translations go from English to seven target languages: Greek, German, Italian, French, Spanish, Russian and Ukrainian. WinoMT was extended with manual translations, including professional Ukrainian translations.
  • Metrics:
    • SE groups sampled translations using a multilingual mDeberta NLI model.
    • S3E uses cosine similarity of multilingual E5 embeddings, which captures finer gender marking on verbs and adjectives.
  • Ambiguous inputs: OPUS-MT models produced only masculine nouns. Russian outputs stayed masculine even when the context was clearly feminine.
  • Debiasing on the Ukrainian "mechanic" example: feminine outputs rose from 72/128 to 128/128 for the "she" sentence. For the ambiguous "they" sentence, debiasing pushed all outputs feminine. The metric flags this overgeneration as increased bias rather than an improvement.
  • Availability and limitations: Code is on GitHub (ieva-raminta/ambiguity_gender_bias_mt). The study covers only two grammatical genders and does not address non-binary readings of "they".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.116

@inproceedings{staliunaite2026uncertainty,
  title = {{Uncertainty Quantification for Evaluating Gender Bias in Machine Translation}},
  author = {Staliunaite, Ieva and Cheng, Julius and Vlachos, Andreas},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.116},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.116}
}

References

71 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. User-centric gender rewriting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 618– 631, Seattle, United States. Association for Computational Linguistics. ↗
  2. Joris Baan, Raquel Fernández, Barbara Plank, and Wilker Aziz. 2024. Interpreting predictive probabilities: Model confidence or human label variation? In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 268–277, St. Julian’s, Malta. Association for Computational Linguistics. find
  3. Srinivas Bangalore, Bergljot Behrens, Michael Carl, Maheshwar Ghankot, Arndt Heilmann, Jean Nitzke, Moritz Schaeffer, and Annegret Sturm. 2016. Syntactic Variance and Priming Effects in Translation, pages 211–238. Springer International Publishing, Cham. find
  4. Josh Barua, Sanjay Subramanian, Kayo Yin, and Alane Suhr. 2024. Using language models to disambiguate lexical choices in translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4837–4848, Miami, Florida, USA. Association for Computational Linguistics. find
  5. Camiel J Beukeboom. 2013. Mechanisms of linguistic bias: How words reflect and maintain stereotypic expectancies. Social cognition and communication, pages 313–330. find
  6. Ondrej Bojar, Christian Buck, Chris Callison-Burch, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Herve Saint-Amand, Radu Soricut, and Lucia Specia, editors. 2013. Proceedings of the Eighth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Sofia, Bulgaria. find
  7. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia, editors. 2014. Proceedings of the Ninth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Baltimore, Maryland, USA. find
  8. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. find our card
  9. Helen S Cairns. 1973. Effects of bias on processing and reprocessing of lexically ambiguous sentences. Journal of Experimental Psychology, 97(3):337. ↗
  10. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48. find
  11. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia, editors. 2012. Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, Montréal, Canada. find
  12. Anna Farkas and Renáta Németh. 2022. How to measure gender bias in machine translation: Real-world oriented machine translators, multiple reference points. Social Sciences & Humanities Open, 5(1):100239. ↗ our card
  13. Jiali Cheng and Hadi Amiri. 2024. FairFlow: Mitigating dataset biases through undecided learning for natural language understanding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21960–21975, Miami, Florida, USA. Association for Computational Linguistics. ↗
  14. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. ↗
  15. Julius Cheng and Andreas Vlachos. 2024. Measuring uncertainty in neural machine translation with similarity-sensitive entropy. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2115–2128, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  16. Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173–181, Florence, Italy. Association for Computational Linguistics. find
  17. Jeremy Cole, Michael Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. Selectively answering ambiguous questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 530–543, Singapore. Association for Computational Linguistics. find
  18. Anna Currey, Maria Nadejde, Raghavendra Reddy Pappagari, Mia Mayer, Stanislas Lauly, Xing Niu, Benjamin Hsu, and Georgiana Dinu. 2022. MT-GenEval: A counterfactual and contextual dataset for evaluating gender accuracy in machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4287– 4299, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  19. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 862–872, New York, NY, USA. Association for Computing Machinery. find
  20. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555. ↗ our card
  21. Alan Garnham, Svenja Vorthmann, and Karolina Ka-planova. 2021. Implicit consequentiality bias in english: A corpus of 300+ verbs. Behavior Research Methods, 53:1530–1550. ↗
  22. Vagrant Gautam, Eileen Bingert, Dawei Zhu, Anne Lauscher, and Dietrich Klakow. 2024. Robust pronoun fidelity with english llms: Are they reasoning, repeating, or just biased? Transactions of the Association for Computational Linguistics, 12:1755–1779. ↗
  23. Sourojit Ghosh and Aylin Caliskan. 2023. Chatgpt perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across bengali and five other low-resource languages. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 901–912. ↗ our card
  24. Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and André F. T. Martins. 2021. Uncertainty-aware machine translation evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3920–3938, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  25. Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1926–1940, Online. Association for Computational Linguistics. find
  26. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Workshop on Widening NLP, pages 60–63, Florence, Italy. Association for Computational Linguistics. Eighth Conference on Machine Translation. Association for Computational Linguistics, Singapore. ↗ our card
  27. Hila Gonen and Kellie Webster. 2020. Automatically identifying gender issues in machine translation using perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1991–1995, Online. Association for Computational Linguistics. find
  28. Liliana Komova. 2024. Gender, Language and Perception: Linguistic Inclusivity in Russian. Ph.D. thesis, Università Ca’Foscari Venezia. find
  29. Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz, editors. 2024. Proceedings of the Ninth Conference on Machine Translation. Association for Computational Linguistics, Miami, Florida, USA. find
  30. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. ↗ our card
  31. John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3414–3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  32. Stephen C. Hora. 1996. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 54(2):217–223. Treatment of Aleatory and Epistemic Uncertainty. ↗
  33. Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2024. Decomposing uncertainty for large language models through input clarification ensembling. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. ↗
  34. Bar Iluz, Yanai Elazar, Asaf Yehudai, and Gabriel Stanovsky. 2024. Applying intrinsic debiasing on downstream tasks: Challenges and considerations for machine translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14914–14921, Miami, Florida, USA. Association for Computational Linguistics. find
  35. Yova Kementchedjhieva, Mark Anderson, and Anders Søgaard. 2021. John praised Mary because _he_? implicit causality bias and its interaction with explicit cues in LMs. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4859–4871, Online. Association for Computational Linguistics. ↗
  36. Hazel H. Kim. 2025. How ambiguous are the rationales for natural language reasoning? a simple approach to handling rationale uncertainty. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10047–10053, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  37. Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors. 2023. Proceedings of the Mikhail Korobov. 2015. Morphological analyzer and generator for russian and ukrainian languages. In Analysis of Images, Social Networks and Texts, pages 320–332, Cham. Springer International Publishing. find
  38. Ananya Kumar, Percy S Liang, and Tengyu Ma. 2019. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. ↗
  39. Selim Kuzucu, Jiaee Cheong, Hatice Gunes, and Sinan Kalkan. 2025. Uncertainty as a fairness measure. Journal of Artificial Intelligence Research, 81. ↗
  40. Moritz Laurer, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022. Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI. Preprint. Publisher: Open Science Framework. ↗
  41. Anne Lauscher, Debora Nozza, Ehm Miltersen, Archie Crowley, and Dirk Hovy. 2023. What about “em”? how commercial machine translation fails to handle (neo-)pronouns. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 377–392, Toronto, Canada. Association for Computational Linguistics. find
  42. Federico Martelli, Stefano Perrella, Niccolò Campolungo, Tina Munda, Svetla Koeva, Carole Tiberius, and Roberto Navigli. 2025. Dibimt: A gold evaluation benchmark for studying lexical ambiguity in machine translation. Computational Linguistics, pages 1–71. ↗ our card
  43. Michael Mendelson and Yonatan Belinkov. 2021. Debiasing methods in natural language understanding make bias more accessible. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1545–1557, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  44. Mante S. Nieuwland and Jos J.A. Van Berkum. 2006. Individual differences and contextual bias in pronoun resolution: Evidence from erps. Brain Research, 1118(1):155–167. ↗
  45. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3953–3962. PMLR. ↗ our card
  46. Silvia Alma Piazzolla, Beatrice Savoldi, and Luisa Bentivogli. 2023. Good, but not always fair: An evaluation of gender bias for three commercial machine translation systems. HERMES, (63):209–225. ↗
  47. Krithika Ramesh, Gauri Gupta, and Sanjay Singh. 2021. Evaluating gender bias in Hindi-English machine translation. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 16–23, Online. Association for Computational Linguistics. ↗
  48. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  49. Ricardo Rei, Nuno M Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, and André FT Martins. 2025. Tower+: Bridging generality and translation specialization in multilingual llms. arXiv preprint arXiv:2506.17080. ↗
  50. Carlo Ricotta and Laszlo Szeidl. 2006. Towards a unifying approach to diversity measures: Bridging the gap between the shannon entropy and rao’s quadratic index. Theoretical Population Biology, 70(3):237– 243. ↗
  51. Kevin Robinson, Sneha Kudugunta, Romina Stella, Sunipa Dev, and Jasmijn Bastings. 2024. MiTTenS: A dataset for evaluating gender mistranslation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4115–4124, Miami, Florida, USA. Association for Computational Linguistics. find
  52. Danielle Saunders and Bill Byrne. 2020. Reducing gender bias in neural machine translation as a domain adaptation problem. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7724–7736, Online. Association for Computational Linguistics. find
  53. Danielle Saunders and Katrina Olsen. 2023. Gender, names and other mysteries: Towards the ambiguous for gender-inclusive translation. In Proceedings of the First Workshop on Gender-Inclusive Translation Technologies, pages 85–93, Tampere, Finland. European Association for Machine Translation. ↗
  54. Beatrice Savoldi, Eleonora Cupin, Manjinder Thind, Anne Lauscher, Andrea Piergentili, Matteo Negri, and Luisa Bentivogli. 2025. mgente: A multilingual resource for gender-neutral language and translation. arXiv preprint arXiv:2501.09409. ↗
  55. Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845–874. ↗ our card
  56. Anthony Sicilia, Mert Inan, and Malihe Alikhani. 2025. Accounting for sycophancy in language model uncertainty estimation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7851–7866, Albuquerque, New Mexico. Association for Computational Linguistics. find
  57. Artūrs Stafanovičs, Toms Bergmanis, and Mārcis Pinnis. 2020. Mitigating gender bias in machine translation with target gender annotations. In Proceedings of the Fifth Conference on Machine Translation, pages 629–638, Online. Association for Computational Linguistics. find our card
  58. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. find our card
  59. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. find
  60. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do LLMs exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011–1026. find
  61. Kees van Deemter. 1998. Ambiguity and idiosyncratic interpretation. Journal of Semantics, 15(1):5–36. ↗
  62. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium. Association for Computational Linguistics. find our card
  63. Eva Vanmassenhove and Johanna Monti. 2021. gENder-IT: An annotated English-Italian parallel challenge set for cross-linguistic natural gender phenomena. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 1–7, Online. Association for Computational Linguistics. find
  64. Jun Wang, Benjamin Rubinstein, and Trevor Cohn. 2022. Measuring and mitigating name biases in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2576–2590, Dublin, Ireland. Association for Computational Linguistics. find
  65. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗ our card
  66. Wenxuan Wang, Wenxiang Jiao, Shuo Wang, Zhaopeng Tu, and Michael R. Lyu. 2024b. Understanding and mitigating the uncertainty in zero-shot translation. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 32:4894–4904. ↗
  67. Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Luxi Xing, and Weihua Luo. 2020. Uncertainty-aware semantic augmentation for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2724–2735, Online. Association for Computational Linguistics. find
  68. Minghao Wu, Yitong Li, Meng Zhang, Liangyou Li, Gholamreza Haffari, and Qun Liu. 2021. Uncertainty-aware balancing for multilingual and multi-domain neural machine translation training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7291–7305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  69. Yongjin Yang, Haneul Yoo, and Hwaran Lee. 2025. MAQA: Evaluating uncertainty quantification in LLMs regarding data uncertainty. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5846–5863, Albuquerque, New Mexico. Association for Computational Linguistics. find
  70. Runzhe Zhan, Xuebo Liu, Derek F. Wong, Cuilian Zhang, Lidia S. Chao, and Min Zhang. 2023. Test-time adaptation for machine translation evaluation by uncertainty minimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 807–820, Toronto, Canada. Association for Computational Linguistics. find
  71. Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020. Uncertainty-aware curriculum learning for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6934– 6944, Online. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/staliunaite-2026-uncertainty-quantification-evaluating-gender-bias/