Elamur Research
☰
home / papers / 2019 / Evaluating Gender Bias in Machine Translation

Evaluating Gender Bias in Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1164.pdf

Summary

Machine translation from English into languages with grammatical gender often forces a system to decide a person's gender. This paper checks whether that decision follows the context or falls back on occupational stereotypes. The authors built WinoMT, a challenge set made by joining the Winogender and WinoBias coreference test sets. In these English sentences, a pronoun reveals the gender of someone named only by their job, and the roles are often cast against stereotype, such as a female doctor. The evaluation needs no reference translations. Each sentence is translated, fast_align links the source entity to its counterpart in the output, and off-the-shelf morphological analyzers read the gender of the translated entity. The method was applied to eight target languages, four commercial MT services and two state-of-the-art Fairseq research models. Every system turned out to be clearly gender-biased. Accuracy was often close to chance, and results were consistently worse for female and anti-stereotypical roles. The benchmark and code are public and can be extended, and Ukrainian and Russian are among the languages covered.

Key points

  • WinoMT: 3,888 instances, balanced between male and female and between stereotypical and anti-stereotypical role assignments. Data and code are at github.com/gabrielStanovsky/mt_gender.
  • Coverage: eight target languages (Spanish, French, Italian, Russian, Ukrainian, Hebrew, Arabic, German). Systems tested were Google Translate, Microsoft Translator, Amazon Translate (it had no Ukrainian model), SYSTRAN, and the Ott et al. (2018) and Edunov et al. (2018) models. Morphology tools included spaCy, Korobov's analyzer for Russian and Ukrainian, and a ta marbuta check for Arabic.
  • Metrics: gender accuracy, ΔG (male–female F1 gap) and ΔS (pro- vs. anti-stereotypical F1 gap). German scored best for most systems. Google Translate reached only 38.4% accuracy on Ukrainian and 37.7% on Russian.
  • Adjective test: adding "handsome" or "pretty" before the entity raised Google Translate's accuracy by +10.4 points for Spanish, +11.2 for Russian and +4.5 for Ukrainian. This only works with oracle coreference, so it is not a practical fix.
  • Human check: native speakers made 9,600 annotations. They agreed with the automatic method 87% of the time on average, against 90% agreement between annotators.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

296 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 278 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1164 · arXiv 1906.00591

@inproceedings{stanovsky2019evaluating,
  title = {{Evaluating Gender Bias in Machine Translation}},
  author = {Stanovsky, Gabriel and Smith, Noah A. and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1164},
  eprint = {1906.00591},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1164}
}

References

28 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meni Adler and Michael Elhadad. 2006. An unsupervised morpheme-based HMM for Hebrew morphological disambiguation. In ACL. find our card
  2. Duygu Altinok. 2018. DEMorphy, German language morphological analyzer. CoRR, abs/1803.00902. ↗
  3. David Alvarez-Melis and Tommi S. Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In EMNLP. find
  4. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In NAACL-HLT. find our card
  5. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NIPS. find our card
  6. Aljoscha Burchardt, Vivien Macketanz, Jon Dehdari, Georg Heigold, Jan-Thorsten Peter, and Philip Williams. 2017. A linguistic evaluation of rule-based, phrase-based, and neural mt engines. The Prague Bulletin of Mathematical Linguistics, 108(1):159–170. find
  7. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  8. Pierre Isabelle, Colin Cherry, and George F. Foster. 2017. A challenge set approach to evaluating machine translation. In EMNLP. find our card
  9. Mikhail Korobov. 2015. Morphological analyzer and generator for Russian and Ukrainian languages. In Mikhail Yu. Khachay, Natalia Konstantinova, Alexander Panchenko, Dmitry I. Ignatov, and Valeri G. Labunets, editors, Analysis of Images, Social Networks and Texts, volume 542 of Communications in Computer and Information Science, pages 320– 332. Springer International Publishing. find
  10. James Kuczmarski. 2018. Reducing gender bias in google translate. find
  11. Hector J. Levesque. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. find
  12. Angelica Mucchi-Faina. 2005. Visible or influential? language reforms and gender (in) equality. Social Science Information, 44(1):189–215. find
  13. Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. CoRR, abs/1810.02268. ↗ our card
  14. Greville G Corbett. 2007. Gender and noun classes. find
  15. Colette G Craig. 1986. Noun Classes and Categorization: Proceedings of a Symposium on Categorization and Noun Classification, volume 7. John Benjamins Publishing Company. find
  16. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In HLT-NAACL. find
  17. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381. ↗
  18. Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In EMNLP. find our card
  19. Joel Escudé Font and Marta R. Costa-Jussà. 2019. Equalizing gender biases in neural machine translation with word embeddings techniques. CoRR, abs/1901.03116. ↗
  20. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. HLT-NAACL. find our card
  21. John A Hawkins. 2015. A Comparative Typology of English and German: Unifying the Contrasts. Routledge. find
  22. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. arXiv preprint arXiv:1806.00187. ↗ our card
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  24. Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In EthNLP@EACL. find our card
  25. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In NAACL-HLT. find our card
  26. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. ↗ our card
  27. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP. find our card
  28. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL-HLT. find our card