Elamur Research⌕
☰
home / papers / 2025 / Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation

Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation

Paper · 2025 · Michelle Kappl

Summary

Machine translation systems often change the gender of people in a sentence. They tend to default to masculine forms and follow occupational stereotypes. The first large-scale test of this, WinoMT (Stanovsky et al., 2019), used English as the source language, where a person's gender is frequently unmarked. Kappl moves the setup to German. German nouns and articles already state gender explicitly, so a wrong gender in the output cannot be blamed on an unclear input.

She builds WinoMTDE, a set of 288 Winograd-style German sentences balanced for both gender and stereotype. The stereotype labels come from German labour-agency statistics. Each translation is word-aligned with fast-align, and language-specific morphology tools read the gender of the aligned noun together with its article.

Six systems are tested on translation into seven grammatically gendered languages. Most still lean toward masculine and stereotypical renderings, and GPT-4o-mini is clearly the most faithful. The study shows that the bias comes from the models and their training data, not from source ambiguity. The dataset and code are public.

Key points

  • WinoMTDE has 144 female and 144 male subjects of interest. Of these, 121 sentences are pro-stereotypical and 121 anti-stereotypical. Nouns that could not be classified, such as PatientIn, were left out of the stereotype split.
  • Stereotype labels follow the German Classification of Occupations 2010 (revised 2020) from the Bundesagentur für Arbeit, with a 50% majority threshold. For example, Managerin counts as anti-stereotypical because 77% of that occupational group is male.
  • The systems tested are Google Translate, Microsoft Translator, Amazon Translate, DeepL, SYSTRAN and GPT-4o-mini. The target languages are Spanish, French, Italian, Ukrainian, Russian, Arabic and Hebrew. DeepL could not translate into Arabic or Hebrew.
  • The metrics are accuracy, ΔG (male F1 minus female F1) and ΔS (the gap between the pro- and anti-stereotypical subsets). Accuracy ranges from 37.0% to 95.8%, and GPT-4o-mini is best on every language pair.
  • Slavic targets are the hardest. For DE→UK, accuracy ranges from 38.2 (SYSTRAN) to 69.6 (GPT-4o-mini). Only GPT-4o-mini and DeepL exceed 50% on Slavic targets. SYSTRAN, a hybrid rule-based system, does well on Romance languages but poorly on Slavic and Semitic ones.

Interesting ideas

  • I1. Choose a source language where the correct answer is grammatically forced, so every error points to the model's prior rather than to real ambiguity. This turns a vague bias question into a clean, measurable test.
  • I2. Label "stereotypical" using external real-world statistics instead of annotator intuition. Then measure how far output distributions drift from a deliberately balanced ground truth.
  • I3. When guessing a noun's gender, use the aligned article together with the noun, because determiners often carry the clearest gender marking.
  • I4. Keep "unknown" separate from "wrong", and report accuracy both with and without unknown cases. Alignment failures otherwise make a system look worse than it is.
  • I5. Treat gender as a property of the person that must survive translation, not as a surface token to be guessed again on the target side.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build evaluation sets where morphology or agreement fully determines the right frame or referent, and use them as negative controls showing whether a component overrides textual evidence with learned habits.
  • I2 (world): Store statistical priors (for example, occupation–gender ratios) as explicit, separate knowledge, and check that the world of the text always lets explicit cues in the text override those priors.
  • I3 (frames): In coreference, combine agreement features from determiners, pronouns and noun morphology, and test pronoun resolution on Winograd-style pairs where the stereotype conflicts with grammar.
  • I4 (learn): Report abstentions from the parser or frame layer as their own category, so tool failures are not mixed into model errors.
  • I5 (back): When translating through the meta-world, carry the gender of each entity as a world attribute and require target-side generation to inflect from it, then check consistency in a round trip.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksGender Studies · 52 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 1,087 worksEvaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 works

Cite

Cite as Michelle Kappl. 2025. Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation. arXiv:2502.19104.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.19104 · arXiv 2502.19104

@misc{kappl2025are,
  title = {{Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation}},
  author = {Michelle Kappl},
  year = {2025},
  doi = {10.48550/arxiv.2502.19104},
  eprint = {2502.19104},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.19104}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845–874. ↗
  2. Meni Adler and Michael Elhadad. 2006. An Unsupervised Morpheme-Based HMM for Hebrew Morphological Disambiguation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 665–672, Sydney, Australia. Association for Computational Linguistics.
  3. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics.
  4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. ↗
  5. Statistik der Bundesagentur für Arbeit. 2020. find
  6. Klassifikation der Berufe 2010 – überarbeitete Fassung 2020. find
  7. Adina D. Sterling, Marissa E. Thompson, Shiya Wang, Abisola Kusimo, Shannon Gilmartin, and Sheri Sheppard. 2020. The confidence gap predicts the gender pay gap among STEM graduates. Proceedings of the National Academy of Sciences of the United States of America, 117(48):30303–30308. ↗
  8. Carmen Canfora and Angelika Ottmann. 2020. Risks in neural machine translation. Translation Spaces, 9(1):58–77. ↗
  9. Dries Vervecken and Bettina Hannover. 2015. Yes I Can! Effects of Gender Fair Job Descriptions on Children’s Perceptions of Job Status, Job Difficulty, and Vocational Self-Efficacy. Social Psychology, 46:76– 92. ↗
  10. Venus Chan and William Ko-Wai Tang. 2024. Gpt for translation: A systematic literature review. SN Computer Science, 5(8):986. ↗
  11. Marta R. Costa-jussà. 2019. An analysis of gender bias studies in natural language processing. Nature Machine Intelligence, 1(11):495–496. ↗
  12. Lucas Nunes Vieira, Minako O’Hagan, and Carol O’Sullivan. 2021. Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases. Information, Communication & Society, 24(11):1515–1532. ↗
  13. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia. Association for Computational Linguistics. find
  14. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  15. Lisa K. Horvath, Elisa F. Merkel, Anne Maass, and Sabine Sczesny. 2016. Does Gender-Fair Language Pay Off? The Social Perception of Professions from a Cross-Linguistic Perspective. Frontiers in Psychology, 6. find
  16. Matthijs Keep, Jeroen Oerlemans, Rik Raes, Milan Tre-soor, and Bert Wijnhoven. 2021. Evaluating gender bias in dutch machine translation. Unpublished. find
  17. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, pages 552–561, Rome, Italy. AAAI Press. find
  18. Sumant Patil and Patrick Davies. 2014. Use of Google Translate in medical communication: evaluation of accuracy. BMJ, 349:g7392. Publisher: British Medical Journal Publishing Group Section: Research. ↗
  19. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. arXiv preprint. ArXiv:1804.09301 [cs]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kappl-2025-all-spanish-doctors-male-evaluating/