Elamur Research⌕
☰
home / papers / 2020 / Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make It

Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make It

Summary

When NMT systems translate into languages with grammatical gender, they often choose the wrong form for nouns that refer to people. They tend to default to masculine forms or to occupational stereotypes. Many existing fixes attach a single gender feature to the whole sentence. This paper asks whether a model can correctly use a known gender label attached to one specific word.

The authors extend the synthetic adaptation data of Saunders and Byrne (2020) with word-level tags (<M>, <F>, <N>) placed after professions. They fine-tune English-to-German and English-to-Spanish models on these sets. At test time they tag the primary entity in WinoMT, using either the reference label or an automatic RoBERTa-based tagger. They also add a check on how the second, non-coreferent entity gets inflected.

Tags strongly improve accuracy on the tagged entity. However, simple schemes apply the same gender to everyone in the sentence. Adaptation examples that contain several differently tagged people largely prevent this. A gender-neutral extension using placeholder inflections works only partly. The paper argues that gender-bias interventions need tests that reveal unintended side effects.

Key points

  • Evaluation: WinoMT has 3888 sentences (1826 male, 1822 female, 240 neutral primary entities). The authors add L2/∆L2, which measures how often the secondary entity takes the primary entity's gender label. A large ∆L2 signals over-generalization.
  • Adaptation data: The base data is the S&B set of 388 template sentences ("The [entity] finished [his|her] work"). Four tagged variants were built:
    • V1 tags professions.
    • V2 also drops the gendered possessive.
    • V3 and V4 contain two tagged entities.
  • Main result: Labeled test sentences give large accuracy gains. The multi-entity sets (V3/V4) keep ∆L2 small, and the authors recommend V3 overall.
  • Automatic tagging: RoBERTa-based tags match human labels on 84% of WinoMT sentences and perform only slightly worse than reference labels.
  • Gender-neutral translation: The authors release a neutral-only WinoMT variant with synthetic placeholder articles and noun endings. The best system reaches 56.5% primary-entity accuracy (V4, en-es). The new sets are published on GitHub (DCSaunders, tagged-gender-coref).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Machine translation

Cited by

47 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 44 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2010.05332

@inproceedings{saunders2020neural,
  title = {{Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make It}},
  author = {Saunders, Danielle and Sallis, Rosie and Byrne, Bill},
  booktitle = {Proceedings of the Second Workshop on Gender Bias in Natural Language Processing},
  year = {2020},
  eprint = {2010.05332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.05332}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohsen Abbasi, Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2019. Fairness in representation: Quantifying stereotyping as a representational harm. In Proceedings of the 2019 SIAM International Conference on Data Mining, pages 801–809. SIAM. find
  2. Lauren Ackerman. 2019. Syntactic and cognitive issues in investigating gendered coreference. Glossa: a journal of general linguistics, 4(1). find
  3. Morrigan Auxland. 2020. Para Todes: A Case Study on Portuguese and Gender-Neutrality. Journal of Languages, Texts and Society, 4:1–23. find
  4. Christine Basta, Marta R. Costa-jussà, and José A. R. Fonollosa. 2020. Towards mitigating gender bias in a decoder-based neural machine translation model by adding contextual information. In Proceedings of the The Fourth Widening Natural Language Processing Workshop, pages 99–102, Seattle, USA, Ju… find
  5. Evan D Bradley, Julia Salkind, Ally Moore, and Sofi Teitsort. 2019. Singular ‘they’and novel pronouns: Gender-neutral, nonbinary, or both? Proceedings of the Linguistic Society of America, 4(1):36–1. find
  6. Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online, July. Association for Computational Linguistics. find our card
  7. Marta R Costa-jussà, Pau Li Lin, and Cristina España-Bonet. 2020. GeBioToolkit: Automatic extraction of gender-balanced multilingual corpus of Wikipedia biographies. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4081–4088. find
  8. Kate Crawford. 2017. The trouble with bias. In Conference on Neural Information Processing Systems, invited speaker. find
  9. Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154, Florence, Italy, August. Association for Computational Linguisti… find
  10. Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems. ACM Trans. Inf. Syst., 14(3):330–347, July. find
  11. Levi CR Hord. 2016. Bucking the linguistic binary. Western Papers in Linguistics, 3(1). find
  12. Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari. 2020. “You sound just like your father” Commercial machine translation systems include stylistic biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1686–1690, Online, July. Association for Com… find
  13. Melvin Johnson. 2018. Providing gender-specific translations in Google Translate. (accessed: Aug 2020). find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. http://www.hpc.cam.ac.uk ↗
  15. Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. Using coreference links to improve Spanish-to-English machine translation. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (COR-BON 2017), pages 30–40, Valencia, Spain, April. Association for Computational Linguist… find our card
  16. Julia Misersky, Asifa Majid, and Tineke M Snijders. 2019. Grammatical gender in German influences how role-nouns are interpreted: Evidence from ERPs. Discourse Processes, 56(8):643–654. find
  17. Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 49–54, Florence, Italy, August. Association for Computational… find
  18. Benjamin Papadopoulos. 2019. Innovaciones al género morfológico en el Español de hablantes genderqueer (Morphological gender innovations in Spanish of genderqueer speakers). eScholarship, University of California. find
  19. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels, October. Association for Computational Linguistics. find our card
  20. Marcelo OR Prates, Pedro H Avelar, and Luı́s C Lamb. 2019. Assessing gender bias in machine translation: a case study with Google Translate. Neural Computing and Applications, pages 1–19. find
  21. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… find our card
  22. Danielle Saunders and Bill Byrne. 2020. Reducing gender bias in neural machine translation as a domain adaptation problem. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7724–7736, Online, July. Association for Computational Linguistics. find
  23. Alyx J Shroy. 2016. Innovations in gender-neutral French: Language practices of nonbinary French speakers on Twitter. Ms., University of California, Davis. find
  24. Artūrs Stafanovičs, Toms Bergmanis, and Mārcis Pinnis. 2020. Mitigating gender bias in machine translation with target gender annotations. In Proceedings of the Fifth Conference on Machine Translation (WMT). find
  25. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy, July. Association for Computational Linguistics. find our card
  26. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… find our card
  27. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium, October-November. Association for Computational Linguistics. find our card
  28. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia, July. Ass… find our card
  29. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  30. Lal Zimman. 2017. Transgender language reform: Some challenges and strategies for promoting trans-affirming, gender-inclusive language. Journal of Language and Discrimination, 1(1):83–104. find
  31. Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651–1661, Florence, Italy,… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/saunders-2020-neural-machine-translation-doesn-t/