Elamur Research⌕
☰
home / papers / 2020 / Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem

Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem

Summary

Saunders and Byrne address gender bias in neural machine translation (NMT). Models trained on corpora where men are mentioned more often than women tend to choose masculine or stereotyped forms when translating into languages with grammatical gender. Debiasing an entire training corpus and retraining is costly and hard to get right, so the authors treat debiasing as domain adaptation. They fine-tune an already trained model on a very small, handcrafted, gender-balanced set of profession sentences, and compare this against fine-tuning on a counterfactual subset of the original data. Naive fine-tuning causes catastrophic forgetting, which they counter in two ways: Elastic Weight Consolidation (EWC) during training, and a lattice rescoring step at inference. In rescoring, the baseline's output is expanded into gender-inflected alternatives, which are then scored by the debiased model. Tested on WinoMT for English into German, Spanish and Hebrew, the handcrafted set beats counterfactual data at a fraction of the training cost. Rescoring keeps general BLEU intact and also works on outputs from commercial black-box systems. This matters because bias can be reduced after training, without access to the original data.

Key points

  • Evaluation uses the WinoMT challenge set (3888 sentences drawn from Winogender and WinoBias) for en-de, en-es and en-he. Metrics are gender accuracy, ∆G, ∆S and M:F, plus BLEU on separate general test sets.
  • EWC-regularised adaptation offers a tunable trade-off between debiasing and translation quality. With balanced tuning, the BLEU loss stays around 0.5 on each language pair.
  • Lattice rescoring with a non-converged adapted model leaves general BLEU unchanged or slightly better, while raising WinoMT accuracy by up to 30% relative to the baseline.
  • Rescoring with the fully converged model costs at most 0.2 BLEU. It improves accuracy by 36% (en-de), 38% (en-es) and 24% (en-he); en-he gains less because its baseline is weaker.
  • On translations from Microsoft, Google, Amazon and SYSTRAN, rescoring raised accuracy for every system. For example, SYSTRAN en-de went from 48.6 to 78.4. The spread in en-de accuracy across systems shrank from 25.5 to 3.6 points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

99 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 87 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.690

@inproceedings{saunders2020reducing,
  title = {{Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem}},
  author = {Saunders, Danielle and Byrne, Bill},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.690},
  url = {https://doi.org/10.18653/v1/2020.acl-main.690}
}

References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lauren Ackerman. 2019. Syntactic and cognitive issues in investigating gendered coreference. Glossa: a journal of general linguistics, 4(1). find
  2. Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri. 2007. OpenFst: A general and efficient weighted finite-state transducer library. In International Conference on Implementation and Application of Automata, pages 11–23. Springer. find
  3. Duygu Altinok. 2018. DEMorphy, German language morphological analyzer. arXiv preprint arXiv:1803.00902. find
  4. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. 2017. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1489–1494, Copenhagen, Denmark. Association… find
  5. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translatio… find our card
  6. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in neural information processing systems, pages 4349–4357. find our card
  7. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLT evaluation campaign, IWSLT 2014. In Proceedings of the International Workshop on Spoken Language Translation, Hanoi, Vietnam, page 57. find our card
  8. Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 204–213, Dublin, Ireland. European Association for Machine Translation. find
  9. Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154, Florence, Italy. Association for Computational Linguistics. find
  10. M. Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. 2017. Multi-domain neural machine translation through unsupervised adaptation. In Proceedings of the Second Conference on Machine Translation, pages 127–137, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  11. Markus Freitag and Yaser Al-Onaizan. 2016. Fast domain adaptation for Neural Machine Translation. CoRR, abs/1612.06897. — [arXiv:1612.06897](https://arxiv.org/abs/1612.06897) find
  12. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135. find
  13. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… find our card
  14. AI HLEG. 2019. Ethics guidelines for trustworthy AI. High-Level Expert Group on Artificial Intelligence. find
  15. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with bloom embeddings. Convolutional Neural Networks and Incremental Parsing. find
  16. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics. find our card
  17. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United… find
  18. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in neural natural language processing. arXiv preprint arXiv:1807.11714. find our card
  19. Paul Michel and Graham Neubig. 2018. Extreme adaptation for personalized neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 312–318, Melbourne, Australia. Association for Computational Linguistics. find
  20. Julia Misersky, Asifa Majid, and Tineke M Snijders. 2019. Grammatical gender in German influences how role-nouns are interpreted: Evidence from erps. Discourse Processes, 56(8):643–654. find
  21. Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 49–54, Florence, Italy. Association for Computational Linguist… find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  23. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics. find our card
  24. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  25. Marcelo OR Prates, Pedro H Avelar, and Luı́s C Lamb. 2019. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, pages 1–19. find
  26. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… find our card
  27. Danielle Saunders, Felix Stahlberg, Adrià de Gispert, and Bill Byrne. 2018. Multi-representation ensembles and delayed SGD updates improve syntax-based NMT. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 319– 325, Melbourne… find
  28. Danielle Saunders, Felix Stahlberg, Adrià de Gispert, and Bill Byrne. 2019. Domain adaptive inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 222–228, Florence, Italy. Association for Computational Linguistics… find
  29. Ora Schwarzwald. 1982. Feminine formation in modern Hebrew. Hebrew Annual Review, 6:153–178. find
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  31. Felix Stahlberg, Christopher Bryant, and Bill Byrne. 2019. Neural grammatical error correction with finite state transducers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa… find
  32. Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne. 2016. Syntactically guided neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 299–305, Berlin, Germany. Association for Computational Lingu… find
  33. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. find our card
  34. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… find our card
  35. Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  36. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium. Association for Computational Linguistics. find our card
  37. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for neural machine translation. In Proceedings of the 13th Conference of the Assoc… find our card
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. find
  39. Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. 2018. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 133–143, Belgium, Brussels. Associatio… find our card
  40. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark.… find our card
  41. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  42. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA). find our card
  43. Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651–1661, Florence, Italy.… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saunders-2020-reducing-gender-bias-neural-machine/