Elamur Research⌕
☰
home / papers / 2020 / Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus

Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus

Summary

Translating from English, which has little grammatical gender, into gender-marked languages forces systems to choose masculine or feminine forms. When the text gives no clue, models tend to default to masculine. The authors ask whether speech translation (ST) can use the audio signal, for example the speaker's voice, to make better gender choices. They build MuST-SHE, a natural benchmark drawn from TED talks in MuST-C for English-Italian and English-French. Each segment is annotated with gender phenomena, split by whether gender can be inferred from the audio (Category 1) or from the utterance content (Category 2). They then compare three systems: a cascade ST system, a cascade given oracle speaker-gender tags, and an end-to-end model. To isolate gender errors from general translation quality, they score each system against both correct and gender-swapped references, using BLEU and accuracy on gender-marked words. The end-to-end model uses audio cues better than the plain cascade for speaker-dependent cases, but it still lags the oracle-tagged cascade. All systems show bias toward masculine forms.

Key points

  • MuST-SHE is a subset of MuST-C with roughly 1,000 (audio, transcript, translation) triplets per language pair (En-It, En-Fr). It is released under CC BY NC ND 4.0, and an English-Spanish section was added in April 2020.
  • Dataset design started from a manual cross-lingual analysis of 2,500 randomly sampled MuST-C sentences. About half of the segments deliberately have a speaker gender that differs from the gender of the phenomenon to translate.
  • Gender-marked words are sparse: about 2,000 out of about 30,000 words, or roughly 1.8 per sentence. Corpus-level BLEU therefore hides differences that word-level accuracy reveals. On accuracy, End2End beats Cascade for En-It but not for En-Fr.
  • Category 1 (speaker's own gender):
    • The plain Cascade does worst and strongly prefers masculine forms for female speakers.
    • End2End is clearly better than Cascade.
    • Cascade+Tag is best by a wide margin.
  • Category 2 (gender cued in the text):
    • End2End is weakest on feminine forms, likely because it was trained on far less data.
    • Cascade+Tag is slightly worse than Cascade, because speaker-gender information can act as noise in these cases.

Interesting ideas

  • I1. Contrastive references: pair each correct reference with a version where only the gendered words are swapped, then report the score difference. This isolates one phenomenon from overall translation quality. It could be reused for other agreement features, such as number, formality or animacy.
  • I2. Splitting by information source: grouping test items by where the disambiguating cue comes from (audio vs. text) shows which input modality a system actually exploits. This makes it a useful diagnostic for any multimodal model.
  • I3. Targeted accuracy over corpus metrics: when a phenomenon touches only a few words per sentence, corpus-level metrics like BLEU dilute it. Scoring only the annotated words exposes differences that the global metric hides.
  • I4. Built-in mismatch cases: deliberately including examples where a tempting cue (speaker gender) does not match the correct answer tests whether models use the cue appropriately or just over-rely on it.
  • I5. Oracle upper bound: adding a system fed the true metadata (gold speaker-gender tags) gives a ceiling. Comparing realistic systems against that ceiling shows how much room remains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

61 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 57 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.619

@inproceedings{bentivogli2020gender,
  title = {{Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus}},
  author = {Bentivogli, Luisa and Savoldi, Beatrice and Negri, Matteo and Gangi, Mattia A. Di and Cattoni, Roldano and Turchi, Marco},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.619},
  url = {https://doi.org/10.18653/v1/2020.acl-main.619}
}

References

46 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tanel Alumäe and Mikko Kurimo. 2010. Efficient Estimation of Maximum Entropy Language Models with n-gram Features: an SRILM Extension. In Proceedings of INTERSPEECH 2010, pages 1820– 1823, Chiba, Japan. find
  2. Alexandre Bérard, Olivier Pietquin, Laurent Besacier, and Christophe Servan. 2016. Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation. In Proceedings of the NIPS Workshop on end-to-end learning for speech and audio processing, Barcelona, Spain. NIPS. ↗
  3. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of BLEU in Machine Translation Research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics. find
  4. Yang Trista Cao and Hal Daumé III. 2019. Toward Gender-Inclusive Coreference Resolution. arXiv preprint arXiv:1910.13913. ↗ our card
  5. Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On Measuring Gender Bias in Translation of Gender-neutral Pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173–181, Florence, Italy. Association for Computational Linguistics. find
  6. Greville G. Corbett. 1991. Gender. Cambridge University Press, Cambridge, UK. 8 https://ict.fbk.eu/ units-hlt-mt-e2eslt/ Kate Crawford. 2017. The Trouble with Bias. In Conference on Neural Information Processing Systems (NIPS) – Keynote, Long Beach, California. ↗
  7. Steven Davis and Paul Mermelstein. 1980. Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4):357–366. ↗
  8. Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012–2017, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  9. Mattia A. Di Gangi, Robert Enyedi, Alessandra Brusadin, and Marcello Federico. 2019b. Robust Neural Machine Translation for Clean and Noisy Speech Transcripts. In Proceedings of the 16th International Workshop on Spoken Language Translation (IWSLT), Hong Kong. ↗
  10. Mattia A. Di Gangi, Matteo Negri, and Marco Turchi. 2019c. Adapting Transformer to End-to-end Spoken Language Translation. In Proceedings of INTER-SPEECH 2019, Graz, Austria. find
  11. Selen Hande Kabil, Hannah Muckenhirn, and Mathew Magimai-Doss. 2018. On Learning to Identify Genders from Raw Speech Signal Using CNNs. In Proceedings of INTERSPEECH 2018, pages 287–291, Hyderabad, India. find
  12. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana. ↗ our card
  13. Ali Can Kocabiyikoglu, Laurent Besacier, and Olivier Kraif. 2018. Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation. In Proceedings of LREC 2018, Miyazaki, Japan. find
  14. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278–2324. ↗
  15. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2019. Gender Bias in Neural Natural Language Processing. arXiv preprint arXiv:1807.11714v2. ↗ our card
  16. Matthias Eck and Chiori Hori. 2005. Overview of the IWSLT 2005 Evaluation Campaign. In Proceedings of the 2nd International Workshop on Spoken Language Translation (IWSLT), Pittsburgh, Pennsylvania. find
  17. Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 49–54, Florence, Italy. Association for Computational Linguistics. find
  18. Mostafa Elaraby, Ahmed Y. Tawfik, Mahmoud Khaled, Hany Hassan, and Aly Osama. 2018. Gender Aware Spoken Language Translation Applied to English-Arabic. In Proceedings of the 2nd International Conference on Natural Language and Speech Processing (ICNLSP), pages 1–6, Algiers, Algeria. find
  19. Jan Niehues, Roldano Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi, and Marcello Federico. 2018. find
  20. The IWSLT 2018 Evaluation Campaign. In Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT), Bruges, Belgium. find
  21. Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154, Florence, Italy. Association for Computational Linguistics. find
  22. Jan Niehues, Roldano Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico. 2019. find
  23. The IWSLT 2019 Evaluation Campaign. In Proceedings of the 16th International Workshop on Spoken Language Translation (IWSLT), Hong Kong, Hong Kong. find
  24. Charles F. Hockett. 1958. A Course in Modern Linguistics. Macmillan, New York, US. ↗
  25. Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2019. Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning. arXiv preprint arXiv:1911.04283. ↗
  26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ASR Corpus Based on Public Domain Audio Books. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, Brisbane, Australia. find
  27. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, pages 448–456, Lille, France. ↗
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of ACL 2002, pages 311–318, Philadelphia, Pennsylvania. Association for Computational Linguistics. find our card
  29. Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A Simple Data Augmentation Method for Automatic Speech Recognition. arXiv preprint arXiv:1904.08779. ↗ our card
  30. Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved Speech-to-Text Translation with the Fisher and Callhome Spanish–English Speech Translation Corpus. In Proceedings of the 10th International Workshop on Spoken Language Translation (IWSLT), Heidelberg, Germany. find
  31. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The Kaldi Speech Recognition Toolkit. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 1–4, Big Island, Hawaii. find
  32. Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahremani, Vimal Manohar, Xingyu Na, Yiming Wang, and Sanjeev Khudanpur. 2016. Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI. In Proceedings of INTER-SPEECH 2016, pages 2751–2755, San Francisco, California. find
  33. Marcelo O. R. Prates, Pedro H. C. Avelar, and Luı́s C. Lamb. 2018. Assessing Gender Bias in Machine Translation: a Case Study with Google Translate. Neural Computing and Applications, pages 1–19. ↗ our card
  34. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  35. Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loı̈c Barrault, Lucia Specia, and Florian Metze. 2018. How2: A Large-scale Dataset For Multimodal Language Understanding. In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL), Montreal, Canada. NIPS. ↗
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural Machine Translation of Rare Words with Subword Units. arXiv preprint arXiv:1508.07909. ↗
  37. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of AMTA, pages 223–231, Cambridge, Massachusetts. find our card
  38. Matthias Sperber, Graham Neubig, Ngoc-Quan Pham, and Alex Waibel. 2019. Self-attentional models for lattice inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1185–1197, Florence, Italy. Association for Computational Linguistics. find
  39. Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward Robust Neural Machine Translation for Noisy Input Sequences. In Proceedings of the 14th International Workshop on Spoken Language Translation (IWSLT), Tokyo, Japan. ↗
  40. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. find our card
  41. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating Gender Bias in Natural Language Processing: Literature Review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics. ↗ our card
  42. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium. Association for Computational Linguistics. find our card
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of NIPS 2017, pages 5998–6008, Long Beach, California. NIPS. find
  44. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. ↗ our card
  45. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  46. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bentivogli-2020-gender-danger-evaluating-speech-translation/