Elamur Research⌕
☰
home / papers / 2021 / Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies

Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies

Summary

Most NLP research on gender bias assumes there are only two genders. This leaves non-binary people invisible and exposed to harm. Dev et al. study how that binary assumption plays out in English language technologies. They first describe how complex gender identity and its language are, including singular they, neopronouns such as xe and ze, the honorific Mx., and chosen names. They then survey non-binary people who have some familiarity with AI to learn which representational and allocational harms they expect from common NLP tasks. The main harms reported are misgendering and erasure. Next, the authors probe GloVe and BERT. Words and pronouns tied to non-binary identities are rare in training text, so the models learn unstable, low-quality representations for them, sometimes with derogatory associations. In a pronoun prediction test, BERT does fairly well with he and she, is weak with they, and fails completely on neopronouns, even when the context makes the correct pronoun clear. The authors argue that fixes must be developed bottom-up together with affected communities. They also warn that treating non-binary as a single third category is harmful in itself.

Key points

  • The paper combines a theoretical account of gender in English with an IRB-reviewed survey of non-binary people (granted Exempt status, all questions optional). The analysis centers on misgendering and erasure.
  • BERT pronoun prediction accuracy: his-he 0.861, her-she 0.785, their-they 0.521, xir-xe 0.0, zir-ze 0.0. The aggregated probabilities of the correct pronoun follow the same pattern, from 0.670 for he down to about 1e-05 for xe. The authors attribute the neopronoun results partly to BERT treating these pronouns as out of vocabulary.
  • Static (GloVe) and contextual (BERT) representations of non-binary-associated terms suffer from underrepresentation in data. This creates a cycle in which models keep reinforcing erasure.
  • The paper credits earlier mitigation work, namely the GICoref coreference dataset (Cao and Daumé III) and they-substitution rewriting (Sun et al., 2021). It notes two remaining problems: neopronouns have too little real-world data, and modeling gender as fixed, discrete classes is questionable.
  • Recommendations: involve affected people in annotation and human-in-the-loop processes, monitor deployed systems for harms over time, report performance for non-binary users transparently, and give users ways to obtain recourse and give feedback.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

105 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 101 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.150 · arXiv 2108.12084

@inproceedings{dev2021harms,
  title = {{Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies}},
  author = {Dev, Sunipa and Monajatipoor, Masoud and Ovalle, Anaelia and Subramonian, Arjun and Phillips, Jeff and Chang, Kai-Wei},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.150},
  eprint = {2108.12084},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.150}
}

References

58 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In SIGCIS Conference. find
  2. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019. Fairness and Machine Learning. fairml-book.org. http://www.fairmlbook.org. ↗
  3. Elizabeth Behm-Morawitz and Dana Mastro. 2008. Mean girls? the influence of gender portrayals in teen movies on emerging adults’ gender-based attitudes and beliefs. Journalism and Mass Communication Quarterly, 85:131 – 146. find
  4. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big. Proceedings of FAccT. find
  5. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. find our card
  6. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. find our card
  7. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  8. Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online. Association for Computational Linguistics. find our card
  9. Yang Trista Cao and Hal Daumé III. 2021. An Analysis of Gender and Bias Throughout the Machine Learning Lifecyle*. Computational Linguistics, pages 1– 46. find
  10. Jessica Clarke. 2019. They, them, and theirs. 132 Harvard Law Review, page 894. find
  11. Kimberle Crenshaw. 1989. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist policies. University of Chicago Legal Forum, 1989(1):139–167. find
  12. Erin Crouch. 2018. Being non-binary in a language without gendered pronouns – estonian. Deep Baltic. find
  13. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairnes… find
  14. Rishikesh Bahadur Desai. 2018. Karnataka’s jogappas can now live a gender-fluid life. The Hindu. find
  15. Sunipa Dev. 2020. The geometry of distributed representations for better alignment, attenuated bias, and improved interpretability. find
  16. Sunipa Dev, Tao Li, Jeff M. Phillips, and Vivek Srikumar. 2020a. On measuring and mitigating biased inferences of word embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7659–7666. find
  17. Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2020b. OSCaR: Orthogonal subspace correction and rectification of biases in word embeddings. arXiv. find
  18. Sunipa Dev and Jeff M. Phillips. 2019. Attenuating bias in word vectors. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 879–887. PMLR. find
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  20. Ethan Fast, Tina Vachovsky, and Michael Bernstein. 2016. Shirtless and dangerous: Quantifying linguistic signals of gender bias in an online fiction writing community. In Proceedings of the International AAAI Conference on Web and Social Media, volume 10. find
  21. Christine Feraday. 2016. For lack of a better word: Neo-identities in non-cisgender, non-straight communities on tumblr. Ryerson University. find
  22. Susan T Fiske. 1993. Controlling other people: The impact of power on stereotyping. American psychologist, 48(6):621. find
  23. Andrew Flowers. 2015. The most common unisex names in america: Is yours one of them? FiveThir-tyEight. find
  24. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In ACL workshop for NLP Open Source Software. find our card
  25. Vasundhara Gautam. 2021. Guest lecture in pronouns: Vasundhara. In Kirby Conrod, editor, Pronoun Studies. find
  26. Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness without demographics in repeated loss minimization. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1929–1938. PMLR. find
  27. Os Keyes. 2018. The misgendering machines: Trans/hci implications of automatic gender recognition. Proc. ACM Hum.-Comput. Interact., 2(CSCW). find
  28. Os Keyes, Zoë Hitzig, and Mwenza Blell. 2021. Truth from the machine: artificial intelligence and the materialization of identity. Interdisciplinary Science Reviews, 46(1-2):158–175. find
  29. Robin Lakoff. Language and woman’s place. Language in society., 2(1). find
  30. Dave Lauer. 2020. You cannot have ai ethics without ethics. In AI and Ethics. find
  31. MAP. 2016. Unjust: How the broken criminal justice system fails transgender people. Movement Advancement Project and Center for American Progress. find
  32. Gretchen McCulloch and Lauren Gawne. 2016. Episode 2: Pronouns. little words, big jobs. Lingth-usiasm. find
  33. Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2020. Man is to person as woman is to location: Measuring gender bias in named entity recognition. In Proceedings of the 31st ACM Conference on Hypertext and Social Media, HT ’20, page 231–232, New York, NY, USA. Associa… find
  34. Alfredo Mirandé. 2016. Hombres mujeres: An indigenous third gender. Men and Masculinities, 19(4):384–409. find
  35. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  36. Micah Rajunov and Scott Duane. 2019. Nonbinary: Memoirs of Gender and Identity. Columbia University Press. find
  37. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237–7256, Online. Association for Compu… find our card
  38. The Blunt Rose. 2020. Nonbinary name list. find
  39. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… find our card
  40. Maarten Sap. 2021. Positive ai with social commonsense models. Allen Institute for Artificial Intelligence. find
  41. Morgan Klaus Scheuerman, Aaron Jiang, Katta Spiel, and Jed R. Brubaker. 2021. Revisiting gendered web forms: An evaluation of gender inputs with (non-)binary people. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA. Association for Computing… find
  42. Julia Serano. 2007. Whipping Girl: A Transsexual Woman on Sexism and the Scapegoating of Femininity. Seal Press. find
  43. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… find our card
  44. Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. 2018. Luminosoin-sight/wordfreq: v2.2. find
  45. Katta Spiel, Os Keyes, and Pınar Barlas. 2019. Patching gender: Non-binary utopias in hci. In Association for Computing Machinery, CHI EA ’19, page 1–11, New York, NY, USA. find
  46. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. find our card
  47. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… find our card
  48. Tony Sun, Kellie Webster, Apu Shah, William Yang Wang, and Melvin Johnson. 2021. They, them, theirs: Rewriting with gender-neutral english. find
  49. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. Proceedings of the EMNLP 2017. find our card
  50. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  51. Yi Chern Tan and L. Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada… find
  52. Nat Thorne, Andrew Kam-Tuck Yip, Walter Pierre Bouman, Ellen Marshall, and Jon Arcelus. 2019. The terminology of identities between, outside and beyond the gender binary - a systematic review. International Journal of Transgenderism. find
  53. Ankith Uppunda, Susan D. Cochran, Jacob G. Foster, Alina Arseniev-Koehler, Vickie M. Mays, and Kai-Wei Chang. 2021. Adapting coreference resolution for processing violent death narratives. find
  54. Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November… find
  55. Shannon Weber. 2019. Queer Media Images: LGBT Perspectives (Born This Way: Biology and Sexuality in Lady Gaga’s Pro-LGBT Media). Lexington Books. find
  56. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. find our card
  57. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find our card
  58. Appendix: Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies A Sexual Orientation Queer Bisexual Asexual Pansexual Straight Gay Questioning I don’t want to answer this question Survey Below, we provide the full Survey on Harms, our rationale for each que… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dev-2021-harms-gender-exclusivity-challenges-non/