Elamur Research⌕
☰
home / papers / 2021 / They, Them, Theirs: Rewriting with Gender-Neutral English

They, Them, Theirs: Rewriting with Gender-Neutral English

Paper · 2021 · Tony Sun, Kellie Webster, Apu Shah et al.

Summary

The paper presents rewriting as a way to support gender-inclusive language. It converts English sentences that refer to one person with gendered pronouns or gendered nouns into versions that use the singular they. Simple find-and-replace is not enough for two reasons. Some pronouns map to more than one form (her can become their or them), and verbs whose subject is the pronoun must change agreement (grows becomes grow) while other verbs stay as they are.

To avoid hand labeling, the authors take Wikipedia sentences with a single gendered referent. A rule-based rewriter turns them into a synthetic parallel corpus. It combines regexes, a SpaCy dependency parse for verb agreement and GPT-2 perplexity to pick between ambiguous pronoun forms. A Fairseq Transformer is then trained on this corpus. On a new hand-curated benchmark of 500 sentence pairs, both the rewriter and the trained model score above 99 BLEU with under 1% WER. This shows that useful inclusive rewriters can be built without human-labeled training data.

Key points

  • Training data:

    • About 100M Wikipedia sentences were filtered with regexes.
    • A sentence was kept only if its pronouns came from the feminine set or the masculine set, never both.
    • This left roughly 15M gendered source sentences.

  • Test benchmark:

    • 500 manually annotated pairs.
    • Drawn equally from five domains: Twitter, Reddit, news, movie quotes and jokes.
    • Each domain has as many masculine sentences as feminine ones.

  • Results (BLEU / WER):

    SystemBLEUWER
    Unchanged source (identity baseline)90.3212.40%
    Rule-based rewriter99.630.63%
    Transformer99.440.99%

  • Model setup:

    • Architecture: 6+6 layers, d_model 512, FFN 1024, 4 heads.
    • Vocabulary: 32k BPE.
    • Optimizer: Adam at 5e-4 with 4000 warmup steps.
    • Data mix: rewritten and identity pairs in a 70/30 ratio, plus gender-swapped sources that share the same neutral target.

  • Error analysis:

    • Roughly three quarters of the rewriter's errors involve pronouns and verbs, often caused by parser POS mistakes.
    • Nearly half of the model's errors come from rare tokens such as whitespace, emojis and symbols.
    • Both systems do worse on feminine sentences. About 70% of the Wikipedia data was masculine.

Interesting ideas

  • I1. Writing a deterministic rewriter and using its output as a large silver parallel corpus lets a learned model generalize past the rules and collapse a multi-step pipeline into one step; the rules also remain a comparable baseline.
  • I2. Verb agreement is fixed only for verbs whose dependency subject is the rewritten pronoun, so other verbs in the sentence stay untouched; edits are scoped by the tree rather than by surface position.
  • I3. For pronouns with several possible targets, the rewriter lists every candidate and keeps the one a language model finds most fluent, a cheap tie-breaker on a closed choice set.
  • I4. Reporting the identity function (unchanged input) as a baseline exposes how much of a high BLEU is free, because most tokens should not change.
  • I5. Training includes identity pairs and gender-swapped duplicates, and the test set is balanced so that strength on one class cannot inflate the score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Rule-based transformations over UD trees (voice, tense, number changes) could mass-produce silver pairs, with the rule engine kept as the explainable reference.
  • I2 (back): When generating text from a modified meaning graph, re-inflect only the heads attached by nsubj to the changed entity, giving a traceable reason for every edit.
  • I3 (back): Mova could resolve such choices explainably from the UD relation (nmod:poss versus obj for her), using a scorer only as a logged last resort.
  • I4 (learn): Every rewrite or repair evaluation should include a "do nothing" negative control so that copying gets no credit.
  • I5 (learn): Silver sets for agreement or coreference rules could add mirrored variants and untouched controls, so learned repairs stay symmetric and do not fire spuriously.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓University of California, Santa BarbaraSanta Barbara · United States35 works · 9,287 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksGender Studies · 30 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Tony Sun, Kellie Webster, Apu Shah, William Yang Wang and Melvin Johnson. 2021. They, Them, Theirs: Rewriting with Gender-Neutral English. arXiv:2102.06788.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.06788 · arXiv 2102.06788

@misc{sun2021they,
  title = {{They, Them, Theirs: Rewriting with Gender-Neutral English}},
  author = {Tony Sun, Kellie Webster, Apu Shah, William Yang Wang and Melvin Johnson},
  year = {2021},
  doi = {10.48550/arxiv.2102.06788},
  eprint = {2102.06788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.06788}
}

References

26 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. ↗
  2. Su Lin Blodgett, Solon Barocas, Hal Daumé III au2, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in nlp.
  3. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2018. Debiaswe. https://bit.ly/2ruopBZ. Accessed on 12.10.2018. ↗
  4. Kate Crawford. 2017. The Trouble With Bias. Keynote at Neural Information Processing Systems (NIPS’17). find
  5. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. ACM. find
  6. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But Do Not Remove Them. In North American Chapter of the Association for Computational Linguistics (NAACL’19). ↗
  7. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  8. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  9. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872. ↗
  10. James Kuczmarski and Melvin Johnson. 2018. Gender-aware natural language translation. find
  11. Chelsea Lee. 2019. Welcome, singular "they". find
  12. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229. find
  13. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pre-trained language models. find
  14. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of the 2019 Conference of the North. ↗
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗
  16. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  17. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. In North American Chapter of the Association for Computational Linguistics (NAACL‘18).
  18. Shige Sakurai. 2017. Personal pronouns, what and why. find
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
  20. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. ↗
  21. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. ↗
  22. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. ↗
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  24. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns. In Transactions of the ACL (TACL’18). ↗
  25. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  26. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In North American Chapter of the Association for Computational Linguistics (NAACL‘18). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2021-they-them-theirs-rewriting-gender/