Elamur Research⌕
☰
home / papers / 2021 / Detoxifying Language Models Risks Marginalizing Minority Voices

Detoxifying Language Models Risks Marginalizing Minority Voices

Paper · 2021 · Albert Xu, Eshaan Pathak, Eric Wallace et al.

Summary

Detoxification methods aim to stop language models from producing offensive text while keeping them useful on benign input. This paper asks whether that protection is fair to everyone. The authors detoxified GPT-2 medium with four methods (DAPT, PPLM, GeDi and output Filtering), all trained on the Jigsaw Civil Comments dataset. They then measured the models in two ways. First, they computed perplexity on tweets written in White-Aligned English (WAE), African-American English (AAE) and tweets containing minority identity mentions (MIM). Second, crowdworkers judged the quality of generated continuations.

Detoxification does reduce toxicity, but it also makes the models worse on harmless text. The loss is much larger for AAE and MIM text, and the gap grows as detoxification gets stronger. The authors trace this to spurious correlations in toxicity data: AAE and identity terms are often labelled toxic because of annotator unfamiliarity and sampling bias. This matters because such systems may push minority users to code-switch or hide their identity. It also shows that controllable generation can quietly inherit dataset biases.

Key points

  • Setup. All four techniques use GPT-2 medium with top-k sampling (k = 50).
    • GeDi relies on a finetuned GPT-2 small as its guide model.
    • Filtering picks the least toxic of 10 samples.
    • Jigsaw examples with 10–50% toxic votes were dropped because annotators disagreed on them.
  • Evaluation data.
    • Paired WAE/AAE tweets from Groenwold et al. (2020), split into toxic and nontoxic with the Perspective API.
    • 1,000 nontoxic MIM tweets collected through the Twitter API.
  • Perplexity results.
    • With PPLM, perplexity on nontoxic WAE tweets rose about 2.1×, but on MIM tweets it rose about 4.3×.
    • Raising GeDi's ω pushed the AAE-to-WAE perplexity ratio from roughly 3 (no detoxification) to nearly 400.
  • Human evaluation. Amazon Mechanical Turk workers rated 100 prompts (50 AAE, 50 WAE), each with a 30-token continuation and three raters per example.
    • Over 80% judged GeDi less topical than baseline GPT-2.
    • Every method except DAPT was rated less fluent.
    • On AAE prompts, quality dropped on every metric, most of all topicality.
  • Code release. The code is public at github.com/albertkx/detoxifying-lms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🎓University of California, BerkeleyBerkeley · United States127 works · 212,523 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap and Dan Klein. 2021. Detoxifying Language Models Risks Marginalizing Minority Voices. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.190 · arXiv 2104.06390

@inproceedings{xu2021detoxifying,
  title = {{Detoxifying Language Models Risks Marginalizing Minority Voices}},
  author = {Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap and Dan Klein},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.190},
  eprint = {2104.06390},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.190}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In ACL.
  2. Stigmatization of Language Detoxified models also have a propensity to avoid certain topics, e.g., Christopher Clark, Mark Yatskar, and Luke Zettle-mentioning a minority identity term. As a practi-moyer. 2019. Don’t take the easy way out: Ensem-cal example, the (detoxified) Microsoft Zo chatbot ble… find
  3. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In EMNLP Findings.
  4. Lisa Green. 2002. African American English: A linguistic introduction. find
  5. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American vernacular English in transformer-based text generation. In EMNLP.
  6. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL. find
  7. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In AIES. find
  8. Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. 2020. Constructing interval variables via faceted Rasch measurement and multi-task deep learning: A hate speech application. arXiv preprint arXiv:2009.10277. — [arXiv:2009.10277](https://arxiv.org/abs/2009.10277) find
  9. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  10. Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In *SEM.
  11. Ben Krause, Akhilesh Deepak Gotmare, Bryan Mc-Cann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. GeDi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367.
  12. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637.
  13. RWJF. 2017. Discrimination in America: Experiences and views. find
  14. Maarten Sap, Dallas Card, Saadia Gabriel, Choi Yejin, and Noah Smith. 2019. The risk of racial bias in hate speech detection. In ACL.
  15. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In ACL.
  16. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2020. Towards controllable biases in language generation. In EMNLP Findings.
  17. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Auto-Prompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP. find
  18. Chloe Rose Stuart-Ulin. 2018. Microsoft’s politically correct chatbot is even worse than its racist one. Quartz. find
  19. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP. find
  20. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. find
  21. Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2021. Challenges in automated debiasing for toxic language detection. In EACL.
  22. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. find
  23. Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2020. Fighting hate speech, silencing drag queens? Artificial intelligence in content moderation and risks to LGBTQ voices online. In Sexuality & Culture. find
  24. Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. 2019. Distributionally robust language modeling. In EMNLP. find
  25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2021-detoxifying-language-models-risks-marginalizing/