Elamur Research⌕
☰
home / papers / 2021 / Challenges in Automated Debiasing for Toxic Language Detection

Challenges in Automated Debiasing for Toxic Language Detection

Summary

Toxic language classifiers often learn shortcuts that harm minority groups. They flag tweets simply for mentioning minority identities, for containing profanity, or for using features of African American English (AAE). This paper asks whether debiasing techniques developed for NLU tasks such as NLI can fix these problems. The authors test two families of methods on the Founta et al. (2018) Twitter dataset:

  • Model-based debiasing: LEARNED-MIXIN ensembles, which target known biases.
  • Bias-agnostic data filtering: removing training examples that look suspiciously easy.

Both families target lexical biases, captured by a word list the authors call TOXTRIG, and dialectal biases, estimated with the Blodgett et al. (2016) dialect model. Neither family works well. Filtering does weaken the spurious correlations in the data, yet the resulting classifiers still rely on lexical and dialect cues. They also keep flagging tweets by Black authors as toxic more often than tweets by white authors.

As a proof of concept, the authors relabel the data instead. They use GPT-3 to translate AAE tweets into white-aligned English (WAE) and flip a toxic label to non-toxic whenever the WAE version is judged non-toxic. This reduces dialectal and racial bias more than any debiasing method tested. The authors conclude that improving data quality matters more than correcting models after training.

Key points

  • Data: 86k English tweets from Founta et al. (2018), with spam removed and the hateful and abusive labels merged. This yields about 32k toxic and 54k non-toxic tweets.
  • TOXTRIG lexicon: It has three categories:
    • non-offensive minority identity mentions (NOI), detected with 26 regular expressions;
    • possibly offensive identity mentions, including slurs and reclaimed slurs (OI);
    • profanities not tied to any social group (ONI).
  • Debiased training: LEARNED-MIXIN (Clark et al., 2019) is used with an SVM bag-of-words bias-only model. Dialect is estimated with a topic model trained on 60M tweets.
  • Relabeling evaluation: A RoBERTa-large classifier trained on the relabeled data ("AAE-relabeled") shows the lowest false positive rate on AAE tweets and the lowest racial disparity, at a cost of about 1 F1 point. Three of the authors checked 50 relabeled tweets and agreed with 84% of the decisions.
  • Limitations and resources: The authors warn against using the GPT-3 translation approach in real applications. Code and data are available at github.com/XuhuiZhou/Toxic_Debias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Bias, fairness & safety

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.274

@inproceedings{zhou2021challenges,
  title = {{Challenges in Automated Debiasing for Toxic Language Detection}},
  author = {Zhou, Xuhui and Sap, Maarten and Swayamdipta, Swabha and Choi, Yejin and Smith, Noah},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.274},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.274}
}

References

50 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Su Lin Blodgett, Solon Barocas, Hal Daumé, III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proc. of ACL. find our card
  2. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proc. of EMNLP. find our card
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proc. of EMNLP. find
  4. Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In Proc. of ICML. find
  5. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP. find our card
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  7. Lisa Green. 2002. African American English: A Linguistic Introduction. Cambridge University Press. find
  8. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proc. of EMNLP. find
  9. Adam M Croom. 2013. How to do things with slurs: Studies in the way of derogatory words. In Language & communication. find
  10. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Abusive Language Workshop (at ACL). find our card
  11. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media. find our card
  12. Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2020. Fighting hate speech, silencing drag queens? artificial intelligence in content moderation and risks to lgbtq voices online. In Sexuality & Culture. find
  13. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American vernacular english in Transformer-Based text generation. In Proc. of EMNLP. find
  14. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proc. of NAACL. find
  15. Jessica Guynn. 2020. What civil rights groups want from facebook boycott: Stop hate speech and harassment of black users. find
  16. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Proc. of NeurIPS. find our card
  17. He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In EMNLP Workshop on Deep Learning Approaches for Low-Resource NLP. find our card
  18. Eun Seo Jo and Timnit Gebru. 2020. Lessons from archives: strategies for collecting sociocultural data in machine learning. In Proc. of FAT. find
  19. Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In Proc. of ACL. find our card
  20. Gabriele Kasper. 1990. Linguistic politeness: current research issues. In Journal of Pragmatics. Elsevier. find
  21. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proc. of EMNLP. find our card
  22. Jae Yeon Kim, Carlos Ortiz, Sarah Nam, Sarah Santiago, and Vivek Datta. 2020. Intersectional bias in hate speech and abusive language datasets. find
  23. Lucas Dixon, John Li, Jeffrey Scott Sorensen, Nithum Thain, and L. Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proc. of AES. find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. In arXiv preprint arXiv:1907.11692. find
  25. Marta Dynel. 2012. Swearing methodologically : the (im)politeness of expletives in anonymous commentaries on youtube. In Journal of English Studies. find
  26. Marta Dynel. 2015. The landscape of impoliteness research. In Journal of Politeness Research. find
  27. Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Proc. of WSM. find
  28. Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. 2020. Hate speech detection and racial bias mitigation in social media based on bert model. In PLOS ONE. Public Library of Science. find
  29. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proc. of EMNLP. find our card
  30. Daniel Preoţiuc-Pietro and Lyle Ungar. 2018. User-level race and ethnicity predictors from twitter text. In Proc. of COLING. find
  31. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP, pages 2383–2392. find our card
  32. Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and V. Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proc. of ICCV. find
  33. Sarah T Roberts. 2019. Behind the screen: Content moderation in the shadows of social media. Yale University Press. find
  34. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL. find
  35. Jonathan Rosa. 2019. Looking like a language, sounding like a race. Oxford University Press. find
  36. Jonathan Rosa and Nelson Flores. 2017. Unsettling race and language: Toward a raciolinguistic perspective. In Language In Society. Cambridge University Press. find
  37. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: the case of the european refugee crisis. In NLP 4 CMC Workshop. find
  38. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proc. of ACL. find our card
  39. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proc. of ACL. find our card
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. find our card
  41. Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proc. of Social NLP. find
  42. Danyaal Yasin. 2018. Black and banned: Who is free speech for? find
  43. Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proc. of AES. Association for Computing Machinery. find
  44. Roy Schwartz, Maarten Sap, Ioannis Konstas, Li Zilles, Yejin Choi, and Noah A Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. In Proc. of CoNLL. find
  45. Arthur K Spears. 1998. African-American language use: Ideology and so-called obscenity. In African-American English: Structure, History and Use. Routledge New York. find
  46. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proc. of EMNLP. find
  47. Björn Technau. 2018. Going beyond hate speech: The pragmatics of ethnic slur terms. Lodz Papers in Pragmatics, 14(1):25–43. find
  48. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020. Mind the trade-off: Debiasing NLU models without degrading the in-distribution performance. In Proc. of ACL. find
  49. Ameya Vaidya, Feng Mai, and Yue Ning. 2019. Empirical analysis of multi-task learning for reducing model bias in toxic comment detection. In Proc. of ICWSM. find
  50. Bertie Vidgen, Helen Margetts, and Alex Harris. 2019. How much online abuse is there? In Alan Turing Institute. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2021-challenges-automated-debiasing-toxic-language/