Elamur Research
☰
home / papers / 2019 / The Risk of Racial Bias in Hate Speech Detection

The Risk of Racial Bias in Hate Speech Detection

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1163.pdf

Summary

Sap et al. examine whether hate speech classifiers penalize African American English (AAE). Annotators often ignore dialect and the speaker's identity, so ordinary in-group language can be marked as toxic. The authors used the Blodgett et al. (2016) lexical dialect model to estimate how likely each tweet is to be AAE. They applied it to two popular Twitter toxicity corpora, DWMW 17 and FDCL 18, and found that AAE markers correlate with offensive or abusive labels.

They then trained BiLSTM-with-attention classifiers, initialized with GloVe, on each corpus and tested them on DEMOGRAPHIC16 and USERLEVELRACE18. The models flagged AAE tweets and tweets from self-identified African American users as toxic far more often than other tweets.

In a Mechanical Turk study, showing annotators a tweet's dialect or the author's likely race reduced how often AAE tweets were rated offensive.

The work shows that text alone does not settle offensiveness. Biased labels feed into models and risk silencing minority voices, which matters for dataset design and content moderation.

Key points

  • A tweet's estimated likelihood of being AAE tracks with its being tagged "offensive" in DWMW 17 (r = 0.42), and with its being tagged "abusive" in FDCL 18 (r = 0.35).
  • The DWMW 17 classifier misclassified 46.3% of non-offensive AAE tweets as offensive, against 9.0% for White-aligned tweets. This breaks the equality-of-opportunity criterion.
  • Effects on the reference corpora:
    • In DEMOGRAPHIC16 (56M tweets), AAE tweets were over twice as likely to receive "offensive"/"abusive" predictions.
    • In USERLEVELRACE18 (5.4M tweets, 4,132 survey users), tweets by AA authors were about 1.5 times as likely to be labelled "offensive".
  • In the priming study, 1,351 tweets each received 5 annotations. The mean rating for "offensive to anyone" fell from 0.55 in the control condition to 0.44 under both dialect and race priming (p < 0.001).
  • Limitations and side findings:
    • The annotator pool was skewed, with 75% self-reported White workers.
    • PerspectiveAPI shows similar bias, and the findings also hold for the Waseem and Hovy (2016) data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

378 papers of our library cite it (of the 140673 whose bibliographies we read); 28 of them have a page here, 350 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1163

@inproceedings{sap2019risk,
  title = {{The Risk of Racial Bias in Hate Speech Detection}},
  author = {Sap, Maarten and Card, Dallas and Gabriel, Saadia and Choi, Yejin and Smith, Noah A.},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1163},
  url = {https://doi.org/10.18653/v1/p19-1163}
}

References

42 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Betty van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. CoRR, abs/1809.07572. find
  2. Wafa Alorainy, Pete Burnap, Han Liu, and Matthew Williams. 2018. Cyber hate classification: ’othering’ language and paragraph embedding. CoRR, abs/1801.07495. — [arXiv:1801.07495](https://arxiv.org/abs/1801.07495) find
  3. Monica Anderson, Skye Toor, Lee Rainie, and Aaron Smith. 2018. Activism in the social media ages. http://www.pewinternet.org/ 2018/07/11/activism-in-the-social-media-age/. Accessed: 2019-03-01. ↗
  4. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American english. In EMNLP. find our card
  5. Pete Burnap and Matthew L. Williams. 2015. Cyber hate speech on Twitter: An application of machine classification and statistical modeling for policy and decision making. Policy & Internet, 7:223–242. find
  6. Anna Chung. 2019. How automated tools discriminate against black language. https:// onezero.medium.com/how-automated-tools-discriminate-against-black-language-2ac8eab8d6db. Accessed: 2019-03-02. find
  7. Jamie Cleland. 2014. Racism, football fans, and online message boards: How social media has added a new dimension to racist discourse in English football. J. Sport Soc. Issues, 38(5):415–431. find
  8. Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM. find our card
  9. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of Conference on AI, Ethics, and Society. find
  10. Walter F. Edwards. 2004. African American Vernacular English: phonology. In A Handbook of Varieties of English: Morphology and Syntax. find
  11. Sarah Florini. 2014. Tweets, tweeps, and signifyin’: Communication and cultural performance on “Black Twitter”. Television & New Media, 15(3):223–237. find
  12. Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In ICWSM. find
  13. Deen Freelon, Charlton D. McIlwain, and Meredith D. Clark. 2016. Beyond the hashtags. http://cmsimpact.org/wp-content/uploads/2016/03/beyond\_ the\_hashtags\_2016.pdf. Accessed: 2019-03-01. ↗
  14. Jennifer Golbeck, Zahra Ashktorab, Rashad O. Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A. Geller, Quint Gergory, Rajesh Kumar Gnanasekaran, Raja Rajan Gunasekaran, Kelly M. Hoffman, Jenny Hot-tle, Vichita Jienjitlert, Shivika Khare, Ryan Lau, Marianna J. Mar… find
  15. Lisa Green. 2002. African American English: A Linguistic Introduction, 8.3.2002 edition edition. Cambridge University Press. find
  16. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In EMNLP. find our card
  17. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP. find
  18. E. Ashby Plant and Patricia G. Devine. 1998. Internal and external motivation to respond without prejudice. J. Pers. Soc. Psychol., 75(3):811–832. find
  19. Daniel Preoţiuc-Pietro and Lyle Ungar. 2018. User-level race and ethnicity predictors from Twitter text. In COLING. find
  20. Jacquelyn Rahman. 2012. The N word: Its history and use in the African American community. Journal of English Linguistics, 40(2):137–171. find
  21. Manoel Horta Ribeiro, Pedro H. Calais, Yuri A. Santos, Virgı́lio A. F. Almeida, and Wagner Meira Jr. 2018. Characterizing and detecting hateful users on Twitter. In ICWSM. find
  22. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In NeurIPS. find our card
  23. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Workshop on NLP for Social Media. find our card
  24. Raghav Kapoor, Yaman Kumar, Kshitij Rajput, Rajiv Ratn Shah, Ponnurangam Kumaraguru, and Roger Zimmermann. 2018. Mind your language: Abuse and offense detection for code-switched languages. CoRR, abs/1809.08652. — [arXiv:1809.08652](https://arxiv.org/abs/1809.08652) find
  25. Maya Sen and Omar Wasow. 2016. Race as a bundle of sticks: Designs that estimate effects of seemingly immutable characteristics. Annual Review of Political Science, 19. find
  26. Filip Klubika and Raquel Fernandez. 2018. Examining a hate speech corpus for hate speech detection and popularity prediction. In LREC. find
  27. Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting tweets against blacks. In AAAI. find
  28. Younghun Lee, Seunghyun Yoon, and Kyomin Jung. 2018. Comparative studies of detecting abusive language on twitter. CoRR, abs/1808.10245. find
  29. Gordon B. Moskowitz and Peizhong Li. 2011. Egali-tarian goals trigger stereotype inhibition: A proactive form of stereotype control. J. Exp. Soc. Psychol., 47(1):103–116. find
  30. Paul Mozur. 2018. A genocide incited on Facebook, with posts from Myanmar’s military. https://www.nytimes.com/2018/10/ 15/technology/myanmar-facebook-genocide.html. Accessed: 2018-12-6. ↗
  31. Gwenn Schurgin O’Keeffe, Kathleen Clarke-Pearson, and Council on Communications and Media. 2011. The impact of social media on children, adolescents, and families. Pediatrics, 127(4):800–804. find
  32. Ji Ho Park and Pascale Fung. 2017. One-step and two-step classification for abusive language detection on Twitter. In Proceedings of the Workshop on Abusive Language Online. find
  33. Arthur K Spears. 1998. African-American language use: Ideology and so-called obscenity. In Salikoko S Mufwene, John R Rickford, Guy Bailey, and John Baugh, editors, African-American English: Structure, History and Use, pages 226–250. Routledge New York. find
  34. Luiz Valério P Trindade. 2018. On the frontline: The rise of hate speech and racism on social media. https://discoversociety.org/ 2018/09/04/on-the-frontline-the-rise-of-hate-speech-and-racism-on-social-media/. Accessed: 2018-12-6. ↗
  35. Yequan Wang, Minlie Huang, xiaoyan zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In EMNLP. find
  36. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In NAACL Student Research Workshop. find
  37. Zeerak Waseem, James Thorne, and Joachim Bingel. 2018. Bridging the gaps: Multi task learning for domain transfer of hate speech detection. In Jennifer Golbeck, editor, Online Harassment, pages 29–55. Springer International Publishing, Cham. find
  38. Apryl Williams and Doris Domoszlai. 2013. Black-Twitter: a networked cultural identity. Harmony Institute. find
  39. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In WWW. find
  40. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical attention networks for document classification. In NAACL. find our card
  41. Danyaal Yasin. 2018. Black and banned: Who is free speech for? https: //www.indexoncensorship.org/2018/ 09/black-and-banned-who-is-free-speech-for/. Accessed: 2018-12-6. find
  42. Ziqi Zhang, David Robinson, and Jonathan A. Tepper. 2018. Detecting hate speech on Twitter using a convolution-GRU based deep neural network. In Proceedings of ESWC. find