Elamur Research⌕
☰
home / papers / 2024 / A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions

A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions

Paper · 2024 · Guilherme H. Resende, Luiz F. Nery, Fabricio Benevenuto et al.

Summary

The paper examines whether widely used toxicity and sentiment scorers treat text containing African American English (AAE) expressions unfairly. The authors test six off-the-shelf tools. Three score toxicity: Google's Perspective, Detoxify and Detoxify Unbiased. Three score sentiment: Vader, TextBlob and Flair. The tools are run on four datasets: Twitter posts, a newly collected set of single-speaker YouTube movie reviews, and the spoken interview corpora CORAAL and Buckeye. AAE usage is measured by counting matches against a hand-built dictionary of AAE expressions, drawn from sources suggested by the team behind the upcoming Oxford Dictionary of African American English. To separate dialect from other factors, the analysis controls for LIWC lexical categories, PoS-tag features and sentence-embedding similarity. Nearly every model and dataset shows bias: the AAE expression count is repeatedly a significant predictor of higher toxicity or more negative sentiment. The effect is stronger in online text than in interviews. The Unbiased variant of Detoxify is also affected, and rule-based tools are less biased than ML ones. This matters because automated moderation may unfairly penalise speakers of the dialect.

Key points

  • Models are grouped two ways: toxicity versus sentiment, and ML-based (Perspective, Detoxify, Detoxify Unbiased, Flair) versus lexical/rule-based (Vader, TextBlob).
  • A motivating example shows how large the gaps can be: an AAE rewording of "All my friends on the porch…" raises Detoxify's score from 0.0012 to 0.6145, and Perspective's from 0.2396 to 0.7886.
  • For the semantic test, the authors pair utterances from the top and bottom 2.5% of standardized scores and keep pairs with cosine similarity above 0.5. This leaves 568 pairs out of 585,679 on YouTube and 243 out of 592,606 on CORAAL/Buckeye. Twitter yielded no qualifying pairs.
  • Among these near-paraphrase pairs, the worse-scored utterance more often contains more AAE expressions:
    • YouTube: 48% of pairs in the biased direction versus 37% in the opposite direction.
    • CORAAL/Buckeye: 77% versus 12%.
    • Both differences are significant under a Binomial test at p < 0.01.
  • The AAE expression list and the YouTube dataset are released for other researchers.

Interesting ideas

  • I1. Bias is probed with pairs of utterances that mean nearly the same thing but receive opposite extreme scores, so differences in topic cannot explain the gap.
  • I2. The direction of each disagreement is counted (biased, tie, reversed), ties are dropped, and a Binomial test is applied, giving a cheap and transparent significance check.
  • I3. The dialect effect is isolated by including the count of dialect expressions as a feature next to LIWC and PoS controls, rather than relying on raw score averages.
  • I4. A curated, expert-sourced dictionary of dialect expressions serves as the measuring instrument instead of noisy demographic inference.
  • I5. The authors explain much of the bias by missing context: reappropriated slurs used inside a community are read as attacks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could include embedding-matched paraphrase pairs as a negative control, checking that frames and answers stay stable when only the dialect or wording changes.
  • I2 (learn): The same sign test could judge whether a new repair rule or parser version shifts errors systematically toward one input style instead of randomly.
  • I3 (learn): Error analysis of parsers could regress failure rates on dialect-marker counts while controlling for PoS mix, showing whether non-standard English really drives the errors.
  • I4 (tokens): A similar AAE expression lexicon (e.g., "neva", "ina", "fuhgit") could feed tokenizer normalization and lemma lookup so that dialect spellings map onto standard lemmas.
  • I5 (frames): The speech-act and implicature layer could model the relation between speaker and target before labelling an utterance hostile, instead of reacting to individual words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksBias, fairness & safety · 517 works

Cite

Cite as Guilherme H. Resende, Luiz F. Nery, Fabricio Benevenuto, Savvas Zannettou and Flavio Figueiredo. 2024. A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions. arXiv:2401.12720.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.12720 · arXiv 2401.12720

@misc{resende2024comprehensive,
  title = {{A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions}},
  author = {Guilherme H. Resende, Luiz F. Nery, Fabricio Benevenuto, Savvas Zannettou and Flavio Figueiredo},
  year = {2024},
  doi = {10.48550/arxiv.2401.12720},
  eprint = {2401.12720},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.12720}
}

References

69 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Large language models associate Muslims with violence. Nature Machine Intelligence 3, 6 (2021), 461–463. ↗
  2. CJ Adams. 2018. New York Times: Using AI to host better conversations. https://blog.google/technology/ai/new-york-times-using-ai-host-better-conversations/. ↗
  3. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). 54–59.
  4. Marzieh Babaeianjelodar, Stephen Lorenz, Josh Gordon, Jeanna Matthews, and Evan Freitag. 2020. Quantifying gender bias in different corpora. In Companion Proceedings of the Web Conference 2020. 752–759. find
  5. Arnetha F Ball. 1992. Cultural preference and the expository writing of African-American adolescents. Written Communication 9, 4 (1992), 501–532. ↗
  6. Ari Ball-Burack, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2021. Differential tweetment: Mitigating racial dialect bias in harmful tweet detection. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 116–128. find
  7. David Bamman, Chris Dyer, and Noah A Smith. 2014. Distributed representations of geographically situated language. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 828–834. find
  8. John Baugh. 1981. Runnin’down Some Lines: The Language and Culture of Black Teenagers. find
  9. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. arXiv preprint arXiv:1608.08868 (2016). ↗
  10. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter universal dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1415–1425.
  11. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016). ↗
  12. Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj, and James W Pennebaker. 2022. The development and psychometric properties of LIWC-22. Austin, TX: University of Texas at Austin (2022), 1–47. ↗
  13. Zhenpeng Chen, Jie M Zhang, Max Hort, Federica Sarro, and Mark Harman. 2022. Fairness testing: A comprehensive survey and analysis of trends. arXiv preprint arXiv:2207.10223 (2022). ↗
  14. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, Vol. 11. 512–515. ↗
  15. Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing age-related bias in sentiment analysis. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–14. Manuscript submitted to ACM find
  16. Joey Lee Dillard. 1977. Lexicon of Black English. ERIC. find
  17. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 67–73. find
  18. Jacob Eisenstein, Brendan O’Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. PloS one 9, 11 (2014), e113114. ↗
  19. Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A Survey of Race, Racism, and Anti-Racism in NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1905–1925. find
  20. Sarah Florini. 2014. Tweets, Tweeps, and Signifyin’ Communication and Cultural Performance on “Black Twitter”. Television & New Media 15, 3 (2014), 223–237. find
  21. Patricia Friedrich. 2020. When Englishes go digital. World Englishes 39, 1 (2020), 67–78. ↗
  22. Patricia Friedrich and Eduardo Diniz de Figueiredo. 2016. The sociolinguistics of digital Englishes. Routledge. ↗
  23. David Garcia, Ingmar Weber, and Venkata Rama Kiran Garimella. 2014. Gender asymmetries in reality and fiction: The bechdel test of social media. In Eighth International AAAI Conference on Weblogs and Social Media. find
  24. Anastasia Giachanou and Fabio Crestani. 2016. Like it or not: A survey of twitter sentiment analysis methods. ACM Computing Surveys (CSUR) 49, 2 (2016), 1–41. find
  25. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N project report, Stanford 1, 12 (2009), 2009. find
  26. A Gomes, D Antonialli, and T Dias-Oliva. 2019. Drag queens and artificial intelligence. Should computers decide what is toxic on the internet. Internet Lab blog (2019). find
  27. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862 (2019). ↗
  28. Mark Graham, Bernie Hogan, Ralph K Straumann, and Ahmed Medhat. 2014. Uneven geographies of user-generated information: Patterns of increasing informational poverty. Annals of the Association of American Geographers 104, 4 (2014), 746–764. ↗
  29. Lisa J Green. 2002. African American English: a linguistic introduction. Cambridge University Press. ↗
  30. Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N Asokan. 2018. All you need is" love" evading hate speech detection. In Proceedings of the 11th ACM workshop on artificial intelligence and security. 2–12. ↗
  31. Laura Hanu and Unitary team. 2020. Detoxify. Github. https://github.com/unitaryai/detoxify. ↗
  32. Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the role of grammar and word choice in bias toward african american english (aae) in hate speech classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 789–798. find
  33. Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google’s perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138 (2017). ↗
  34. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. 168–177. find
  35. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Unintended machine learning biases as social barriers for persons with disabilitiess. ACM SIGACCESS Accessibility and Computing (2020), 1–1. ↗
  36. Clayton Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, Vol. 8. 216–225. find
  37. Sen Jia, Thomas Lansdall-Welfare, and Nello Cristianini. 2015. Measuring gender bias in news images. In Proceedings of the 24th International Conference on World Wide Web. 893–898. find
  38. Jigsaw. [n. d.]. Perspective API. https://perspectiveapi.com/. Accessed: 2023-01-30. ↗
  39. Jigsaw. 2019. How Latin America’s Second Largest Social Platform Moderates More than 150K Comments a Month. https://medium.com/jigsaw/how-latin-americas-second-largest-social-platform-moderates-more-than-150k-comments-a-month-df0d8a3ac242. ↗
  40. Tyler Kendall and Charlie Farrington. 2021. The Corpus of Regional African American Language (Version 2021.07). Eugene, OR: The Online Resources for African American Language Project. find
  41. Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. NAACL HLT 2018 (2018), 43. ↗
  42. Animesh Koratana and Kevin Hu. 2018. Toxic speech detection. URL: https://web.stanford. edu/class/archive/cs/cs224n/cs224n 1194 (2018). ↗
  43. Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing Toxic Content Classification for a Diversity of Perspectives.. In SOUPS@ USENIX Security Symposium. 299–318. ↗
  44. Steven Loria. 2018. textblob Documentation. Release 0.15 2 (2018). find
  45. Patricia Georgiou Marie Pellat. 2018. Perspective Launches In Spanish With El País. https://medium.com/jigsaw/perspective-launches-in-spanish-with-el-pa%C3%ADs-dc2385d734b2. ↗
  46. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119. ↗
  47. Meena Devii Muralikumar, Yun Shan Yang, and David W McDonald. 2023. A Human-Centered Evaluation of a Toxicity Detection API: Testing Transferability and Unpacking Latent Attributes. ACM Transactions on Social Computing (2023). Manuscript submitted to ACM A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions ↗
  48. Lisa Nakamura. 2013. Cybertypes: Race, ethnicity, and identity on the Internet. Routledge. find
  49. Nikolaos Pappas, Georgios Katsimpras, and Efstathios Stamatatos. 2013. Distinguishing the popularity between topics: a system for up-to-date opinion retrieval and mining in the web. In Computational Linguistics and Intelligent Text Processing: 14th International Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part II 14. Springer, 197–209. find
  50. Daniel Borkan Patricia Georgiou, Marie Pellat. 2019. Parlons-en! Perspective and Tune are now available in French. https://medium.com/jigsaw/ perspective-tune-are-now-available-in-french-c4cf1ca198f2. ↗
  51. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001. find
  52. Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005. The Buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability. Speech Communication 45, 1 (2005), 89–95. ↗
  53. Filipe N Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Marcos André Gonçalves, and Fabrício Benevenuto. 2016. Sentibench-a benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science 5, 1 (2016), 1–29. ↗
  54. Max Roser, Hannah Ritchie, and Esteban Ortiz-Ospina. 2015. Internet. Our World in Data (2015). https://ourworldindata.org/internet. ↗
  55. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics. 1668–1678.
  56. Geneva Smitherman. 2000. Black talk: Words and phrases from the hood to the amen corner. Houghton Mifflin Harcourt. find
  57. Kaikai Song, Ting Yao, Qiang Ling, and Tao Mei. 2018. Boosting image sentiment analysis with visual attention. Neurocomputing 312 (2018), 218–228. ↗
  58. Ezekiel Soremekun, Sakshi Udeshi, and Sudipta Chattopadhyay. 2022. Astraea: Grammar-based fairness testing. IEEE Transactions on Software Engineering 48, 12 (2022), 5188–5211. ↗
  59. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics 37, 2 (2011), 267–307. ↗
  60. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the first ACL workshop on ethics in natural language processing. 53–59.
  61. Mike Thelwall. 2014. Heart and soul: Sentiment strength detection in the social web with sentistrength, 2017. Cyberemotions: Collective emotions in cyberspace (2014). find
  62. Pranav Narayanan Venkit and Shomir Wilson. 2021. Identification of bias against people with disabilities in sentiment analysis and toxicity detection models. arXiv preprint arXiv:2111.13259 (2021). ↗
  63. Hao Wang, Doğan Can, Abe Kazemzadeh, François Bar, and Shrikanth Narayanan. 2012. A system for real-time twitter sentiment analysis of 2012 us presidential election cycle. In Proceedings of the ACL 2012 system demonstrations. 115–120. find
  64. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems 33 (2020), 5776–5788. find
  65. Maciej Widawski. 2015. African American slang: A linguistic description. Cambridge University Press. find
  66. Theresa Wilson, Paul Hoffmann, Swapna Somasundaran, Jason Kessler, Janyce Wiebe, Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005. OpinionFinder: A system for subjectivity analysis. In Proceedings of HLT/EMNLP 2005 Interactive Demonstrations. 34–35.
  67. Austin P Wright, Omar Shaikh, Haekyu Park, Will Epperson, Muhammed Ahmed, Stephane Pinel, Duen Horng Chau, and Diyi Yang. 2021. RECAST: Enabling user recourse and interpretability of toxicity detection models with interactive visualization. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–26. find
  68. Ali Yadollahi, Ameneh Gholipour Shahraki, and Osmar R Zaiane. 2017. Current state of text sentiment analysis from opinion to emotion mining. ACM Computing Surveys (CSUR) 50, 2 (2017), 1–33. ↗
  69. Min Yang, Qiang Qu, Xiaojun Chen, Chaoxue Guo, Ying Shen, and Kai Lei. 2018. Feature-enhanced attention network for target-dependent sentiment classification. Neurocomputing 307 (2018), 91–97. Received 22 February 2024; revised 12 March 2024; accepted 5 June 2024 Manuscript submitted to ACM ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/resende-2024-comprehensive-view-biases-toxicity-sentiment/