Elamur Research⌕
☰
home / papers / 2025 / Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English

Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English

Paper · 2025 · Rebecca Dorn, Christina Chance, Casandra Rusti et al.

Summary

The paper examines whether text-based emotion recognition treats African American Vernacular English (AAVE) differently from General American English (GAE). The authors take 2.7M geo-tagged Los Angeles tweets and give each one a continuous dialect density score (DDM). The score averages normalized detectors for individual sociolinguistic features, including some specific to Los Angeles. They annotate 875 tweets, balanced between high and low AAVE density, for emotion presence and intensity. To get a reference that respects who can read the dialect, high-AAVE tweets receive "silver" labels from African American, AAVE-fluent annotators only. Low-AAVE tweets are labeled by everyone.

Against this reference, GPT and BERT-based models produce false anger predictions on AAVE at more than twice their GAE rate. Profanity-based dialect features push model and outgroup judgments much more than ingroup ones. Mapped onto Census tracts, predicted anger rises and predicted joy falls with the share of African American residents. The authors argue that emotion AI can therefore encode the "angry Black person" stereotype. Annotation by the authoring community and dialect-aware design are needed before such systems are deployed.

Key points

  • Data: 5.8M geo-tagged tweets from Los Angeles County (Sep 2010 – Nov 2014) were filtered to 2.7M. The authors dropped tweets of ≤5 tokens and tweets with links, anonymized usernames and turned emojis into text descriptors.
  • DDM: each AAVE feature is approximated separately with regex, dependency parsing, NER or perplexity, then normalized and averaged into one scalar. Examples include habitual "be", negative concord and the Los Angeles continuative "steady".
  • Annotation: 11 annotators labeled 875 tweets. Ingroup annotators marked more emotion overall, except disgust on high-AAVE tweets, where outgroup annotators marked more.
  • Model errors: SpanEmo wrongly predicts anger 25% of the time for GAE tweets, a rate that climbs to 60% for AAVE tweets. Anger and disgust false positives more than double on AAVE across the tested models.
  • Neighborhood effect: the share of African American residents correlates with predicted anger (r = 0.27, p < 0.01) and negatively with joy (r = −0.10, p < 0.01). Emotions were collapsed into 7 categories for the study.

Interesting ideas

  • I1. Dialect is measured as a graded density built from separate per-feature detectors rather than as a yes/no label, which keeps each contributing feature visible and auditable.
  • I2. Reference labels are taken only from raters competent in the variety of each subset, instead of pooling all raters into one majority vote.
  • I3. Error rates are reported per dialect stratum, with false positives in focus, so the gap between groups shows up even when overall accuracy looks fine.
  • I4. Regressions over individual dialect features reveal which surface cues, such as profanity, actually drive a judgment.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): a per-sentence AAVE feature score (habitual be, negative concord, "steady") could decide when to apply UD dialect conversion rules. It could also flag parses where the perceptron parser or gparse is likely to be unreliable.
  • I2 (learn): when agreeing silver data across witness parsers or LLM teacher tiers, each witness's vote could be weighted by its proven competence on that dialect stratum. An unfamiliar teacher would then count like an outgroup annotator.
  • I3 (learn): evaluation could split results by dialect density, with low-AAVE text serving as a control for high-AAVE text. This would catch components that only fail on marked varieties.
  • I4 (frames): the same check could test whether prag's irony, hostility or speech-act decisions fire on lexical profanity alone. In-group banter like "You're a fool" should be read as play, not insult.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksBias, fairness & safety · 517 works

Cite

Cite as Rebecca Dorn, Christina Chance, Casandra Rusti, Charles Bickham Jr., Kai-Wei Chang, Fred Morstatter and Kristina Lerman. 2025. Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English. arXiv:2511.10846.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.10846 · arXiv 2511.10846

@misc{dorn2025reinforcing,
  title = {{Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English}},
  author = {Rebecca Dorn, Christina Chance, Casandra Rusti, Charles Bickham Jr., Kai-Wei Chang, Fred Morstatter and Kristina Lerman},
  year = {2025},
  doi = {10.48550/arxiv.2511.10846},
  eprint = {2511.10846},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.10846}
}

References

78 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H Samy Alim. 2006. Roc the mic right: The language of hip hop culture. Routledge. find
  2. Laberiano Andrade-Arenas, Cesar Yactayo-Arias, and Félix Pucuhuayla-Revatta. 2024. Therapy and Emotional Support through a Chatbot. International Journal of Online & Biomedical Engineering 20, 2 (2024). ↗
  3. John Baugh. 1979. Linguistic Style-Shifting in Black English. PhD dissertation. University of Pennsylvania, Philadelphia. find
  4. John Gordon Baugh Jr. 1979. Linguistic Style-Shifting in Black English. University of Pennsylvania. find
  5. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic Dialectal Variation in Social Media: A Case Study of African-American English. arXiv:1608.08868 (Aug. 2016). http://arxiv.org/abs/1608.08868 arXiv:1608.08868 [cs]. ↗
  6. Su Lin Blodgett and Brendan O’Connor. 2017. Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English. arXiv:1707.00061 [cs.CY] https://arxiv.org/abs/1707.00061 ↗
  7. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter Universal Dependency Parsing for African-American and Mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, Melbourne, Australia, 1415–1425. doi:10.18653/v1/P18-1131 ↗
  8. Deon W. Brown, Fantasy T. Lozada, Zewelanji N. Serpell, Vivian A. Dzokoto, and Julie C. Dunsmore. 2025. African American College Students’ Contextual Emotion Expression. Affective Science (Jan. 2025). doi:10.1007/s42761-024-00286-6 ↗
  9. Rafael Calvo and Sunghwan Kim. 2012. Emotions in text: Dimensional and categorical models. Computational Intelligence early view (09 2012). doi:10.1111/j.1467-8640.2012.00456.x ↗
  10. Christina Chance. 2022. Zoom Audio Transcription Accuracy for African American Vernacular English. Southern California Synposium for Natural Language Processing (2022). find
  11. Christina Chance and Kai-Wei Chang. 2025. Re-imagining Virtual Communities: Ethical Guidelines for Studying Black Twitter. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8. 541–553. find
  12. Georgios Chochlakis, Gireesh Mahajan, Sabyasachee Baruah, Keith Burghardt, Kristina Lerman, and Shrikanth Narayanan. 2023. Using emotion embeddings to transfer knowledge between emotions, languages, and annotation formats. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. find
  13. Georgios Chochlakis, Alexandros Potamianos, Kristina Lerman, and Shrikanth Narayanan. 2024. The strong pull of prior knowledge in large language models and its impact on emotion recognition. arXiv preprint arXiv:2403.17125 (2024). ↗
  14. Alan S Cowen and Dacher Keltner. 2017. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the national academy of sciences 114, 38 (2017), E7900–E7909. ↗
  15. Archie Crowley. 2025. “And it just becomes queer slang”: Race, linguistic innovation, and appropriation within trans communities in the US South. Journal of Linguistic Anthropology (2025), e70008. Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English • find
  16. Nicholas Deas, Jessica Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. 2023. Evaluation of African American Language Bias in Natural Language Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6805– 6824. doi:10.18653/v1/2023.emnlp-main.421 ↗
  17. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547 (2020). ↗
  18. Rebecca Dorn, Lee Kezar, Fred Morstatter, and Kristina Lerman. 2024. Harmful speech detection by language models exhibits genderqueer dialect bias. In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1–12. find
  19. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). ↗
  20. Paul Ekman. 1992. An Argument for Basic Emotions. Cognition and Emotion 6, 3 (1992), 169–200. doi:10.1080/02699939208411068 ↗
  21. Paul Ekman and Wallace V Friesen. 1978. Facial action coding system. Environmental Psychology & Nonverbal Behavior (1978). find
  22. Explosion. 2025. spaCy: Industrial-strength Natural Language Processing in Python. https://spacy.io/ Online; Accessed 29-September-2025. https://spacy.io/. ↗
  23. Sarah Florini. 2014. Tweets, Tweeps, and Signifyin’: Communication and Cultural Performance on “Black Twitter”. Television & New Media 15, 3 (2014), 223–237. arXiv:https://doi.org/10.1177/1527476413480247 doi:10.1177/1527476413480247 ↗
  24. Justin P Friesen, Kerry Kawakami, Larissa Vingilis-Jaremko, Regis Caprara, David M Sidhu, Amanda Williams, Kurt Hugenberg, Rosa Rodríguez-Bailón, Elena Cañadas, and Paula Niedenthal. 2019. Perceiving happiness in an intergroup context: The role of race and attention to the eyes in differentiating between true and false smiles. Journal of Personality and Social Psychology 116, 3 (2019), 375. ↗
  25. Stephan Gramley, Vivian Gramley, and Kurt-Michael Pätzold. 2020. A survey of modern English. Routledge. find
  26. Lisa Green. 2021. Aspect and predicate phrases in African-American vernacular English (1 ed.). Routledge, London, 41–74. doi:10.4324/ 9781003165330-3 find
  27. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American Vernacular English in Transformer-Based Text Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 5877–5883. doi:10.18653/v1/2020.emnlp-main.473 ↗
  28. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). ↗
  29. Abhay Gupta, Ece Yurtseven, Philip Meng, and Kevin Zhu. 2024. AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark. In Proceedings of the Third Workshop on NLP for Positive Impact, Daryna Dementieva, Oana Ignat, Zhijing Jin, Rada Mihalcea, Giorgio Piatti, Joel Tetreault, Steven Wilson, and Jieyu Zhao (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 327–333. doi:10.18653/v1/2024.nlp4pi-1.28 ↗
  30. Amy G Halberstadt, Alison N Cooke, Pamela W Garner, Sherick A Hughes, Dejah Oertwig, and Shevaun D Neupert. 2022. Racialized emotion recognition accuracy and anger bias of children’s faces. Emotion 22, 3 (2022), 403. find
  31. Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 2021. Mitigating racial biases in toxic language detection with an equity-based ensemble framework. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1–11.
  32. Sooji Han, Rui Mao, and Erik Cambria. 2022. Hierarchical Attention Network for Explainable Depression Detection on Twitter Aided by Metaphor Concept Mappings. (9 2022). http://arxiv.org/abs/2209.07494 ↗
  33. Kasia Hitczenko, Henry R Cowan, Matthew Goldrick, and Vijay A Mittal. 2022. Racial and Ethnic Biases in Computational Approaches to Psychopathology. Schizophrenia Bulletin 48, 2 (March 2022), 285–288. doi:10.1093/schbul/sbab131 ↗
  34. Kurt Hugenberg and Galen V. Bodenhausen. 2004. Ambiguity in Social Categorization: The Role of Prejudice and Facial Affect in Race Categorization. Psychological Science 15, 5 (2004), 342–345. arXiv:https://doi.org/10.1111/j.0956-7976.2004.00680.x doi:10.1111/j.0956-7976.2004.00680.x PMID: 15102145. ↗
  35. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). ↗
  36. Kokil Jaidka, Salvatore Giorgi, H Andrew Schwartz, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2020. Estimating geographic subjective well-being from Twitter: A comparison of dictionary and data-driven language methods. Proceedings of the national academy of sciences 117, 19 (2020), 10165–10171. ↗
  37. Alexander Johnson, Kevin Everson, Vijay Ravi, Anissa Gladney, Mari Ostendorf, and Abeer Alwan. 2022. Automatic Dialect Density Estimation for African American English. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, Vol. 2022-September. International Speech Communication Association, 1283–1287. doi:10.21437/Interspeech.2022-796 ↗
  38. Taylor Jones. 2015. Toward a Description of African American Vernacular English Dialect Regions Using “Black Twitter”. American Speech 90, 4 (Nov. 2015), 403–440. doi:10.1215/00031283-3442117 • Dorn et al. ↗
  39. Melanie Hines Knapp. 2015. African American vernacular English (Aave) in the classroom: The attitudes and ideologies of urban educators toward Aave. University of Mississippi Electronic Theses and Dissertations (2015). find
  40. William Labov. 1969. Contraction, Deletion, and Inherent Variability of the English Copula. Language 45, 4 (1969), 715–762. http://www.jstor.org/stable/412333 ↗
  41. Latimer AI. 2025. Latimer AI. https://www.latimer.ai/. Accessed: 2025-03-27. ↗
  42. Stanley E Legum et al. 1971. The Speech of Young Black Children in Los Angeles. (1971). find
  43. Fantasy T Lozada, Tennisha N Riley, Evandra Catherine, and Deon W Brown. 2022. Black emotions matter: Understanding the impact of racial oppression on Black youth’s emotional development: Dismantling systems of racism and oppression during adolescence. Journal of research on adolescence 32, 1 (2022), 13–33. ↗
  44. Joshua L. Martin and Kevin Tang. 2020. Understanding Racial Disparities in Automatic Speech Recognition: The Case of Habitual “be”. In Interspeech 2020. 626–630. doi:10.21437/Interspeech.2020-2893 ↗
  45. Stefan Martin and Walt Wolfram. 2021. The sentence in African-American vernacular English (1 ed.). Routledge, London, 11–40. doi:10.4324/9781003165330-2 ↗
  46. Lynda Mugglestone. 2003. Talking proper: The rise of accent as social symbol. OUP Oxford. find
  47. Ashritha R Murthy and KM Anil Kumar. 2021. A review of different approaches for detecting emotion from text. In IOP Conference Series: Materials Science and Engineering, Vol. 1110. IOP Publishing, 012009. find
  48. Durga Niraula. 2014. 2014 Population and Poverty at Split Tract data.lacounty.gov. [Accessed 28-04-2025]. find
  49. Jenny Park, Mary Catherine Beach, Dingfen Han, Richard D. Moore, P. Todd Korthuis, and Somnath Saha. 2020. Racial disparities in clinician responses to patient emotions. Patient Education and Counseling 103, 9 (2020), 1736–1744. doi:10.1016/j.pec.2020.03.019 ↗
  50. W Gerrod Parrott. 2001. Emotions in social psychology: Essential readings. psychology press. find
  51. Laura Patnaude, Carolina Vásquez Lomakina, Akshat Patel, Gulhan Bizel, et al. 2021. Public emotional response on the Black Lives Matter movement in the summer of 2020 as analyzed through Twitter. International Journal of Marketing Studies 13, 1 (2021), 1–69. ↗
  52. Flor Miriam Plaza-del Arco, Alba A. Cercas Curry, Amanda Cercas Curry, and Dirk Hovy. 2024. Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions. 5696–5710 pages. https://aclanthology.org/2024.lrec-main.506/ ↗
  53. Robert Plutchik and Henry Kellerman. 1980. Emotion, theory, research, and experience: theory, research and experience. Academic press. find
  54. Alyssa Powell. 2023. The Switch: Understanding the Perception of African American Linguistic Repertoires. (2023). find
  55. Geoffrey K Pullum. 2018. Slurs and obscenities: lexicography, semantics, and philosophy. Bad words: philosophical perspectives on slurs (2018), 168–92. find
  56. Lauren Rhue. 2018. Racial Influence on Automated Perceptions of Emotions. SSRN Electronic Journal (2018). doi:10.2139/ssrn.3281765 ↗
  57. John Rickford. 2003. African American Vernacular English. In The Oxford Encyclopedia of Linguistics (2 ed.). Oxford University Press. find
  58. John R. Rickford. 2021. The creole origins of African-American vernacular English: Evidence from copula absence (1 ed.). Routledge, London, 169–220. doi:10.4324/9781003165330-7 ↗
  59. Abel Salinas and Fred Morstatter. 2024. The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance. arXiv preprint arXiv:2401.03729 (2024). ↗
  60. Diana Santos and Belinda Maia. 2018. Language, emotion, and the emotions: A computational introduction. Language and Linguistics compass 12, 6 (2018), e12279. ↗
  61. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2021. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. arXiv preprint arXiv:2111.07997 (2021). ↗
  62. Kelly Simpson. 2012. The Great Migration: Creating a New Black Identity in Los Angeles. find
  63. Geneva Smitherman. 2021. Word from the hood: The lexicon of African-American vernacular English (1 ed.). Routledge, London, 223–248. doi:10.4324/9781003165330-8 ↗
  64. Arthur K Spears. 2007. African American communicative practices: Performativity, semantic license and augmentation. Talkin Black talk: Language, education, and social change (2007), 100–111. find
  65. Arthur K Spears. 2013. African-American language use: Ideology and so-called obscenity. In African-American English. Routledge, 226–250. find
  66. Arthur K Spears. 2019. Rickford’s list of African American English grammatical features: an update. In The Routledge companion to the work of John R. Rickford. Routledge, 79–89. ↗
  67. Ian Stewart. 2014. Now we stronger than ever: African-American English syntax in Twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics. 31–37.
  68. LA Times. 2010. Mapping L.A. - Los Angeles Times maps.latimes.com. https://maps.latimes.com/neighborhoods/index.html. [Accessed 28-04-2025]. ↗
  69. Sofie van Haperen, Justus Uitermark, and Walter Nicholls. 2022. The Swarm versus the Grassroots: places and networks of supporters and opponents of Black Lives Matter on Twitter. Social Movement Studies 22, 2 (2022), 171–189. doi:10.1080/14742837.2022.2031954 ↗
  70. William R Van Riper. 1986. General American: an ambiguity. In Dialect and language variation. Elsevier, 123–135. Reinforcing Stereotypes of Anger: Emotion AI on African American Vernacular English • ↗
  71. J Celeste Walley-Jean. 2009. Debunking the myth of the “angry Black woman”: An exploration of anger in young African American women. Black Women, Gender & Families 3, 2 (2009), 68–86. find
  72. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837.
  73. Nicole H Weiss, Emmanuel D Thomas, Melissa R Schick, Miranda E Reyes, and Ateka A Contractor. 2022. Racial and ethnic differences in emotion regulation: A systematic review. Journal of clinical psychology 78, 5 (2022), 785–808. ↗
  74. Virginia B. Wickline, Wendy Bailey, and Stephen Nowicki. 2009. Cultural In-Group Advantage: Emotion Recognition in African American and European American Faces and Voices. The Journal of Genetic Psychology 170, 1 (March 2009), 5–30. doi:10.3200/GNTP.170.1.5-30 ↗
  75. Tyia K. Wilson and Amy L. Gentzler. 2021. Emotion regulation and coping with racial stressors among African Americans across the lifespan. Developmental Review 61 (Sept. 2021), 100967. doi:10.1016/j.dr.2021.100967 ↗
  76. Adia Harvey Wingfield. 2010. Are Some Emotions Marked “Whites Only”? Racialized Feeling Rules in Professional Workplaces. Social Problems 57, 2 (May 2010), 251–268. doi:10.1525/sp.2010.57.2.251 ↗
  77. Samira Zad, Maryam Heidari, James H Jr Jones, and Ozlem Uzuner. 2021. Emotion Detection of Textual Data: An Interdisciplinary Survey. IEEE, Seattle, WA, USA, 0255–0261. doi:10.1109/AIIoT52608.2021.9454192 ↗
  78. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. VALUE: Understanding Dialect Disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 3701–3720. doi:10.18653/v1/2022.acl-long.258 Joy Love Sadness Anger Disgust Fear Surprise Overall Model ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dorn-2025-reinforcing-stereotypes-anger-emotion-ai/