Elamur Research⌕
☰
home / papers / 2026 / Evaluating the Usage of African-American Vernacular English in Large Language Models

Evaluating the Usage of African-American Vernacular English in Large Language Models

Paper · 2026 · Deja Dunlap and R. Thomas McCoy

Summary

LLMs are mostly tested on Standard American English, so it is unclear whether they can faithfully produce minoritized dialects. This work measures how three LLMs (Llama-3-8B-Instruct-Turbo, GPT-4o-mini, Gemma-2-27B-IT) handle African American Vernacular English (AAVE), treating AAVE as a linguistic system in its own right rather than as a deviation from SAE. The human baseline comes from two sources: sociolinguistic interviews in CORAAL and a sample of TwitterAAE. The models were then prompted to imitate each genre, producing simulated interviews and batches of tweets. The authors compared how often seven AAVE grammatical features occur, and in which contexts, in human versus model text. The comparison relies on automatic analysis, not human annotators, so it scales and can be repeated. The models generally use these features too rarely or in the wrong places. Many context patterns typical of native speakers barely appear in model output. Manual reading and sentiment analysis also showed stereotyped themes in the generated interviews. The framework is designed to transfer to other dialects.

Key points

  • The human data are 272 CORAAL interviews (109,234 sentences after cleaning) and a random 110,000-tweet subset of the 1.1M-tweet TwitterAAE.
  • The model data are 1,000 generated interviews per model and 750 batches of 250 tweets (about 187,500 tweets). Interview prompts drew speaker gender and city in proportion to their shares in CORAAL.
  • The examined constructions cover ain't, habitual be, perfective done, null copula, double comparatives, stacked (multiple) modals and negative concord.
  • Llama underused six of these features compared with humans. For example, it used habitual be at only about 14% of the human rate.
  • In a manual review of 10 interviews and 250 tweets per model, the tweets were mostly unproblematic. The interviews kept returning to single-parent homes, economic hardship, "the streets" and violence.
  • The authors suggest that safety-oriented post-training may push models away from nonstandard grammar, which could indirectly erase legitimate dialect patterns.

Interesting ideas

  • I1. Instead of asking whether a model obeys a grammar rule, the paper compares how often a construction appears and in which contexts against a human reference corpus, which exposes distributional mismatches that a correctness test would miss.
  • I2. The prompts asked for the dialect without naming any target features, so the measured feature usage reflects the model's own notion of the variety rather than the prompt's wording.
  • I3. Prompt variables such as gender and city were sampled to mirror their proportions in the human corpus, which keeps the human and model collections comparable.
  • I4. The paper treats dialect grammar as a legitimate system and warns that alignment that suppresses "improper" language can erase real patterns.
  • I5. The comparison runs automatically over large corpora without human annotators, making it scalable and reproducible.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Generated text could be profiled by the frequency and context of constructions and compared with human corpora of the same genre, flagging output that is grammatical but statistically unnatural.
  • I2 (learn): When LLM teacher tiers generate silver or test data, prompts should avoid naming the phenomenon being measured, so the evaluation is not contaminated by the request.
  • I3 (learn): Synthetic evaluation sets could sample genre, speaker and topic variables to match a reference corpus, giving an aligned baseline in the spirit of a negative control.
  • I4 (tokens): The tagger and lemmatizer should analyse forms like ain't, habitual be or null copula as valid structures rather than routing them to repair rules as errors.
  • I5 (tree): UD-based detectors could count dialect constructions across treebanks and generated text, for example by checking for a missing cop relation or an invariant be with habitual sense.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yale UniversityNew Haven · United States95 works · 12,765 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Deja Dunlap and R. Thomas McCoy. 2026. Evaluating the Usage of African-American Vernacular English in Large Language Models. arXiv:2602.21485.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2602.21485 · arXiv 2602.21485

@misc{dunlap2026evaluating,
  title = {{Evaluating the Usage of African-American Vernacular English in Large Language Models}},
  author = {Deja Dunlap and R. Thomas McCoy},
  year = {2026},
  doi = {10.48550/arxiv.2602.21485},
  eprint = {2602.21485},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2602.21485}
}

References

50 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nemeroff, A.: What is AI slop? A technologist explains this new and largely unwelcome form of online content. The Conversation. (2025) find
  2. Blodgett, S.L., Green, L., O’Connor, B.: Demographic Dialectal Variation in Social Media: A Case Study of African-American English. Proceedings of EMNLP. (2016)
  3. van Ess, H.: Reporter’s Guide to Detecting AI-Generated Content. Global Investigative Journalism Network (GIJN). (2025) find
  4. Heller, M.: The Commodification of Language. Annual Review of Anthropology 39(1), 101–114 (2010). find
  5. Small, Z.: Black Artists Say A.I. Shows Bias, With Algorithms Erasing Their History. The New York Times. (2023) find
  6. Towns, A.R.: The (racial) biases of communication: Rethinking media and blackness. Social Identities 21(5), 474–488 (2015). find
  7. Lin, F., Mao, S., La Malfa, E., Hofmann, V., de Wynter, A., Wang, X., Chen, S.Q., Wooldridge, M.J., Pierrehumbert, J., Wei, F.: Assessing dialect fairness and robustness of large language models in reasoning tasks. In Proceedings of the ACL. (2025). find
  8. Gupta, A., Yurtseven, E., Meng, P., Zhu, K.: AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark. In Proceedings of the Third Workshop on NLP for Positive Impact. (2024)
  9. Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M.-E., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., Kompatsiaris, I., Kinder-Kurlanda, K., Wagner, C., Karimi, F., Fernandez, M., Alani, H., Berendt, B., Kruegel, T., Heinze, C., Broelemann, K., Kasneci, G., Tiropanis, T., Staab, S.: Bias in data-driven artificial intelligence systems—An introductory survey. WIREs Data Mining and Knowledge Discovery 10(3), e1356 (2020). find
  10. Hofmann, V., Kalluri, P.R., Jurafsky, D., King, S.: AI generates covertly racist decisions about people based on their dialect. Nature 633(8028), 147–154. (2024). find
  11. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners. OpenAI (2019) Evaluating Usage of AAVE in LLMs D. Dunlap, R.T. McCoy find
  12. Fleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., Klein, D.: Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination. In Proceedings of EMNLP. (2024).
  13. Blodgett, S.L., Barocas, S., Daumé III, H., Wallach, H.: Language (Technology) is Power: A critical survey of "bias" in NLP. In ACL. (2020) — Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  14. Rickford, J.R.: African American Vernacular English: Features, evolution, educational implications. Wiley-Blackwell (1999) find
  15. Dacon, J., Liu, H., Tang, J.: Evaluating and Mitigating Inherent Linguistic Bias of African American English through Inference. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1442–1454. (2022)
  16. Dacon, J.: Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English. In Proceedings of the Second Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pp. 55– 63. (2022).
  17. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL, pp. 4171–4186. (2019).
  18. OpenAI: GPT-4 Technical Report. arXiv preprint arXiv:2303.08774. (2023). ↗
  19. Anthropic: Introducing the next generation of Claude. (2024). find
  20. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Singh Koura, P., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M., Subramanian, R., Tan, X.E., Tang, B., Taylor, R., Williams, A., Kuan, J.X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., Scialom, T.: Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288 (2023) ↗
  21. Salinas, A., Haim, A., Nyarko, J.: What’s in a name? Auditing large language models for race and gender bias. arXiv preprint arXiv:2402.14875 (2024) ↗
  22. Zhou, R., Wan, G., Gabriel, S., Li, S., Gates, A.J., Sap, M., Hartvigsen, T.: Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English. arXiv preprint arXiv:2503.04099 (2025) ↗
  23. Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J.R., Jurafsky, D., Goel, S.: Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences 117(14), 7684–7689. (2020). find
  24. Bolukbasi, T., Chang, K.-W., Zou, J.Y., Saligrama, V., Kalai, A.T.: Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in Neural Information Processing Systems 29. (2016).
  25. Caliskan, A., Bryson, J.J., Narayanan, A.: Semantics derived automatically from language corpora contain human-like biases. Science 356(6334), 183–186. (2017). find
  26. Buolamwini, J., Gebru, T.: Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency, pp. 77–91. (2018). find
  27. Nadeem, M., Bethke, A., Reddy, S.: StereoSet: Measuring stereotypical bias in pretrained language models. In ACL, pp. 5356–5371. (2021).
  28. Gallegos, I.O., Rossi, R.A., Barrow, J., Tanjim, M.M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., Ahmed, N.K.: Bias and fairness in large language models: A survey. Computational Linguistics 50(3), 1097–1179. (2024).
  29. An, J., Huang, D., Lin, C., Tai, M.: Measuring gender and racial biases in large language models: Intersectional evidence from automated resume evaluation. PNAS Nexus 4(3), pgaf089. (2025). find
  30. Felkner, V., Chang, H.-C.H., Jang, E., May, J.: WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models. In ACL. (2023). find
  31. Nangia, N., Vania, C., Bhalerao, R., Bowman, S.R.: CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of EMNLP, pp. 1953–1967. (2020).
  32. Blodgett, S.L., O’Connor, B.: Racial disparity in natural language processing: A case study of social media African-American English. arXiv preprint arXiv:1707.00061. (2017). ↗
  33. Gonen, H., Goldberg, Y.: Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. In Proceedings of NAACL, pp. 609–614. (2019).
  34. Lu, K., Mardziel, P., Wu, F., Amancharla, P., Datta, A.: Gender bias in neural natural language processing. In Logic, Language, and Security: Essays Dedicated to Andre Scedrov on the Occasion of his 65th Birthday, pp. 189–202. (2020)
  35. Chen, Y., Raghuram, V.C., Mattern, J., Mihalcea, R., Jin, Z.: Causally testing gender bias in LLMs: A case study on occupational bias. In Findings of NAACL, pp. 4984–5004. (2025). find
  36. Garimella, A., Amarnath, A., Kumar, K., Yalla, A.P., Chhaya, N., Srinivasan, B.V., et al.: He is very intelligent, she is very beautiful? On mitigating social biases in language modelling and generation. In Findings of ACL, pp. 4534–4545. (2021). find
  37. Bardol, F.: ChatGPT Reads Your Tone and Responds Accordingly – Until It Does Not – Emotional Framing Induces Bias in LLM Outputs. arXiv preprint arXiv:2507.21083 (2025) ↗
  38. Linzen, T., Dupoux, E., Goldberg, Y.: Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. TACL 4, 521–535 (2016).
  39. Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., Baroni, M.: Colorless Green Recurrent Networks Dream Hierarchically. In Proceedings of NAACL, pp. 1195– 1205. (2018).
  40. Wilcox, E.G., Futrell, R., Levy, R.: Using computational models to test syntactic learnability. Linguistic Inquiry 55(4), 805–848. (2024). find
  41. Wilcox, E., Levy, R., Morita, T., Futrell, R.: What do RNN Language Models Learn about Filler–Gap Dependencies? In BlackboxNLP, pp. 211–221. (2018). — What do RNN Language Models Learn about Filler-Gap Dependencies?
  42. Marvin, R., Linzen, T.: Targeted Syntactic Evaluation of Language Models. In EMNLP, pp. 1192–1202. (2018). find
  43. Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., Bowman, S.R.: BLiMP: The benchmark of linguistic minimal pairs for English. TACL 8, 377–392. (2020).
  44. Sandler, M., Choung, H., Ross, A., David, P.: A linguistic comparison between human and ChatGPT-generated conversations. In International Conference on Pattern Recognition and Artificial Intelligence, pp. 366–380. (2024) find
  45. Muñoz-Ortiz, A., Gómez-Rodríguez, C., Vilares, D.: Contrasting linguistic patterns in human and LLM-generated news text. Artificial Intelligence Review 57(10), 265. (2024). Evaluating Usage of AAVE in LLMs D. Dunlap, R.T. McCoy find
  46. Kendall, T., Farrington, C.: The Corpus of Regional African American Language. The Online Resources for African American Language Project. (2023). find
  47. Gemma Team: Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. (2024). ↗
  48. Zanuttini, R., Wood, J., Zentz, J., Horn, L.: The Yale Grammatical Diversity Project: Morphosyntactic variation in North American English. Linguistics Vanguard. (2025). find
  49. Green, L.J.: African American English: a Linguistic Introduction. Cambridge University Press. (2002). find
  50. Hutto, C., Gilbert, E.: VADER: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the International AAAI Conference on Web and Social Media. (2014). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dunlap-2026-evaluating-usage-african-american-vernacular/