Elamur Research⌕
☰
home / papers / 2025 / My LLM might Mimic AAE - But When Should It?

My LLM might Mimic AAE - But When Should It?

Paper · 2025 · Sandra Camille Sandoval, Christabel Acquaye, Kwesi Adu Cobbina et al.

Summary

The paper investigates two questions about large language models and African American English (AAE). First, do Black Americans want generative AI to produce AAE, and in which situations? Second, how authentic is LLM-generated AAE when a model is explicitly asked to produce it? The authors recruited Black American participants through Prolific and ran two studies. A survey (n = 104) asked how a model should behave in scenarios that ranged from professional to personal. An annotation study (n = 228) collected judgments of texts written by GPT 4o-mini, Llama 3 and Mixtral, alongside human reference transcripts taken from the CORAAL corpus. Participants mostly wanted Mainstream U.S. English (MUSE) as the default in formal or task-oriented use. They were more receptive to AAE in casual and personal contexts, and above all they wanted to decide for themselves when the switch happens. When models received suitable prompts and in-context examples, annotators rated their AAE about as authentic as the real CORAAL speech and did not see it as mocking or offensive. The work offers community-grounded guidance for designing dialect-aware generation and releases a dataset of judgments.

Key points

  • Survey scenarios covered professional and personal AI assistants, a customer-service bot, email and SMS autocomplete, and an educational avatar. They were split into "reply" behaviour (the model answers the user) and "continuation" behaviour (the model completes the user's own text).
  • The annotation study produced 8,654 judgments over 1,357 examples. Texts were rated on coherence, explicit presence of AAE features, and offensiveness.
  • Human baseline texts were exchanges from CORAAL, a corpus of transcribed interviews with Black Americans. It was chosen because it is conversational, informal and regionally varied.
  • Survey respondents came from all four major U.S. regions (53 South, 21 Northeast, 17 West, 17 Midwest). Their ages ran from 25 to 64, with a mean of 34. Participants were paid about 2.5× the U.S. federal minimum wage.
  • The related work the paper cites reports commercial ASR error rates of 35% for Black speakers versus 19% for White speakers.

Interesting ideas

  • I1. Genuine human text from CORAAL was mixed into the same annotation pool as model output, so model scores can be compared directly against a real-speech reference rather than read in isolation.
  • I2. The paper treats the question of whether a variety should be used as separate from whether a model can produce it, and asks the affected community about the first question directly.
  • I3. Reply settings (the model speaks for itself) are distinguished from continuation settings (the model writes on the user's behalf), because people hold different expectations for each.
  • I4. Annotators rated several separate axes (coherence, feature presence, offensiveness) instead of giving one overall quality score, which keeps fluency apart from dialect fidelity and from harm.
  • I5. The paper defines AAE through checkable grammatical markers such as double negatives, variable subject-verb agreement and dropped copulas, rather than through vague impressions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation sets could hide gold human sentences among generated or parsed outputs as calibration controls, showing what score "authentic" text actually gets from judges.
  • I2 (back): Generation could keep the choice of register as an explicit, user-controlled parameter that defaults to the mainstream form, rather than letting the model infer dialect implicitly.
  • I3 (back): The text → graph → text path could mark whether output speaks in Mova's own voice or continues the user's text, preserving the user's dialect only in the second case.
  • I4 (answer): Graded answers could likewise be judged on independent axes such as correctness, fluency and appropriateness, so that a failure on one axis is not hidden by success on another.
  • I5 (tree): A dialect detector built on these UD-visible patterns (e.g. a missing copula cop in a predicate clause) could trigger dialect-specific conversion rules before the parser commits to a mainstream analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Sandra Camille Sandoval, Christabel Acquaye, Kwesi Adu Cobbina, Mohammad Nayeem Teli and Hal Daumé III. 2025. My LLM might Mimic AAE - But When Should It?. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.273

@inproceedings{sandoval2025my,
  title = {{My LLM might Mimic AAE - But When Should It?}},
  author = {Sandra Camille Sandoval, Christabel Acquaye, Kwesi Adu Cobbina, Mohammad Nayeem Teli and Hal Daumé III},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.273},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.273}
}

References

37 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. April Baker-Bell. 2020. Dismantling anti-Black linguistic racism in English language arts classrooms: Toward an anti-racist Black language pedagogy. Theory Into Practice, 59(1):8–21. ↗
  3. K. Bashir-Ali. 2006. Language learning and the definition of one’s social, cultural and racial identity. TESOL Quarterly, 40(3):628–639. ↗
  4. John Baugh. 2005. Linguistic profiling. In Black linguistics, pages 167–180. Routledge. find
  5. Kara Becker. 2013. Ethnolect, dialect, and linguistic repertoire in New York City. Manuscript Submitted for Publication. In Malcah Yaeger-Dror and Lauren Hall-Lew (eds.), New Perspectives on the Concept of Ethnolect: Publication of the American Dialect Society (Pads) Durham: Duke University Press. find
  6. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. arXiv preprint arXiv:1608.08868. ↗
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  8. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating LLMs by human preference. Preprint, arXiv:2403.04132. ↗
  9. Jay L. Cunningham, Su Lin Blodgett, Hal Daumé III, Christina Harrington, Hanna Wallach, and Michael Madaio. 2024. Understanding the impacts of language technologies’ performance disparities on African American language speakers. In Proceedings of the Conference on Language Technologies, Location of the Conference. University of Washington; Microsoft Research; Carnegie Mellon University; Google Research. find
  10. Christina N. Harrington, Radhika Garg, Amanda Woodward, and Dimitri Williams. 2022. “It’s Kind of Like Code-Switching”: Black Older Adults’ Experiences with a Voice Assistant for Health Information Seeking. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. find
  11. Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the role of grammar and word choice in bias toward African American English (AAE) in hate speech classification. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 789–798. find
  12. Jazmia Henry. 2021. Aave corpora. Accessed: 2024-06-01. find
  13. Jane H Hill. 1998. Language, race, and white public space. American anthropologist, 100(3):680–689. ↗
  14. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. Dialect prejudice predicts AI decisions about people’s character, employability, and criminality. Preprint, arXiv:2403.00742. ↗
  15. Yolanda Holt. 2018. Mechanisms of Vowel Variation in African American English. Journal of Speech Language and Hearing Research, 61:1. ↗
  16. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts. Preprint, arXiv:2401.04088. ↗
  17. Tyler Kendall and Charlie Farrington. 2023. The Corpus of Regional African American Language. ↗
  18. Sharese King. 2020. From African American Vernacular English to African American Language: Rethinking the study of race and language in African Americans’ Speech. Annual Review of Linguistics, 6:285–300. ↗
  19. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, et al. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. ↗
  20. Sonja L. Lanehart, Jennifer Bloomquist, and Ayesha M. Malik. 2015. Language Use in African American Communities: An Introduction. In The Oxford Handbook of African American Language. Oxford University Press. ↗
  21. Latimer. 2023. LATIMER’S BLACK GPT. find
  22. Timo Lenzner. 2012. Effects of survey question comprehensibility on response quality. Field Methods, 24(4):409–428. ↗
  23. A. Maedche, Christine Legner, Alexander Benlian, Benedikt Berger, Henner Gimpel, T. Hess, O. Hinz, Stefan Morana, and Matthias Söllner. 2019. AI-Based Digital Assistants. Business & Information Systems Engineering, 61:535–544. ↗
  24. Wilermine Previlon, Alice Rozet, Jotsna Gowda, Bill Dyer, Kevin Tang, and Sarah Moeller. 2024. Leveraging syntactic dependencies in disambiguation: The case of African American English. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10403–10415, Torino, Italia. ELRA and ICCL.
  25. John R Rickford, Greg J Duncan, Lisa A Gennetian, Ray Yun Gou, Rebecca Greene, Lawrence F Katz, Ronald C Kessler, Jeffrey R Kling, Lisa Sanbon-matsu, Andres E Sanchez-Ordoñez, et al. 2015. Neighborhood effects on use of African-American Vernacular English. Proceedings of the National Academy of Sciences, 112(38):11817–11822. ↗
  26. John R Rickford and Sharese King. 2016. Language and linguistics on trial: Hearing Rachel Jeantel (and other vernacular speakers) in the courtroom and beyond. Language, pages 948–988. ↗
  27. Harrison Santiago, Joshua Martin, Sarah Moeller, and Kevin Tang. 2022. Disambiguation of morpho-syntactic features of African American English–the case of habitual be. arXiv preprint arXiv:2204.12421. ↗
  28. Hanna L Smokoski. 2016. Voicing the other: mock aave on social media. find
  29. Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. 2020a. Large-scale modeling of media dialog with discourse patterns and knowledge grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). *Equal Contribution. find
  30. U.S. Census Bureau. 2022. Quickfacts: United states. find
  31. Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. 2020b. A large-scale open-source corpus of media dialog. CoRR, abs/2004.03090. *Equal Contribution. ↗
  32. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372. ↗
  33. Walt Wolfram and Natalie Schilling. 2015. American English: dialects and variation. John Wiley & Sons. find
  34. Elijah Mayfield, Michael Madaio, Shrimai Prabhumoye, David Gerritsen, Brittany McLaughlin, Ezekiel Dixon-Román, and Alan W Black. 2019. Equity beyond bias in language technologies for education. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 444–460, Florence, Italy. Association for Computational Linguistics. find
  35. M. Moslimani, C. Tamir, A. Budiman, L. Noe-Bustamante, and L. Mora. 2023. Facts About the U.S. Black Population. find
  36. Camille O’Quin. 2021. Exploring African American Vernacular English and Disproportionality in Special Education. Phd thesis, Illinois State University. ↗
  37. Claudio Santos Pinhanez, Raul Fernandez, Marcelo Carpinette Grave, Julio Nogima, and Ron Hoory. 2024. Creating an African American-Sounding TTS: Guidelines, Technical Challenges, and Surprising Evaluations. In Proceedings of the 29th International Conference on Intelligent User Interfaces, IUI ’24, page 259–273, New York, NY, USA. Association for Computing Machinery. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sandoval-2025-my-llm-might-mimic-aae/