Elamur Research⌕
☰
home / papers / 2026 / LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering

LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering

Paper · 2026 · Huan Wu, Ali Emami, Muhammad Furquan Hassan et al.

Summary

Instruction-tuned LLMs tend to treat African American English (AAE) as an error and steer users' text toward Standard American English (SAE), even when the user writes in AAE. The authors build REAL-AAE from TwitterAAE tweets. Each tweet is translated to SAE with gemini-3-flash-preview and then back to AAE, giving AAE/SAE/AAE_back triplets. Native speakers and automatic metrics check the triplets. They audit six open-weight models (14B–70B) in two ways. The first is discriminative consistency, using a new metric, conditional Dialect Group Invariance (cDGI). The second is generative preference, measured with perplexity, log-probability bias and multiple-choice continuations. A feature-level analysis then finds which AAE markers trigger the bias. Syntactic features, especially negative concord, trigger it most strongly. For mitigation, they use training-free activation steering. Causal tracing picks which layers to steer and how strongly. Steering beats prompting, context-aware decoding (CAD) and LoRA while mostly keeping SAE behaviour intact. This matters because silent "correction" erases speakers' voices in tools that edit or continue their text.

Key points

  • The corpus. REAL-AAE has 17,479 triplets, 3.5× larger than the previous largest real-AAE parallel resource.
    • AAE vs AAE_back BERTScore F1 is 0.95, computed with Twitter-RoBERTa.
    • Three native AAE speakers report 91.3% semantic agreement.
    • Only 16 of 17,495 translation attempts failed.
  • cDGI. The metric measures AAE/SAE label agreement only on two kinds of examples:
    • examples whose predicted label survives back-translation;
    • examples whose AAE_back annotators judged natural (578 such test sentences).
  • Models. The audited models are Phi-4, Phi-3, Mistral, Gemma-3, DeepSeek-R1 and Llama-3.1-70B; all six prefer SAE continuations under AAE context.
  • Steering results.
    • AAE Match rises by 18–24 points (e.g. Phi-4 goes from 39.3% to 58.3%), and SAE Match changes by no more than 3 points.
    • Log-probability bias falls by 7–21%, and DGI rises by 0.04–0.14.
    • MMLU and GSM8K accuracy moves by at most 2.5 points.
    • Bias is reduced, not removed: AAE Match after steering is still only 50–66%.
  • Human evaluation. On 200 contexts, native speakers judged the steered continuations a better dialect match than the base ones (73.0% vs 9.8%). Steered outputs were also flagged as exaggerated far less often (5.1% vs 35.2%).

Interesting ideas

  • I1. An invariance score is computed only on items that stay stable under round-trip translation, so translator noise is not blamed on the model; this suits any comparison done through an automatic converter.
  • I2. The round trip original → standard → back-translated original works as a cheap content-preservation check without needing a gold reference.
  • I3. Bias is broken down by linguistic feature class instead of reported as one number, which revealed that syntax (negative concord) dominates.
  • I4. Because negative concord is where models most often "correct" the dialect, it is worth checking that dialectal multiple negation is not misread as a change of meaning.
  • I5. The mitigation is judged on three things at once: the target gain (AAE Match), side-effect monitors (SAE perplexity and SAE Match should stay near baseline), and unrelated-capability checks (MMLU/GSM8K).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When comparing parses of dialect text with its standardized version, count only pairs whose round-trip conversion is stable, as a negative control against blaming the converter.
  • I2 (back): The text → graph → text compressor could score itself by re-parsing its output and comparing the second graph with the first, flagging lossy constructions.
  • I3 (tree): Group parser and UD dialect-conversion errors by marker type (e.g. "ain't nobody", multiple negation) to see which constructions need dedicated conversion rules.
  • I4 (frames): Add test cases where dialectal negative concord must yield the same frame polarity as the standard paraphrase, so who did (not do) what stays correct.
  • I5 (learn): Every self-induced repair rule could be accepted only if it improves the target slice while standard-English parsing and downstream answer tests stay within a fixed tolerance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🎓Emory UniversityAtlanta · United States36 works · 39,779 citations🎓York UniversityToronto · Canada13 works · 3,075 citations🎓University of GuelphGuelph · Canada4 works · 266 citations

Also on the byline: Wilfrid Laurier University.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as Huan Wu, Ali Emami, Muhammad Furquan Hassan, Osaretin Igbinoba, Osakpolor Idusuyi, Osamede Igbinoba, Faiza Khan Khattak and Laleh Seyyed-Kalantari. 2026. LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering. arXiv:2607.06845.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.06845 · arXiv 2607.06845

@misc{wu2026llms,
  title = {{LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering}},
  author = {Huan Wu, Ali Emami, Muhammad Furquan Hassan, Osaretin Igbinoba, Osakpolor Idusuyi, Osamede Igbinoba, Faiza Khan Khattak and Laleh Seyyed-Kalantari},
  year = {2026},
  doi = {10.48550/arxiv.2607.06845},
  eprint = {2607.06845},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.06845}
}

References

47 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin et al. 2024a. Phi-3 technical report: A highly capable language model locally on your phone. Technical Report MSR-TR-2024-12, Microsoft Research. ↗
  2. Marah Abdin et al. 2024b. Phi-4 technical report. Technical Report MSR-TR-2024-57, Microsoft Research. ↗
  3. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems, volume 37, pages 136037– 136083.
  4. Jeffrey Basoah, Jay L. Cunningham, Erica Adams, Alisha Bose, Aditi Jain, Kaustubh Yadav, Zhengyang Yang, Katharina Reinecke, and Daniela Rosner. 2025. Should AI mimic people? Understanding AI-supported writing technology among Black users. Proceedings of the ACM on Human-Computer Interaction, 9(7):1–51. find
  5. Su Lin Blodgett. 2021. Sociolinguistically driven approaches for just natural language processing. Ph.D. thesis, University of Massachusetts Amherst. ↗
  6. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computational Linguistics. ↗
  7. Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. Persona vectors: Monitoring and controlling character traits in language models. Preprint, arXiv:2507.21509. ↗
  8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. ↗
  9. Jamell Dacon. 2022. Towards a deep multi-layered dialectal language analysis: A case study of African-American English. In Proceedings of the Second Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pages 55–63, Seattle, Washington. Association for Computational Linguistics.
  10. Jamell Dacon, Haochen Liu, and Jiliang Tang. 2022. Evaluating and mitigating inherent linguistic bias of African American English through inference. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1442–1454, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  11. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics. ↗
  12. Nicholas Deas, Jessica Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. 2023. Evaluation of African American language bias in natural language generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6805– 6824, Singapore. Association for Computational Linguistics. find
  13. Kareem Elozeiri, Mervat Abassy, Omar Kallas, Fahim Dalvi, Preslav Nakov, Kentaro Inui, and Nadir Durrani. 2026. Can dialects be steered like languages? Sparse neurons and distributed directions in Arabic LLMs. Preprint, arXiv:2607.03936. ↗
  14. Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, and Dan Klein. 2024. Linguistic bias in ChatGPT: Language models reinforce dialect discrimination. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13541–13564, Miami, Florida, USA. Association for Computational Linguistics.
  15. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. ↗
  16. Gemma Team. 2025. Gemma 3 technical report. Technical report, Google DeepMind. ↗
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  18. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American Vernacular English in transformer-based text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5877–5883, Online. Association for Computational Linguistics. ↗
  19. Daya Guo, Dejian Yang, Haowei Zhang, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633– 638. ↗
  20. Abhay Gupta, Jacob Cheung, Philip Meng, Shayan Sayyed, Kevin Zhu, Austen Liao, and Sean O’Brien. 2025. EnDive: A cross-dialect benchmark for fairness and performance in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16830–16855, Suzhou, China. Association for Computational Linguistics. ↗
  21. Abhay Gupta, Ece Yurtseven, Philip Meng, and Kevin Zhu. 2024. AAVENUE: Detecting LLM biases on NLU tasks in AAVE via a novel benchmark. In Proceedings of the Third Workshop on NLP for Positive Impact, pages 327–333, Miami, Florida, USA. Association for Computational Linguistics.
  22. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. find
  23. Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the role of grammar and word choice in bias toward African American English (AAE) in hate speech classification. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 789–798. find
  24. Muhammad Furquan Hassan, Faiza Khan Khattak, and Laleh Seyyed-Kalantari. 2025. Dialectic preference bias in large language models. Proceedings of the AAAI Symposium Series, 5(1):365–369. find
  25. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. ↗
  26. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. AI generates covertly racist decisions about people based on their dialect. Nature, 633(8028):147–154. ↗
  27. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  28. Ðord̄e Klisura, Astrid R Bernaga Torres, Anna Karen Gárate-Escamilla, Rajesh Roshan Biswal, Ke Yang, Hilal Pataci, and Anthony Rios. 2025. A multi-agent framework for mitigating dialect biases in privacy policy question-answering systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32318–32337, Vienna, Austria. Association for Computational Linguistics. ↗
  29. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. ↗
  30. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. Association for Computing Machinery.
  31. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, and Furu Wei. 2025. Assessing dialect fairness and robustness of large language models in reasoning tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6317–6342, Vienna, Austria. Association for Computational Linguistics. ↗
  32. Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. 2024. In-context vectors: Making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 32287–32307. PMLR. ↗
  33. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, volume 35. ↗
  34. Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Maarten Sap. 2025. Rejected dialects: Biases against African American language in reward models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7468–7487, Albuquerque, New Mexico. Association for Computational Linguistics. find
  35. Mistral AI. 2025. Mistral Small 3.1. Model release. find
  36. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering Llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thailand. Association for Computational Linguistics. ↗
  37. Sandra Camille Sandoval, Christabel Acquaye, Kwesi Adu Cobbina, Mohammad Nayeem Teli, and Hal Daumé III. 2025. My LLM might mimic AAE - but when should it? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5277–5302, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  38. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. ↗
  39. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884–5906, Seattle, United States. Association for Computational Linguistics. ↗
  40. Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 783–791, Mexico City, Mexico. Association for Computational Linguistics. ↗
  41. Dipankar Srirag, Aditya Joshi, and Jacob Eisenstein. 2025. Predicting the target word of game-playing conversations using a low-rank dialect adapter for decoder models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 8–17, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  42. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. Preprint, arXiv:2308.10248. ↗
  43. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics.
  44. Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang, and Nanyun Peng. 2025. DialectGen: Benchmarking and improving dialect robustness in multimodal generation. Preprint, arXiv:2510.14949. ↗
  45. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics. ↗
  46. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A framework for cross-dialectal English NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.
  47. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to AI transparency. Preprint, arXiv:2310.01405. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-llms-silently-correct-african-american/