Elamur Research⌕
☰
home / papers / 2025 / Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

Paper · 2025 · Eileen Pan, Anna Seo Gyeong Choi, Maartje Ter Hoeve et al.

Summary

The paper asks whether LLMs answer simple knowledge and reasoning questions less accurately when the questions are written in non-"standard" English dialects rather than Standard American English (SAE), and which grammatical features cause the gap. The authors take three multiple-choice benchmarks, BoolQ, SciQ and MMLU. They rewrite only the question texts into six dialects with the Multi-VALUE package, which is built on the eWAVE atlas: African American, Appalachian, Chicano, Indian, Singaporean and Southern English. They then test Gemma-2B, Mistral 7B and GPT4o-mini zero-shot through LM Eval Harness.

Accuracy falls for almost every dialect, model and dataset. The drop is much larger when the analysis is limited to questions the model already got right in SAE, reaching roughly 20 points. Applying single grammar rules separately shows that three rules (existential "it", zero copula and "y'all") explain most of the loss. This matters because it points to cheap, targeted fixes that could help many dialects at once.

Key points

  • Datasets: BoolQ (9,427 questions), SciQ (11,679) and MMLU (14,042). Answers and support passages were left in SAE, and questions Multi-VALUE could not process were excluded.
  • Dialect variants used Multi-VALUE's default rules and frequencies, while single-rule tests forced each rule to fire every time it could apply.
  • Among questions answered correctly in SAE, the worst case was Gemma-2B on Singaporean English MMLU, at 78.4% (−21.6 points). Singaporean English was generally hit hardest.
    • Its questions also showed the largest perplexity increase under a FineWeb model.
  • Of 20 grammar rules with statistically significant effects, 9 were shared across dialects.
  • Existential it, drop copula be NP and y'all each account for at least 45% of a dialect's degradation. For the American dialects (Appalachian, African American, Southern) the share is 64–85%.

Interesting ideas

  • I1. Measure the dialect gap only on items the model already solves in the standard form, so that general weakness is separated from a true dialect penalty.
  • I2. Isolate one grammar rule at full application rate, then stack the remaining dialect rules on top to see what share of the total loss each rule explains.
  • I3. Perturb only the input question while holding answers and context fixed, so every comparison is a clean paired test.
  • I4. Use the perplexity of the rewritten text under a reference LM as a cheap early signal of which variety will be hardest.
  • I5. Because a few high-impact rules are shared by several dialects, augmenting training with just those rules may improve many varieties at once.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Our evaluations could report robustness only on sentences Mova handles correctly in canonical form, which gives a cleaner negative-control-style baseline.
  • I2 (tree): We could perturb treebank sentences one rule at a time (zero copula, existential it) and measure the LAS drop per rule, showing exactly which constructions break the perceptron parser or gparse.
  • I3 (answer): School-test and fable-QA readers could be stress-tested with dialect-rewritten questions over unchanged stories, checking whether frame extraction still reaches the same answer.
  • I4 (learn): Perplexity shifts could rank which perturbed sentences to send to LLM teacher tiers or witness parsers first.
  • I5 (tree): We could add a small set of rule-based transformations to the UD dialect conversion and silver data, with trees adjusted accordingly (e.g., a copula-less predicate), instead of trying to model every dialect.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States120 works · 59,940 citations🏢AppleCupertino · United States20 works · 2,146 citations🎓Cornell TechNew York · United States7 works · 1,320 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Eileen Pan, Anna Seo Gyeong Choi, Maartje Ter Hoeve, Skyler Seto and Allison Koenecke. 2025. Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks. Findings of the Association for Computational Linguistics: EMNLP 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-emnlp.1139

@inproceedings{pan2025analyzing,
  title = {{Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks}},
  author = {Eileen Pan, Anna Seo Gyeong Choi, Maartje {Ter Hoeve}, Skyler Seto and Allison Koenecke},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.1139},
  url = {https://doi.org/10.18653/v1/2025.findings-emnlp.1139}
}

References

33 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tanel Alumäe and Allison Koenecke. 2025. Striving for open-source and equitable speech-to-speech translation. ↗
  2. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic Dialectal Variation in Social Media: A Case Study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 1119–1130, Austin, Texas. Association for Computational Linguistics.
  3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044. ↗
  4. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. A framework for few-shot language model evaluation. find
  5. David Gil. 2003. English goes asian: Number and (in) definiteness in the singlish noun phrase. In Noun phrase structure in the languages of Europe, pages 467–514. Mouton de Gruyter. ↗
  6. Emma Harvey, Rene Kizilcec, and Allison Koenecke. 2025. A framework for auditing chatbots for dialect-based quality-of-service harms. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’25, New York, NY, USA. Association for Computing Machinery. find
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  8. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. Ai generates covertly racist decisions about people based on their dialect. Nature, 633(8028):147–154. ↗
  9. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55.
  10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  11. David Johnson and Lewis VanBrackle. 2012. Linguistic discrimination in writing assessment: How raters react to african american “errors,” esl errors, and standard english errors on a state-mandated writing exam. Assessing Writing, 17(1):35–54. ↗
  12. Taylor Jones, Jessica Rose Kalbfeld, Ryan Hancock, and Robin Clark. 2019. Testifying while black: An experimental study of court reporter accuracy in transcription of african american english. Language, 95(2):e216–e252. ↗
  13. Allison Koenecke, Anna Seo Gyeong Choi, Katelyn X Mei, Hilke Schellmann, and Mona Sloane. 2024. Careless whisper: Speech-to-text hallucination harms. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1672–1681. find
  14. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the national academy of sciences, 117(14):7684–7689. ↗
  15. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. 2020. eWAVE. find
  16. Jakob RE Leimgruber. 2013. The trouble with world englishes: Rethinking the concept of ‘geographical varieties’ of english. English Today, 29(3):3–7. find
  17. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, and Furu Wei. 2025. Assessing dialect fairness and robustness of large language models in reasoning tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6317–6342, Vienna, Austria. Association for Computational Linguistics. ↗
  18. Yanchen Liu, William Held, and Diyi Yang. 2023. Dada: Dialect adaptation via dynamic aggregation of linguistic rules. arXiv preprint arXiv:2305.13406. ↗
  19. Hanjia Lyu, Jiebo Luo, Jian Kang, and Allison Koenecke. 2025. Characterizing bias: Benchmarking large language models in simplified versus traditional chinese. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 2815–2846. find
  20. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380. ↗
  21. Joshua L Martin and Kevin Tang. 2020. Understanding racial disparities in automatic speech recognition: The case of habitual" be". In Interspeech, pages 626–630. find
  22. OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. find
  23. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  24. Lee Rainie. 2025. Close encounters of the AI kind: Main report. find
  25. John Russell Rickford and Russell John Rickford. 2007. Spoken soul: The story of black English. Turner Publishing Company. find
  26. Michael J Ryan, William Held, and Diyi Yang. 2024. Unintended impacts of llm alignment on global representation. arXiv preprint arXiv:2402.15018. ↗
  27. G. Smitherman. 1986. Talkin and Testifyin: The Language of Black America. Wayne State University Press. find
  28. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. 2025. Evaluating dialect robustness of language models via conversation understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 24–38, Abu Dhabi. Association for Computational Linguistics. ↗
  29. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, and 89 others. 2024. Gemma: Open models based on gemini research and technology. arXiv. find
  30. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209. ↗
  31. Marcia Farr Whiteman. 2013. Dialect influence in writing. In Writing, pages 153–166. Routledge. find
  32. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022a. Value: Understanding dialect disparity in nlu. arXiv preprint arXiv:2204.03031. ↗
  33. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2022b. Multi-value: A framework for cross-dialectal english nlp. arXiv preprint arXiv:2212.08011. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2025-analyzing-dialectical-biases-llms-knowledge/