Elamur Research⌕
☰
home / papers / 2026 / Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles

Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles

Paper · 2026 · Irti Haq and Belen Saldias

Summary

LLMs are known to treat users worse when they state membership in a vulnerable group. This paper asks whether the trigger is the stated identity or the way identity shows up in language. The authors ran a 2×3×4 factorial experiment. Task content was held fixed while two cues were varied independently: an explicit self-description (Black, Singaporean, White) and a dialect instruction (AAVE, Singlish, Standard English). They used 2,219 BOLD prompts across four sensitive domains, giving over 24,000 responses from Gemma-3-12B and Qwen-3-VL-8B. Responses were scored for refusal, BERTScore against Wikipedia-derived references, and Relative Negative Regard. Mixed-effects models then separated the effect of each cue from prompt difficulty. The results point opposite ways. Explicit labels, especially for Black users, made Gemma refuse more and lowered similarity to the references. Dialect cues pushed refusals close to zero and slightly raised similarity, but they also yielded less sanitized, more negative content, particularly on politics and religion. The authors read this as evidence that safety alignment keys on explicit words and generalizes poorly to implicit signals.

Key points

  • Design: 2 models × 3 identities × 4 domains (Gender, Race, Religion, Politics), with 2,219 BOLD prompts and over 24,000 responses. Analysis used generalized linear mixed-effects models with prompt text as a random effect.
  • Refusal detection: a keyword heuristic was checked by hand on 504 responses, with precision 1.000 and recall 0.985.
  • Refusal analysis covers Gemma-3-12B only: Qwen-3-VL-8B almost never refused. The authors treat Qwen as a sign that these disparities come from a particular safety regimen rather than from LLMs in general.
  • Refusals depend heavily on content: the prompt random-effect σ was 10.02 for refusal, against 0.00 for BERTScore and 0.03 for regard. Refused outputs were not scored further, which creates survivorship bias.
  • Limitations acknowledged:
    • Dialect text was injected synthetically, not written by native speakers.
    • Prompting was one-shot only.
    • Gender (male) and occupation were held constant.
    • Quality was judged by automated proxies.

Interesting ideas

  • I1. Keep the content of a request fixed and toggle the "who says it" signal and the "how it is said" signal as separate factors, so any change in output can be traced to one cue
  • I2. Distinguish identity that is declared outright from identity that can only be inferred from style; systems may respond to the first and miss the second entirely
  • I3. Score each output against several references and keep the best match, since a short prompt can have many valid continuations
  • I4. Before trusting a cheap rule-based detector, audit a few hundred cases by hand and report precision and recall
  • I5. Watch for survivorship bias: if failed or refused items are dropped before scoring, the hardest items vanish and quality looks better than it is

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): build paired evaluation sets where only the register or dialect of a sentence changes, as a negative control. The frames and answers produced should stay identical, and any drift points to a brittle stage.
  • I2 (frames): prag could record speaker-signaled attributes (explicit self-description versus stylistic cues) as separate, traceable evidence. Later steps would then cite which signal they relied on.
  • I3 (answer): when grading Mova's free-text answers on fables or school tests, compare against a set of acceptable answers rather than one gold string. This avoids penalizing valid alternatives.
  • I4 (learn): self-induced repair rules and silver-data filters could each carry a small, manually checked audit with precision and recall attached, so their reliability is explicit.
  • I5 (learn): log items where a parser or reader gave up and report them alongside the scores. This keeps difficult sentences from silently leaving the evaluation pool.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Irti Haq and Belen Saldias. 2026. Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles. arXiv:2604.21152.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2604.21152 · arXiv 2604.21152

@misc{haq2026dialect,
  title = {{Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles}},
  author = {Irti Haq and Belen Saldias},
  year = {2026},
  doi = {10.48550/arxiv.2604.21152},
  eprint = {2604.21152},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.21152}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717 [cs.LG] https://arxiv.org/abs/2406.11717 ↗
  2. Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. 2024. Measuring implicit bias in explicitly unbiased large language models. arXiv preprint arXiv:2402.04105 (2024). ↗
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862 [cs.CL] https://arxiv.org/abs/2204.05862 ↗
  4. Ari Ball-Burack, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2021. Differential Tweetment: Mitigating Racial Dialect Bias in Harmful Tweet Detection. In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, Madeleine Clare Elish, William Isaac, and Richard S. Zemel (Eds.). ACM, 116–128. doi:10.1145/3442188.3445875 ↗
  5. A. Bavaresco, R. Bernardi, L. Bertolazzi, D. Elliott, R. Fernández, A. Gatt, E. Ghaleb, M. Giulianelli, M. Hanna, A. Koller, A. F. T. Martins, P. Mondorf, V. Neplenbroek, S. Pezzelle, B. Plank, D. Schlangen, A. Suglia, A. K. Surikuchi, E. Takmaz, and A. Testoni. 2025. LLMs instead of human judges? A large-scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 238–255. doi:10.18653/v1/2025.acl-short.20 ↗
  6. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency. ACM. doi:10.1145/ 3442188.3445922 find
  7. J. M. Cunningham. 2020. African American language is not good English. https://human.libretexts.org/Bookshelves/Composition/ Specialized_Composition/Book:_Bad_Ideas_About_Writing_(Ball_and_Loewe)/02:_Bad_Ideas_About_Who_Good_Writers_are/2.07: _African_American_Language_is_not_Good_English ↗
  8. Nicholas Deas, Jessica Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. 2023. Evaluation of African American language bias in natural language generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 6805–6824. find
  9. Nicholas Deas, Blake Vente, Amith Ananthram, Jessica A Grieser, Desmond U Patton, Shana Kleiner, James R Shepard Iii, and Kathleen McKeown. 2025. Data caricatures: on the representation of African American language in pretraining corpora. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 29192–29217. ↗
  10. J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y. Pruksachatkun, K.-W. Chang, and R. Gupta. 2021. BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency. 862–872. doi:10.1145/3442188.3445924 ↗
  11. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv:2209.07858 [cs.CL] https://arxiv.org/abs/2209.07858 ↗
  12. Lisa J Green. 2002. African American English: a linguistic introduction. Cambridge University Press. ↗
  13. A. F. Gupta. 1994. The step-tongue: Children’s English in Singapore. Multilingual Matters. Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles FAccT ’26, June 25–28, 2026, Montreal, QC, Canada find
  14. Camille Harris, Matan Halevy, Ayanna M. Howard, Amy S. Bruckman, and Diyi Yang. 2022. Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification. In FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 - 24, 2022. ACM, 789–798. doi:10.1145/3531146.3533144 ↗
  15. Anthony Harvey and Akash Kumar Karmaker. 2025. A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms. In FAccT ’25: 2025 ACM Conference on Fairness, Accountability, and Transparency. ACM. doi:10.1145/3715275.3732137 ↗
  16. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. AI generates covertly racist decisions about people based on their dialect. Nature 633, 8028 (2024), 147–154. ↗
  17. Nina Markl. 2022. Language variation and algorithmic bias: understanding algorithmic bias in British English automatic speech recognition. In FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 - 24, 2022. ACM, 521–534. doi:10.1145/3531146.3533117 ↗
  18. Kaisen Mei, Shirin Fereidooni, and Aylin Caliskan. 2023. Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks. In FAccT ’23: 2023 ACM Conference on Fairness, Accountability, and Transparency. ACM. doi:10.1145/3593013.3594109 ↗
  19. Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Maarten Sap. 2025. Rejected Dialects: Biases Against African American Language in Reward Models. In Findings of the Association for Computational Linguistics: NAACL 2025, Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 7483–7502. doi:10.18653/v1/2025.findings-naacl.417 ↗
  20. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774 ↗
  21. Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. 2026. Llm targeted underperformance disproportionately impacts vulnerable users. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 39116–39124. find
  22. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Kevin Duh, Helena Gomez, and Steven FAccT ’26, June 25–28, 2026, Montreal, QC, Canada Irti Haq and Belén Saldías Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 5377–5400. doi:10.18653/v1/2024.naacl-long.301 ↗
  23. E. Sheng, K.-W. Chang, P. Natarajan, and N. Peng. 2019. The woman worked as a babysitter: On biases in language generation. arXiv Preprint (2019). doi:10.48550/arXiv.1909.01326 ↗
  24. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs. arXiv:2308.13387 [cs.CL] https://arxiv.org/abs/2308.13387 ↗
  25. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of Risks posed by Language Models. In FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 - 24, 2022. ACM, 214–229. doi:10.1145/3531146.3533088 ↗
  26. J. Wong. 2005. “Why you so Singlish one?” A semantic and cultural interpretation of the Singapore English particle one. Language in Society 34, 2 (2005), 239–275. doi:10.1017/S0047404505050104 ↗
  27. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haq-2026-dialect-vs-demographics-quantifying-llm/