Elamur Research⌕
☰
home / papers / 2026 / DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English

DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English

Paper · 2026 · Jio Oh, Paul Vicinanza, Thomas Butler et al.

Summary

Most English speakers worldwide do not use Standard American English (SAE). LLMs often misidentify other dialects and produce caricatured text for their speakers. Existing generators lean on eWAVE, which covers only grammar, and apply its features wholesale. The authors present DialectLLM, a pipeline that builds parallel multi-turn dialogs in nine English varieties along three axes: vocabulary, spelling, and morphosyntax.

The pipeline works in four steps:

  • LLMs propose candidate lexical and orthographic mappings, and eWAVE supplies grammatical ones.
  • Native linguists keep the authentic features and rate how common each one is.
  • The linguists also mark which features suit a user's speech but not an assistant's reply.
  • LLMs rewrite SAE seed dialogs using these vetted rules, followed by quality checks.

Human judges strongly preferred the output over a recent eWAVE+LLM method. The resulting benchmark, DialectLLM-Bench, shows that 17 LLMs, including frontier ones, stay below 70% accuracy. They are pulled toward American or British labels. Fine-tuning on the data sharply raises scores, making it a practical resource for dialect-aware assistants.

Key points

  • The dialogs cover nine varieties: US, AU, GB, CA, IN, IE, NG, PH and SC.
    • There are 3,680 parallel dialogs with 1, 2, 4 or 8 turns, in three dataset variants (User, Model, OL).
    • In total this gives 400k+ generated dialogs.
  • Linguists rated features on eWAVE's 1–4 scale, which was turned into application probabilities of 0/30/60/100%.
    • Depending on the dialect, as many as 90% of eWAVE features were judged unfit for model replies.
  • In pairwise naturalness tests, annotators chose DialectLLM over Trans-EnV in 98.8% of cases.
  • DialectLLM-Bench has 50k+ dialogs and 97k+ QA pairs, covering dialect identification and response completion.
    • Canadian English is especially hard: Claude-Opus-4.1 gets it right only 33% of the time.
    • Qwen3-235B gets it right 15% of the time and labels 69% of Canadian dialogs as US.
  • SFT on Qwen3-8B and Gemma3-12B used about 365K samples with seed-disjoint splits.
    • Turn-1 accuracy rose from roughly 30% to about 90%.
    • Training only for response completion also lifted classification by 8–17 pp.
    • Training only for classification lowered response completion by about 10.6 pp.

Interesting ideas

  • I1. Keep two separate feature sets: forms the system must recognize in input, and a much smaller set it should actually produce, so output stays natural rather than imitating every attested variant.
  • I2. Turn graded expert prevalence ratings into sampling probabilities when transforming text, instead of applying a feature always or never; this produces realistic, non-exaggerated variation.
  • I3. Build mapping tables in layers: one LLM proposes candidate correspondences (e.g., spelling and word pairs like colour/color, flat/apartment), a second LLM filters them, and humans give final approval.
  • I4. Guard against memorization by using splits where no seed item is shared between train and test, and by plotting accuracy against the fraction of training data used.
  • I5. Compare models against a trivial biased guesser, such as always answering "British", because ambiguous short inputs can make shallow defaults look competent.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): The generator could emit neutral standard forms while the parser still accepts dialectal grammar, with an explicit "comprehend-only" flag on each variant rule.
  • I2 (tree): When converting treebanks into dialect variants for parser training, each construction could be applied with a frequency-weighted probability, avoiding grotesque training trees.
  • I3 (tokens): A vetted orthographic and lexical variant table would let the lemmatizer normalize British/Irish/Indian spellings and words to a single lemma, without guessing.
  • I4 (learn): Silver-data training runs could split by source lexical seed and report learning curves, as a sanity check alongside negative controls.
  • I5 (answer): School-test and fable QA scores could be reported next to majority-answer baselines, flagging items where the input does not uniquely determine the answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksDialogue systems · 624 works

Cite

Cite as Jio Oh, Paul Vicinanza, Thomas Butler, Steven Euijong Whang, Dezhi Hong and Amani Namboori. 2026. DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English. arXiv:2601.22888.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2601.22888 · arXiv 2601.22888

@misc{oh2026dialectllm,
  title = {{DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English}},
  author = {Jio Oh, Paul Vicinanza, Thomas Butler, Steven Euijong Whang, Dezhi Hong and Amani Namboori},
  year = {2026},
  doi = {10.48550/arxiv.2601.22888},
  eprint = {2601.22888},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.22888}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, and Dan Klein. Linguistic bias in ChatGPT: Language models reinforce dialect discrimination. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13541–13564, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. emnlp-main.750. URL https://aclanthology.org/2024.emnlp-main.750/. ↗
  2. Nicholas Deas, Blake Vente, Amith Ananthram, Jessica A Grieser, Desmond U. Patton, Shana Kleiner, James R. Shepard Iii, and Kathleen McKeown. Data caricatures: On the representation of African American language in pretraining corpora. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29192–29217, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1416. URL https://aclanthology.org/2025.acl-long.1416/. ↗
  3. Jacopo Amidei, Jose Gregorio Ferreira De Sá, Rubén Nieto Luna, and Andreas Kaltenbrunner. Exploring the impact of language switching on personality traits in LLMs. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 2370–2378, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.162/. ↗
  4. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. Multi-VALUE: A framework for cross-dialectal English NLP. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.44. URL https://aclanthology.org/2023.acl-long.44. ↗
  5. William Held, Caleb Ziems, and Diyi Yang. TADA : Task agnostic dialect adapters for English. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 813–824, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.51. URL https://aclanthology.org/2023.findings-acl.51/. ↗
  6. Yanchen Liu, William Held, and Diyi Yang. DADA: Dialect adaptation via dynamic aggregation of linguistic rules. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.850. URL https://aclanthology.org/2023.emnlp-main.850/. ↗
  7. Zedian Xiao, William Held, Yanchen Liu, and Diyi Yang. Task-agnostic low-rank adapters for unseen English dialects. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7857–7870, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.487. URL https://aclanthology.org/2023.emnlp-main.487/. ↗
  8. Faye Holt, William Held, and Diyi Yang. Perceptions of language technology failures from South Asian English speakers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 4067–4081, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-acl.241. URL https://aclanthology.org/2024.findings-acl.241/. ↗
  9. Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. DIALECTBENCH: An NLP benchmark for dialects, varieties, and closely-related languages. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14412–14454, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.777. URL https://aclanthology.org/2024.acl-long.777/. ↗
  10. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, and Furu Wei. Assessing dialect fairness and robustness of large language models in reasoning tasks. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6317–6342, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.317. URL https://aclanthology.org/2025.acl-long.317/. ↗
  11. Abhay Gupta, Jacob Cheung, Philip Meng, Shayan Sayyed, Kevin Zhu, Austen Liao, and Sean O’Brien. EnDive: A cross-dialect benchmark for fairness and performance in large language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16830–16855, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.913. URL https://aclanthology.org/2025.findings-emnlp.913/. ↗
  12. Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, and Edward Choi. Trans-env: A framework for evaluating the linguistic robustness of LLMs against english varieties. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=YIpvHrQAks. ↗
  13. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. Ai generates covertly racist decisions about people based on their dialect. Nature, 633(8028):147–154, 2024. ↗
  14. Genevieve Smith, Eve Fleisig, Madeline Bossi, Ishita Rustagi, and Xavier Yin. Standard language ideology in ai-generated language. arXiv preprint arXiv:2406.08726, 2024. ↗
  15. Rada Mihalcea, Oana Ignat, Longju Bai, Angana Borah, Luis Chiruzzo, Zhijing Jin, Claude Kwizera, Joan Nwatu, Soujanya Poria, and Thamar Solorio. Why ai is weird and shouldn’t be this way: Towards ai for everyone, with everyone, by everyone. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27):28657–28670, Apr. 2025. doi: 10.1609/aaai. v39i27.35092. URL https://ojs.aaai.org/index.php/AAAI/article/view/35092. ↗
  16. Nicholas Deas, Jessica Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. Evaluation of African American language bias in natural language generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6805–6824, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.421. URL https://aclanthology.org/2023.emnlp-main.421/. ↗
  17. Zheng-Xin Yong, Ruochen Zhang, Jessica Zosa Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Indra Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Yin Lin Tan, Long Phan, Rowena Garcia, Thamar Solorio, and Alham Fikri Aji. Prompting multilingual large language models to generate code-mixed texts: The case of south East Asian languages. In Genta Winata, Sudipta Kar, Marina Zhukova, Thamar Solorio, Mona Diab, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali, editors, Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching, pages 43–63, Singapore, December 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.calcs-1.5/. ↗
  18. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. Natural language processing for dialects of a language: A survey. ACM Comput. Surv., 57(6), February 2025. ISSN 0360-0300. doi: 10.1145/3712060. URL https://doi.org/10.1145/3712060. ↗
  19. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 2024. find
  20. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. VALUE: Understanding dialect disparity in NLU. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.258. URL https://aclanthology.org/2022.acl-long.258/. ↗
  21. Sarah E. Finch, Ellie S. Paek, Ikseon Choi, and Jinho D. Choi. Finding a voice: Exploring the potential of African American dialect and voice generation for chatbots. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25789–25806, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1252. URL https://aclanthology.org/2025.acl-long.1252/. ↗
  22. Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang, and Nanyun Peng. Dialectgen: Benchmarking and improving dialect robustness in multimodal generation. arXiv preprint arXiv:2510.14949, 2025. ↗
  23. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. eWAVE. 2020. URL https://ewave-atlas.org/. ↗
  24. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. Evaluating dialect robustness of language models via conversation understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 24–38, Abu Dhabi, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025. sumeval-2.3/. ↗
  25. Anwoy Chatterjee, H S V N S Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. POSIX: A prompt sensitivity index for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14550–14565, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.852. URL https://aclanthology.org/2024.findings-emnlp.852/. ↗
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA, 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://doi.org/10.3115/1073083.1073135. ↗
  27. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. ↗
  28. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. ↗
  29. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. ↗
  30. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. ↗
  31. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, September 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z. ↗
  32. Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, and Steven Whang. Erbench: An entity-relationship based automatically verifiable hallucination benchmark for large language models. Advances in Neural Information Processing Systems, 37:53064–53101, 2024. find
  33. Jio Oh, Geon Heo, Seungjun Oh, Hyunjin Kim, JinYeong Bak, Jindong Wang, Xing Xie, and Steven Euijong Whang. Better think with tables: Tabular structures enhance llm comprehension for data-analytics requests. arXiv preprint arXiv:2412.17189, 2024. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oh-2026-dialectllm-dialect-aware-dialog-ue/