Elamur Research⌕
☰
home / papers / 2026 / The Hrunting of AI: Where and How to Improve English Dialectal Fairness

The Hrunting of AI: Where and How to Improve English Dialectal Fairness

Paper · 2026 · Wei Li and Adrian de Wynter

Summary

Large language models handle mainstream English well but perform worse on its dialects, and fixing this is hard when native-speaker data is scarce. The authors test the standard pipeline for adapting an LLM to a locale on four English dialects (AAVE, Yorkshire, Geordie, Cornish/West Country) plus West Frisian as a related low-resource control. The pipeline has four steps: generating task-specific data (TSD), human rating, LLM-as-a-judge rating and fine-tuning. Native speakers scored LLM outputs on a binary rubric.

The central finding is that the pattern of agreement among human annotators reappears in two other places: in how well LLM judges agree with humans, and in judge F1. Fine-tuning does not remove this pattern for the English dialects and sometimes widens it. West Frisian, however, improves strongly after fine-tuning. The authors attribute the pattern to two factors: low resourceness and the lexically ambiguous nature of dialects.

The results also contain good news. Several models already produce high-quality AAVE and Yorkshire text. The authors conclude that statistical methods alone will not close the gap, so linguistics-informed tools and native-speaker involvement are required.

Key points

  • Data. 1,000 prompts were transcreated per locale from WildChat, GSM8K, OpenOrca, OpenCode and SHP (cooking/baking planning). Each item was labelled per criterion by three native speakers, and the final label came from a majority vote.

  • Rubric. Six binary criteria cover:

    • correct locale,
    • cultural and argumentative accuracy,
    • correct code,
    • grammaticality,
    • no truncation,
    • full prompt compliance.

    An output counts as good only if it passes all six.

  • Human agreement (Gwet's AC1). The average was 0.75. By locale: AAVE 0.98, Yorkshire 0.96, Geordie 0.67, Frisian 0.61, Cornish 0.51.

  • Models and generation quality. The models were GPT-5-mini, GPT-4.1, Claude Opus 4.1 and Qwen3 8B (generation and judging), plus Phi-4 (judging only). Qwen3 was the model fine-tuned as a judge.

    • Share of good outputs, excluding Qwen3: AAVE ~95.7%, Yorkshire ~97.5%, Cornish 40.4%, Geordie ~61.2%, Frisian ~27.7%.
    • Coding was the weakest task.

  • Error analysis and release. Frequent errors were drifting into the wrong language (Cornish the Celtic language, Dutch) and borrowing Scottish/Northern words into the wrong dialect. Code and data are released under MIT at github.com/adewynter/Beowulf.

Interesting ideas

  • I1. An all-or-nothing rubric made of several narrow binary checks keeps labels transparent and lets errors be traced to a specific criterion.
  • I2. Inter-annotator agreement serves as a diagnostic ceiling: if humans disagree, a judge trained on their consensus will mirror that disagreement rather than overcome it. AC1 is preferred over Cohen's κ because skewed labels distort κ less.
  • I3. A closely related but structurally distinct control language (West Frisian) exposes whether an observed effect is specific to the target group or general.
  • I4. One prompt strategy asks the judge about a single criterion per call and combines the verdicts in code. After fine-tuning this matched the best strategy, and it gives finer-grained, inspectable decisions.
  • I5. Edit distance from the standard variety served as a cheap, coarse proxy for dialect divergence, and it was enough to help explain where models fail.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test and fable-QA answers could be graded against separate binary checks (right entity, right event order, well-formed sentence), so a failure points to the stage that broke.
  • I2 (learn): When building silver data from witness parsers, compute AC1 among the witnesses per genre or dialect and treat low-agreement slices as unreliable training signal rather than just more data.
  • I3 (learn): Include a related control register or dialect in evaluations as a negative control, so improvements from self-induced repair rules can be checked for genuine specificity.
  • I4 (learn): LLM teacher tiers could be queried one property at a time (attachment, label, role), with verdicts merged programmatically, keeping each teacher decision explainable.
  • I5 (tree): Measure edit distance of dialect text to standard English to predict where UD dialect conversion and the parser will need explicit linguistic rules instead of more statistical training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of YorkYork · United Kingdom17 works · 6,570 citations🎓Boston CollegeBoston · United States3 works · 15 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksBias, fairness & safety · 517 works

Cite

Cite as Wei Li and Adrian de Wynter. 2026. The Hrunting of AI: Where and How to Improve English Dialectal Fairness. arXiv:2603.15187.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.15187 · arXiv 2603.15187

@misc{li2026hrunting,
  title = {{The Hrunting of AI: Where and How to Improve English Dialectal Fairness}},
  author = {Wei Li and Adrian {de Wynter}},
  year = {2026},
  doi = {10.48550/arxiv.2603.15187},
  eprint = {2603.15187},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.15187}
}

References

134 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chaymaa Abbas, Mariette Awad, and Razane Tajeddine. 2025. Can small-scale data poisoning exacerbate dialect-linked biases in large language models? Preprint, arXiv:2507.19195. ↗
  2. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. Phi-4 technical report. Preprint, arXiv:2412.08905. ↗
  3. Azzedine Aftiss, Salima Lamsiyah, Christoph Schommer, and Said Ouatik El Alaoui. 2025. Empirical evaluation of pre-trained language models for summarizing Moroccan Darija news articles. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 77–85, Abu Dhabi, UAE. Association for Computational Linguistics.
  4. Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. 2025. Opencodereasoning: Advancing data distillation for competitive coding. ↗
  5. Sadam Al-Azani, Nora Alturayeif, Haneen Abouelresh, and Alhanoof Alhunief. 2024. A comprehensive framework and empirical analysis for evaluating large language models in arabic dialect identification. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–7. find
  6. Renad Al-Monef, Hassan Alhuzali, Nora Alturayeif, and Ashwag Alasmari. 2025. Absher: A benchmark for evaluating large language models understanding of saudi dialects. Preprint, arXiv:2507.10216. ↗
  7. Hind Al-Owais and Ashraf Elnagar. 2025. Arabic dialect detection using large language models: A comparative analysis. In 2025 International Conference on New Trends in Computing Sciences (ICTCS), pages 454–459. find
  8. Abdullah Alabdullah, Lifeng Han, and Chenghua Lin. 2025. Advancing dialectal arabic to modern standard arabic machine translation. Preprint, arXiv:2507.20301. ↗
  9. Dimah Alahmadi. 2025. Human versus AI: A comparative study of zero-shot LLMs and transformer models against human annotations for Arabic sentiment analysis. International Journal of Advanced Computer Science and Applications, 16(8). ↗
  10. Anton Alexandrov, Veselin Raychev, Dimitar I Dimitrov, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024. BgGPT 1.0: Extending english-centric LLMs to other languages. arXiv preprint arXiv:2412.10893. ↗
  11. Abdulsalam obaid Alharbi, Abdullah Alsuhaibani, Abdulrahman Abdullah Alalawi, Usman Naseem, Shoaib Jameel, Salil Kanhere, and Imran Razzak. 2025. Evaluating large language models on health-related claims across Arabic dialects. In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 95–103, Abu Dhabi, UAE. Association for Computational Linguistics.
  12. Hana Alostad. 2025. Large language models as kuwaiti annotators. Big Data and Cognitive Computing, 9(2). ↗
  13. Alanoud Alotaibi, Farrukh Nadeem, and Mohamed Hamdy. 2025. Weakly supervised deep learning for arabic tweet sentiment analysis on education reforms: Leveraging pre-trained models and llms with snorkel. IEEE Access, 13:30523–30542. ↗
  14. Malik H. Altakrori, Nizar Habash, Abdelhakim Freihat, Younes Samih, Kirill Chirkunov, Muhammed AbuOdeh, Radu Florian, Teresa Lynn, Preslav Nakov, and Alham Fikri Aji. 2025. DialectalArabicMMLU: Benchmarking dialectal capabilities in arabic and multilingual language models. Preprint, arXiv:2510.27543. ↗
  15. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025. Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics.
  16. Anthropic. 2025. Introducing Claude Sonnet 4.5. find
  17. Katherine Atwell, Mert Inan, Anthony B. Sicilia, and Malihe Alikhani. 2024. Combining discourse coherence with large language models for more inclusive, equitable, and robust task-oriented dialogue. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3538– 3552, Torino, Italia. ELRA and ICCL. find
  18. Lama Ayash, Hassan Alhuzali, Ashwag Alasmari, and Sultan Aloufi. 2025. Saudiculture: A benchmark for evaluating large language models’ cultural competence within saudi arabia. Journal of King Saud University - Computer and Information Sciences, 37:123. ↗
  19. Hassan Barmandah. 2025. Saudi-dialect-allam: Lora fine-tuning for dialectal arabic generation. Preprint, arXiv:2508.13525. ↗
  20. Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adserà, and Mikhail Belkin. 2026. Toward universal steering and monitoring of AI models. Science, 391(6787):787–792. ↗
  21. David Beauchemin, Yan Tremblay, Mohamed Amine Youssef, and Richard Khoury. 2025. A set of quebec-french corpus of regional expressions and terms. Preprint, arXiv:2510.05026. ↗
  22. Ayah Beidas, Kousar Mohi, Fatme Ghaddar, Imtiaz Ahmad, and Sa’Ed Abed. 2025. Cross-dialectal arabic translation: comparative analysis on large language models. Frontiers of Artificial Intelligence. ↗
  23. Jaione Bengoetxea, Mikel Zubillaga, Ekhi Azurmendi, Maite Heredia, Julen Etxaniz, Markel Ferro, and Jeremy Barnes. 2025. HiTZ at VarDial 2025 NorSID: Overcoming data scarcity with language transfer and automatic data annotation. In Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 209–219, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  24. Hunzalah Hassan Bhatti and Firoj Alam. 2025. Beyond mcq: An open-ended arabic cultural qa benchmark with dialect variants. Preprint, arXiv:2510.24328. ↗
  25. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world’s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Ireland. Association for Computational Linguistics.
  26. Jennifer Bloomquist, Lisa J. Green, and Sonja L. Lanehart. 2015. The Oxford Handbook of African American Language. Oxford University Press. find
  27. Alaa Bouomar and Noorhan Abbas. 2025. AraSim: Optimizing Arabic dialect translation in children’s literature with LLMs and similarity scores. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 93–102, Abu Dhabi, UAE. Association for Computational Linguistics.
  28. Faeze Brahman, Vered Shwartz, Rachel Rudinger, and Yejin Choi. 2021. Learning to rationalize for non-monotonic reasoning with distant supervision. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12592–12601. find
  29. Katarzyna Buczek. 2020. For and against Anglo-Frisian: The linguistic debate on the matter. Mostly Medieval: In Memory of Jacek Frisiak, pages 245–274. find
  30. Minh Duc Bui, Carolin Holtermann, Valentin Hofmann, Anne Lauscher, and Katharina von der Wense. 2025. Large language models discriminate against speakers of German dialects. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8223–8251, Suzhou, China. Association for Computational Linguistics. find
  31. Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. Llms are few-shot in-context low-resource language learners. arXiv preprint arXiv:2403.16512. ↗
  32. Stergios Chatzikyriakidis, Dimitris Papadakis, Sevasti-Ioanna Papaioannou, and Erofili Psaltaki. 2025. GRDD+: An extended greek dialectal dataset with cross-architecture fine-tuning evaluation. Preprint, arXiv:2511.03772. ↗
  33. Domenic V. Cicchetti and Alvan R. Feinstein. 1990. High agreement but low kappa: II. resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551– 558. ↗
  34. William Coggins, Jasmine McKenzie, Sangpil Youm, Pradham Mummaleti, Juan Gilbert, Eric Ragan, and Bonnie J Dorr. 2025. That ain’t right: Assessing LLM performance on QA in African American and West African English dialects. In Proceedings of the 9th Widening NLP Workshop, pages 123–129, Suzhou, China. Association for Computational Linguistics.
  35. David Crystal. 2008. A Dictionary of Linguistics and Phonetics, 6 edition. Wiley-Blackwell. find
  36. Patricia Cukor-Avila. 2003. The complex grammatical history of African-American and white vernaculars in the South, page 82–105. Studies in English Language. Cambridge University Press. find
  37. Nicholas Deas and Kathleen McKeown. 2025. Artificial impressions: Evaluating large language model behavior through the lens of trait impressions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19429–19455, Suzhou, China. Association for Computational Linguistics. ↗
  38. Anastasiia Demidova, Hanin Atwany, Nour Rabih, and Sanad Sha’ban. 2024. Arabic train at NADI 2024 shared task: LLMs’ ability to translate Arabic dialects into Modern Standard Arabic. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 729–734, Bangkok, Thailand. Association for Computational Linguistics.
  39. Antonios Dimakis, John Pavlopoulos, and Antonios Anastasopoulos. 2025. Dialect normalization using large language models and morphological rules. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23696–23714, Vienna, Austria. Association for Computational Linguistics. find
  40. Rebecca Dorn, Christina Chance, Casandra Rusti, Charles Bickham Jr., Kai-Wei Chang, Fred Morstatter, and Kristina Lerman. 2025. Reinforcing stereotypes of anger: Emotion AI on African American Vernacular English. Preprint, arXiv:2511.10846. ↗
  41. Samhaa R. El-Beltagy and Mohamed A. Abdallah. 2024. Exploring retrieval augmented generation in arabic. Procedia Computer Science, 244:296–307. 6th International Conference on AI in Computational Linguistics.
  42. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. NileChat: Towards linguistically diverse and culturally aware LLMs for local communities. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10978–11002, Suzhou, China. Association for Computational Linguistics. ↗
  43. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988–6008. PMLR. find
  44. Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. DIALECTBENCH: An NLP benchmark for dialects, varieties, and closely-related languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14412–14454, Bangkok, Thailand. Association for Computational Linguistics. find
  45. Fahim Faisal and Antonios Anastasopoulos. 2024a. Data-augmentation-based dialectal adaptation for llms. arXiv preprint arXiv:2404.08092. ↗
  46. Fahim Faisal and Antonios Anastasopoulos. 2024b. Data-augmentation-based dialectal adaptation for LLMs. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 197–208, Mexico City, Mexico. Association for Computational Linguistics. find
  47. Fahim Faisal and Antonios Anastasopoulos. 2025. Testing the boundaries of LLMs: Dialectal and language-variety tasks. In Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 68–92, Abu Dhabi, UAE. Association for Computational Linguistics. find
  48. Fahim Faisal, Md Mushfiqur Rahman, and Antonios Anastasopoulos. 2025. Dialectal toxicity detection: Evaluating LLM-as-a-judge consistency across language varieties. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 12429–12452, Suzhou, China. Association for Computational Linguistics. find
  49. Joseph L. Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3):613–619. ↗
  50. Raquel Freitag and Túlio Gois. 2024. Performance in a dialectal profiling task of llms for varieties of brazilian portuguese. In Anais do XV Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 317–326, Porto Alegre, RS, Brasil. SBC. In portuguese. find
  51. Lisa J. Green. 2012. African American English: A Linguistic Introduction. Cambridge University Press. find
  52. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American Vernacular English in transformer-based text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5877–5883, Online. Association for Computational Linguistics. ↗
  53. Kilem Li Gwet. 2010. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61:29–48. find
  54. Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2023. Are large language model-based evaluators the solution to scaling up multilingual evaluation? arXiv preprint arXiv:2309.07462. ↗
  55. Zineb Ferhat Hamida, Ahlem Drif, and Silvia Giordano. 2024. Arabic offensive language detection based on zero-shot prompting llm in social media. In 2024 1st International Conference on Electrical, Computer, Telecommunication and Energy Technologies (ECTE-Tech), pages 1–5. Mohamed Hannani, Abdelhadi Soudi, and Kristof Van Laerhoven. Evaluating ChatGPT-4 and machine learning models for sentiment analysis on a multi-script Moroccan Arabic corpus: Insights, challenges, and future direction. Journal of Data Mining & Digital Humanities. find
  56. Mohamed Hannani, Abdelhadi Soudi, and Kristof Van Laerhoven. 2024. Assessing the performance of ChatGPT-4, fine-tuned BERT and traditional ML models on Moroccan Arabic sentiment analysis. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pages 489–498, Miami, USA. Association for Computational Linguistics.
  57. Md Arid Hasan, Prerona Tarannum, Krishno Dey, Imran Razzak, and Usman Naseem. 2024. Do large language models speak all languages equally? a comparative study in low-resource settings. arXiv preprint arXiv:2408.02237. ↗
  58. William Held, David Hall, Percy Liang, and Diyi Yang. 2025. Relative scaling laws for llms. Preprint, arXiv:2510.24626. ↗
  59. Ahmed Ibrahim, Abdullah Hosseini, Hoda Helmy, Wafa Lakhdhar, and Ahmed Serag. 2025. Bridging dialectal gaps in Arabic medical LLMs through model merging. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 338– 346, Suzhou, China. Association for Computational Linguistics. find
  60. Michael Ibrahim. 2024. CUFE at NADI 2024 shared task: Fine-tuning llama-3 to translate from Arabic dialects to Modern Standard Arabic. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 769–773, Bangkok, Thailand. Association for Computational Linguistics. ↗
  61. Fred Ivey. 1996. Memories of a long and happy life PLUS My own dictionary of Cornish dialect words. Packet Publishing. Available at https://homepages.rootsweb.com/~marcie/kernow/language.html. ↗
  62. Judith Jesch. 2017. Old Norse in England. Wiley. find
  63. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.
  64. Vani Kanjirangat, Ljiljana Dolamic, and Fabio Rinaldi. 2025. Exploring data and parameter efficient strategies for arabic dialect identifications. Preprint, arXiv:2509.13775. ↗
  65. Kernow Goth. 2020. Cornish dialect words and phrases. Accessed 23 December 2025. find
  66. Omkar Khade, Shruti Jagdale, Abhishek Phaltankar, Gauri Takalikar, and Raviraj Joshi. 2024. Challenges in adapting multilingual llms to low-resource languages using lora peft tuning. arXiv preprint arXiv:2411.18571. ↗
  67. Eeham Khan, Firas Saidani, Owen Van Esbroeck, Richard Khoury, and Leila Kosseim. 2025. Low-resource dialect adaptation of large language models: A french dialect case-study. Preprint, arXiv:2510.22747. ↗
  68. Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David W. Graham, F.Q. Hu, and 29 others. 2023. The semantic scholar open data platform. ArXiv, abs/2301.10140. ↗
  69. M Paul Lewis, Gary F Simons, and Charles D Fennig, editors. 2020. Ethnologue: languages of Africa and Europe, volume 24. SIL International. Online version: http://www.ethnologue. com. ↗
  70. Peerat Limkonchotiwat, Kanruethai Masuk, Surapon Nonesung, Chalermpun Mai-On, Sarana Nutanong, Wuttikorn Ponwitayarat, and Potsawee Manakul. 2025. Assessing thai dialect performance in llms with automatic benchmarks and human evaluation. Preprint, arXiv:2504.05898. ↗
  71. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael Wooldridge, Janet B. Pierrehumbert, and Furu Wei. 2025. One language, many gaps: Evaluating dialect fairness and robustness of large language models in reasoning tasks. Preprint, arXiv:2410.11005. ↗
  72. Samar Mohamed Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Taher Abdelfadil, Shady Shehata, and Muhammad Abdul-Mageed. 2025. JAWAHER: A multidialectal dataset of Arabic proverbs for LLM benchmarking. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12320–12341, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  73. Mohamed Mahdi. 2025. How well do llms understand tunisian arabic? Preprint, arXiv:2511.16683. ↗
  74. Rim Mahouachi and Hela Mahersia. 2025. Multilingual and tunisian dialect analysis of banking app reviews using llms. Expert Systems with Applications, 290:128260. ↗
  75. Barrie Markham Rodes. 2015. Yorkshire dialect words of Old Norse origin. find
  76. Gonzalo Martínez, Marina Mayor-Rocher, Cris Pozo Huertas, Nina Melero, María Grandury, and Pedro Reviriego. 2025. Spanish is not just one: A dataset of spanish dialect recognition for llms. Data in Brief, 63:112088. find
  77. James Milroy and Lesley Milroy. 2012. Routledge. find
  78. Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Maarten Sap. 2025. Rejected dialects: Biases against African American language in reward models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7468–7487, Albuquerque, New Mexico. Association for Computational Linguistics. find
  79. Rasheed Mohammad, Omer S. Alkhnbashi, and Mohammad Hammoudeh. 2024. Optimizing large language models for arabic healthcare communication: A focus on patient-centered NLP applications. Big Data and Cognitive Computing, 8(11). find
  80. Leila Moudjari and Farah Benamara. 2025. Are dialects better prompters? a case study on Arabic subjective text classification. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17356–17371, Vienna, Austria. Association for Computational Linguistics. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. ↗
  81. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics.
  82. Salikoko Sangol Mufwene. 2025a. African american english. Accessed 23 December 2025. find
  83. Salikoko Sangol Mufwene. 2025b. African american vernacular english. Accessed 23 December 2025. find
  84. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707. ↗
  85. Omer Nacar. 2025. Ui-level evaluation of allam 34b: Measuring an arabic-centric llm via humain chat. Preprint, arXiv:2508.17378. ↗
  86. Lynnette Hui Xian Ng and Luo Qi Chan. 2024. What talking you?: Translating code-mixed messaging texts to english. Preprint, arXiv:2411.05253. Northumbria University Newcastle. 10 geordie phrases you should know. ↗
  87. Jaclyn Ocumpaugh, Xiner Liu, and Andres Felipe Zambrano. 2025. Language models and dialect differences. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, LAK ’25, page 204–215, New York, NY, USA. Association for Computing Machinery. find
  88. Office for National Statistics. 2021a. Census. find
  89. Office for National Statistics. 2021b. National identity, England and Wales: Census 2021. find
  90. Viktória Ondrejová and Marek Šuppa. 2024. Can LLMs handle low-resource dialects? a case study on translation and common sense reasoning in šariš. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 130–139, Mexico City, Mexico. Association for Computational Linguistics. find
  91. Open AI. 2025a. Introducing GPT-4.1 in the API. find
  92. Open AI. 2025b. Introducing GPT-5. find
  93. Yulia Otmakhova, Hung Thinh Truong, Rahmad Mahendra, Zenan Zhai, Rongxin Zhu, Daniel Beck, and Jey Han Lau. 2025. FLUKE: A linguistically-driven and task-agnostic framework for robustness evaluation. Preprint, arXiv:2504.17311. ↗
  94. Eileen Pan, Anna Seo Gyeong Choi, Maartje Ter Hoeve, Skyler Seto, and Allison Koenecke. 2025. Analyzing dialectical biases in LLMs for knowledge and reasoning benchmarks. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 20882–20893, Suzhou, China. Association for Computational Linguistics.
  95. Shantipriya Parida, Shakshi Panwar, Kusum Lata, Sanskruti Mishra, and Sambit Sekhar. 2024. Building pre-train llm dataset for the indic languages: a case study on hindi. Preprint, arXiv:2407.09855. ↗
  96. Benedikt Perak, Slobodan Beliga, and Ana Meštrović. 2024. Incorporating dialect understanding into LLM using RAG and prompt engineering techniques for causal commonsense reasoning. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 220–229, Mexico City, Mexico. Association for Computational Linguistics. find
  97. Tabia Tanzin Prama, Christopher M. Danforth, and Peter Sheridan Dodds. 2025. Llms for low-resource dialect translation using context-aware prompting: A case study on sylheti. Preprint, arXiv:2511.21761. ↗
  98. Faisal Qarah and Tawfeeq Alsanoosy. 2025. Evaluation of arabic large language models on moroccan dialect. Engineering, Technology & Applied Science Research, 15(3):22478–22485. ↗
  99. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. ↗
  100. Nathaniel Romney Robinson, Shahd Abdelmoneim, Kelly Marchisio, and Sebastian Ruder. 2025. AL-QASIDA: Analyzing LLM quality and accuracy systematically in dialectal Arabic. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22048–22065, Vienna, Austria. Association for Computational Linguistics.
  101. Michael J Ryan, William Held, and Diyi Yang. 2024. Unintended impacts of LLM alignment on global representation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16121–16140, Bangkok, Thailand. Association for Computational Linguistics.
  102. K. M. Jubair Sami, Dipto Sumit, Ariyan Hossain, and Farig Sadeque. 2025. A comparative analysis of retrieval-augmented generation techniques for bengali standard-to-dialect machine translation using llms. Preprint, arXiv:2512.14179. ↗
  103. Kayla Schroeder and Zach Wood-Doughty. 2024. Can you trust llm judgments? reliability of llm-as-a-judge. arXiv preprint arXiv:2412.12509. ↗
  104. Guokan Shang, Hadi Abdine, Ahmad Chamma, Amr Mohamed, Mohamed Anwar, Abdelaziz Bounhar, Omar El Herraoui, Preslav Nakov, Michalis Vazirgiannis, and Eric P. Xing. 2025a. Nile-chat: Egyptian language models for Arabic and Latin scripts. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 306–322, Suzhou, China. Association for Computational Linguistics.
  105. Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis, and Eric Xing. 2025b. Atlas-chat: Adapting large language models for low-resource Moroccan Arabic dialect. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 9–30, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  106. Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, and 1 others. 2024. Atlas-chat: Adapting large language models for low-resource moroccan arabic dialect. arXiv preprint arXiv:2409.17912. ↗
  107. David Simpson. 2024. Geordie dictionary. Accessed 23 December 2025. find
  108. Sunayana Sitaram, Adrian de Wynter, Isobel McCrum, Qilong Gu, and Si-Qing Chen. 2025. A multilingual, culture-first approach to addressing misgendering in LLM applications. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31147–31171, Suzhou, China. Association for Computational Linguistics. find
  109. Matthew Spriggs. 2003. Where Cornish was spoken and when: a provisional synthesis. Cornish Studies, 11. find
  110. Dipankar Srirag, Aditya Joshi, and Jacob Eisenstein. 2025. Predicting the target word of game-playing conversations using a low-rank dialect adapter for decoder models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 8–17, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  111. Patrick Stiles. 1995. Remarks on the ’Anglo-Frisian’ thesis. riesische Studien II: Beiträge des Föhrer Symposiums zur Friesischen Philologie vom 7.–8. April 1994. find
  112. Chihiro Taguchi, Seng Mai, Keita Kurabe, Yusuke Sakai, Georgina Agyei, Soudabeh Eslami, and David Chiang. 2025. Languages still left behind: Toward a better multilingual machine translation benchmark. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20142–20154, Suzhou, China. Association for Computational Linguistics. find
  113. Megh Thakkar, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. 2024. A deep dive into the trade-offs of parameter-efficient preference alignment techniques. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5732–5745, Bangkok, Thailand. Association for Computational Linguistics. ↗
  114. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624. ↗
  115. Hao Tran and Annita Stell. 2024. Beyond borders or building new walls? Australian Review of Applied Linguistics, 47(3):284–308. ↗
  116. UNESCO. 2010. Atlas of the world’s languages in danger. UNESCO Publishing. find
  117. Azmine Toushik Wasi, Raima Islam, Mst Rafia Islam, Farig Sadeque, Taki Hasan Rafi, and Dong-Kyu Chae. 2025a. Dialectal bias in bengali: An evaluation of multilingual large language models across cultural variations. In Companion Proceedings of the ACM on Web Conference 2025, WWW ’25, page 1380–1384, New York, NY, USA. Association for Computing Machinery. find
  118. Azmine Toushik Wasi, Raima Islam, Mst Rafia Islam, Farig Y. Sadeque, Taki Hasan Rafi, and Dong-Kyu Chae. 2025b. Exploring bengali creative storytelling capabilities of large language models across cultural variations. In Companion Publication of the 2025 Conference on Computer-Supported Cooperative Work and Social Computing, CSCW Companion ’25, page 214–218, New York, NY, USA. Association for Computing Machinery. find
  119. Dominic Watt and William Allen. 2003. Tyneside english. Journal of the International Phonetic Association, 33(2):267–271. find
  120. Søren Wichmann and Matthias Urban. 2012. Toward an automated classification of englishes. In The Oxford Handbook of the History of English. Oxford University Press. ↗
  121. Walt Wolfram. 2020a. Rural african american vernacular english. In Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors, The Electronic World Atlas of Varieties of English. find
  122. Walt Wolfram. 2020b. Urban african american vernacular english. In Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors, The Electronic World Atlas of Varieties of English. find
  123. Walt Wolfram and Erik R. Thomas. 2002. The Development of African American English. ↗
  124. Adrian de Wynter. 2025a. Awes, laws, and flaws from today’s LLM research. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12834–12854, Vienna, Austria. Association for Computational Linguistics. find
  125. Adrian de Wynter. 2025b. Labelling data with unknown references. Preprint, arXiv:2506.03083. ↗
  126. Adrian De Wynter, Ishaan Watts, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Nektar Ege Altıntoprak, Lena Baur, Samantha Claudet, Pavel Gaj-dušek, Qilong Gu, Anna Kaminska, Tomasz Kaminski, Ruby Kuo, Akiko Kyuba, Jongho Lee, Kartik Mathur, Petter Merok, Ivana Milovanović, Nani Paananen, and 13 others. 2025. RTP-LX: Can LLMs evaluate toxicity in multilingual scenarios? Proceedings of the AAAI Conference on Artificial Intelligence, 39(27):27940–27950. find
  127. Zhichao Xu, Ashim Gupta, Tao Li, Oliver Bentham, and Vivek Srikumar. 2024. Beyond perplexity: Multidimensional safety evaluation of LLM compression. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15359–15396, Miami, Florida, USA. Association for Computational Linguistics. find
  128. Silvana Yakhni and Ali Chehab. 2025a. Can LLMs translate cultural nuance in dialects? a case study on Lebanese Arabic. In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 114–135, Abu Dhabi, UAE. Association for Computational Linguistics.
  129. Silvana Yakhni and Ali Chehab. 2025b. Fine-tuning llms for low-resource dialect translation: The case of lebanese. Preprint, arXiv:2505.00114. ↗
  130. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  131. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, and 1 others. 2024. A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Processing Systems, 37:46819– 46836. find
  132. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations. ↗
  133. Runtao Zhou, Guangya Wan, Saadia Gabriel, Sheng Li, Alexander J Gates, Maarten Sap, and Thomas Hartvigsen. 2025a. Disparities in llm reasoning accuracy and explanations: A case study on african american english. Preprint, arXiv:2503.04099. ↗
  134. Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang, and Nanyun Peng. 2025b. DialectGen: Benchmarking and improving dialect robustness in multimodal generation. Preprint, arXiv:2510.14949. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2026-hrunting-ai-where-improve-english/