Elamur Research⌕
☰
home / papers / 2025 / Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties

Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties

Paper · 2025 · Jiyoung Lee, Seungho Kim, Jieun Han et al.

Summary

LLM benchmarks are written almost entirely in Standard American English (SAE), so it is unclear how well models serve people who use regional dialects or write English as a second language (ESL). The authors built Trans-EnV, a pipeline that rewrites existing SAE benchmark items into a chosen English variety.

Linguistic features come from expert resources: eWAVE for dialects and the English Grammar Profile for learner English at CEFR levels A and B. For each feature, an LLM drafts a step-by-step transformation guideline. A feature transformer LLM then applies the variety's features one at a time, while a semantic checker LLM verifies that the meaning is preserved. Learner variants also reflect the writer's first language, and advanced vocabulary is first replaced with simpler wording.

Six benchmarks were converted into 38 varieties, and seven current LLMs were tested on them. Accuracy fell on most varieties, by up to 12.5% for dialects and 46.3% for ESL, and reasoning tasks were hit hardest. The framework makes robustness to English variation measurable at scale.

Key points

  • Dialect selection: The 18 dialects were chosen from the 77 varieties in eWAVE, which documents 235 features and was compiled by 84 linguists from 175 publications. KNN clustering over the feature profiles separated English dialects from creoles and pidgins. Only features marked at the highest presence level define each dialect.
  • ESL varieties: The 20 ESL varieties draw on the English Grammar Profile, which maps 1,222 "can-do" grammar features to CEFR levels. A lower level is simulated by stripping out constructions that belong to higher levels.
  • Main results: Larger drops appear for L1s that are harder for English speakers to learn, such as Arabic, Chinese-Mandarin and Japanese, compared with easier L1 categories (shown on GSM8K and ARC). Models specialised for reasoning proved more robust.
  • Open-ended tasks: IFEval, AlpacaFarm and MT-Bench were run on 8 dialects and 10 ESL variants with four models: Qwen2.5-72B-Instruct, DeepSeek-R1-Distill-Llama-70B, LLaMA-3.3-70B-Instruct and o4-mini. GPT-4.1 served as judge.
    • CEFR B inputs scored above CEFR A inputs.
    • Irish, Newfoundland and Welsh English scored below AAVE, Australian and Tristan da Cunha English.
  • Validation and release: Transformation coverage exceeded 80%. The framework was checked through statistical tests, human evaluation, and ongoing advice from a second language acquisition researcher. Code is on GitHub and the datasets are on Hugging Face.

Interesting ideas

  • I1. Variation is decomposed into a catalogue of expert-documented features, each with its own written rewrite procedure, applied one after another under a meaning-preservation check; this makes each change traceable instead of producing an opaque "dialect-ify" rewrite.
  • I2. A lower proficiency level is defined subtractively, by removing constructions that the grammar profile assigns to higher levels; this gives a principled, checkable notion of "simpler text."
  • I3. Because the transformations keep meaning fixed, any change in the answer is a robustness failure by construction; this is an invariance test that needs no new gold labels.
  • I4. Results are grouped by how far a variety, or a learner's L1, sits from SAE, which turns scattered scores into a gradient that shows where robustness breaks down.
  • I5. A lexical simplification pass runs before grammatical features are injected, so vocabulary and syntax effects can be studied separately.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build negative-control evaluation sets by applying single eWAVE features to clean sentences, then check that the parser and frame extractor yield the same who-did-what-to-whom as on the SAE original.
  • I2 (back): Constrain generation or retelling to EGP constructions of a target CEFR level, so Mova can produce graded simplified versions of tales and explain which constructions it avoided.
  • I3 (answer): Rerun school tests and fable QA on dialect and learner variants, then use Mova's explainable pipeline to locate which stage (tokens, tree, frames) diverged when an answer flips.
  • I4 (tree): Report LAS per variety bucket ordered by distance from SAE, and prioritise UD dialect conversion or repair rules for high-impact features such as multiple negation or dropped copulas.
  • I5 (tokens): Test the tagger and lemmatizer on vocabulary-simplified versus grammar-altered variants independently, to see whether errors come from unknown words or from non-standard morphosyntax.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Seoul National UniversitySeoul · South Korea41 works · 45,671 citations🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea51 works · 3,167 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh and Edward Choi. 2025. Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties. arXiv:2505.20875.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.20875 · arXiv 2505.20875

@misc{lee2025trans,
  title = {{Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties}},
  author = {Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh and Edward Choi},
  year = {2025},
  doi = {10.48550/arxiv.2505.20875},
  eprint = {2505.20875},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.20875}
}

References

80 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. ↗
  2. Yuki Arase, Satoru Uchida, and Tomoyuki Kajiwara. Cefr-based sentence difficulty annotation and assessment. arXiv preprint arXiv:2210.11766, 2022. ↗
  3. Peter Bakker. Pidgins and creoles with germanic lexifier languages. In Oxford Research Encyclopedia of Linguistics. 2024. ↗
  4. Abdullah Barayan, Jose Camacho-Collados, and Fernando Alva-Manchego. Analysing zero-shot readability-controlled sentence simplification. arXiv preprint arXiv:2409.20246, 2024. ↗
  5. Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1–4. Springer, 2009. ↗
  6. Markus Bieswanger. Varieties of english in current english language teaching. Stellenbosch Papers in Linguistics, 38(1):27–47, 2008.
  7. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. Systematic inequalities in language technology performance across the world’s languages. arXiv preprint arXiv:2110.06733, 2021. ↗
  8. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. Twitter Universal Dependency parsing for African-American and mainstream American English. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1131. URL https://aclanthology.org/P18-1131/. ↗
  9. Sorana-Daniela Bolboaca and Lorentz Jäntschi. Pearson versus spearman, kendall’s tau correlation analysis on structure-activity relationships of biologic active compounds. Leonardo Journal of Sciences, 5(9): 179–200, 2006. find
  10. Jack K Chambers and Peter Trudgill. Dialectology. Cambridge University Press, 1998. find
  11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. ↗
  12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. ↗
  13. Nicholas Deas, Jessi Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. Evaluation of african american language bias in natural language generation. arXiv preprint arXiv:2305.14291, 2023. ↗
  14. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. ↗
  15. Abhimanyu Dubey et al. The llama 3 herd of models, 2024. arXiv preprint arXiv:2407.21783. ↗
  16. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039–30069, 2023.
  17. Fahim Faisal, Sharlina Keshava, Antonios Anastasopoulos, et al. Sd-qa: Spoken dialectal question answering for the real world. arXiv preprint arXiv:2109.12072, 2021. ↗
  18. Asma Farajidizaji, Vatsal Raina, and Mark Gales. Is it possible to modify text to a target readability level? an initial investigation using zero-shot large language models. arXiv preprint arXiv:2309.12551, 2023. ↗
  19. Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, and Dan Klein. Linguistic bias in chatgpt: Language models reinforce dialect discrimination. arXiv preprint arXiv:2406.08818, 2024. ↗
  20. Thomas Gaillat, Andrew Simpkin, Nicolas Ballier, Bernardo Stearns, Annanda Sousa, Manon Bouyé, and Manel Zarrouk. Predicting cefr levels in learners of english: The use of microsystem criterial features in a machine learning approach. ReCALL, 34(2):130–146, 2022. ↗
  21. Jeroen Geertzen, Theodora Alexopoulou, Anna Korhonen, et al. Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat). In Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project, pages 240–254, 2013. find
  22. Google. Gemini 2.5 pro. https://cloud.google.com/vertex-ai/generative-ai/docs/model s/gemini/2-5-pro, 2025. ↗
  23. Google. Gemini 2.0 flash. https://cloud.google.com/vertex-ai/generative-ai/docs/model s/gemini/2-0-flash, 2025. ↗
  24. Maharshi Gor, Hal Daumé III, Tianyi Zhou, and Jordan Boyd-Graber. Do great minds think alike? investigating human-ai complementarity in question answering with caimira. arXiv preprint arXiv:2410.06524, 2024. ↗
  25. Sylviane Granger. The international corpus of learner english. English language corpora: Design, analysis and exploitation, pages 57–71, 1993. find
  26. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. ↗
  27. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. ↗
  28. Abhay Gupta, Philip Meng, Ece Yurtseven, Sean O’Brien, and Kevin Zhu. Aavenue: Detecting llm biases on nlu tasks in aave via a novel benchmark. arXiv preprint arXiv:2408.14845, 2024. ↗
  29. M Obaidul Hamid and Richard B Baldauf Jr. Second language errors and features of world englishes. World Englishes, 32(4):476–494, 2013. find
  30. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. ↗
  31. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. Ai generates covertly racist decisions about people based on their dialect. Nature, 633(8028):147–154, 2024. ↗
  32. Scott Jarvis. Methodological rigor in the study of transfer: Identifying l1 influence in them interlanguage lexicon. Language learning, 50(2):245–309, 2000. ↗
  33. Jennifer Jenkins. Repositioning english and multilingualism in english as a lingua franca. Englishes in Practice, 2(3):49–85, 2015. ↗
  34. Nicola Jones. Ai now beats humans at basic tasks—new benchmarks are needed, says major report. Nature, 628(8009):700–701, 2024. ↗
  35. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. Natural language processing for dialects of a language: A survey. ACM Computing Surveys, 57 (6):1–37, 2025. ↗
  36. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. Gluecos: An evaluation benchmark for code-switched nlp. arXiv preprint arXiv:2004.12376, 2020. ↗
  37. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. eWAVE. 2020. URL https://ewave-atlas.org/. ↗
  38. Locky Law. Application of generative artificial intelligence (genai) in language teaching and learning: A scoping literature review. Computers and Education Open, page 100174, 2024. ↗
  39. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, 2023. URL https://arxiv.org/abs/2211.09110. ↗
  40. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers. Patterns, 4(7), 2023. ↗
  41. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael Wooldridge, Janet B Pierrehumbert, and Furu Wei. One language, many gaps: Evaluating dialect fairness and robustness of large language models in reasoning tasks. arXiv preprint arXiv:2410.11005, 2024. ↗
  42. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. ↗
  43. Zoe Pei-sui Luk and Yasuhiro Shirai. Is the acquisition order of grammatical morphemes impervious to l1 knowledge? evidence from the acquisition of plural-s, articles, and possessive’s. Language Learning, 59 (4):721–754, 2009. find
  44. Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O Cohen, Valentina Borghesani, Anton Pashkov, et al. Large language models surpass human experts in predicting neuroscience results. Nature human behaviour, 9(2):305–315, 2025. find
  45. Hamzah Faleh Migdadi, Kamariah Yunus, and AF Al Garni. A global view towards understanding of standard and non-standard varieties of english. International journal of academic research in business and social sciences, 10(2):103–115, 2020. ↗
  46. Atsushi Mizumoto, Sachiko Yasuda, and Yu Tamura. Identifying chatgpt-generated texts in efl students’ writing: Through comparative analysis of linguistic fingerprints. Applied Corpus Linguistics, 4(3):100106, 2024. ↗
  47. Akira Murakami and Theodora Alexopoulou. L1 influence on the acquisition order of english grammatical morphemes: A learner corpus study. Studies in Second Language Acquisition, 38(3):365–401, 2016. ↗
  48. Council of Europe. Council for Cultural Co-operation. Education Committee. Modern Languages Division. Common European framework of reference for languages: Learning, teaching, assessment. Cambridge University Press, 2001. find
  49. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. URL https://arxiv.org/ab s/2303.08774. ↗
  50. OpenAI. Gpt-4o mini. https://platform.openai.com/docs/models/gpt-4o-mini, 2024. ↗
  51. OpenAI. Gpt-4.1 mini model card, 2025. URL https://platform.openai.com/docs/models/gp t-4.1-mini. Accessed: 2025-05-16. ↗
  52. OpenAI. Openai o4-mini. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3 722c1/o3-and-o4-mini-system-card.pdf, 2025. ↗
  53. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim, 2019. Leibniz-Institut für Deutsche Sprache. doi: 10.14618/ids-pub-9021. URL http://nbn-resolvi ng.de/urn:nbn:de:bsz:mh39-90215. ↗
  54. Anne O’Keeffe and Geraldine Mark. The english grammar profile of learner competence: Methodology and key findings. International Journal of Corpus Linguistics, 22(4):457–489, 2017. ↗
  55. English Profile. What is english profile? Website: http://www.englishprofile. org/. Accessed: January, 2012. ↗
  56. Manon Reusens, Philipp Borchert, Jochen De Weerdt, and Bart Baesens. Native design bias: Studying the impact of english nativeness on language model performance. arXiv preprint arXiv:2406.17385, 2024. ↗
  57. Amanda Ross and Victor L Willson. One-way anova. In Basic and advanced statistical tests: Writing results sections and creating tables and figures, pages 21–24. Springer, 2017. ↗
  58. Sebastian Ruder, Jonathan H Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel A Sarr, Xinyi Wang, et al. Xtreme-up: A user-centric scarce-data benchmark for under-represented languages. arXiv preprint arXiv:2305.11938, 2023. ↗
  59. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  60. Abdelrahman Abdalla Salih. The future of english and its varieties: An applied linguistic perspective. English Language Teaching, 14(4):16–24, 2021. ↗
  61. Dipankar Srirag, Aditya Joshi, Jordan Painter, and Diptesh Kanojia. Besstie: A benchmark for sentiment and sarcasm classification for varieties of english. arXiv preprint arXiv:2412.04726, 2024. ↗
  62. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. Evaluating dialect robustness of language models via conversation understanding. arXiv preprint arXiv:2405.05688, 2024. ↗
  63. Min-Chang Sung, Kitaek Kim, and Bora Nam. Influence of topic-prominent l1s on the use of l2 english copula be: a corpus-based study. International Review of Applied Linguistics in Language Teaching, (0), 2024. ↗
  64. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. ↗
  65. Gemma Team. Gemma 3. 2025. URL https://arxiv.org/abs/2503.19786. ↗
  66. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. ↗
  67. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/. ↗
  68. Satoru Uchida. Generative ai and cefr levels: Evaluating the accuracy of text generation with chatgpt-4o through textual features. Vocabulary Learning and Instruction, 14(1):2078–2078, 2025. find
  69. Veruska Salvador Vicente. English-based pidgins and creoles: from social to cognitive hypotheses of acquisition. Revista Virtual de Estudos da Linguagem, 5(9):1–30, 2007. find
  70. Dong-Ok Won, Yu Kyoung Shin, Ho-Jung Kim, and Isaiah WonHo Yoo. Advancing language assessment with gpt: Is it nonnative-language friendly? Language Assessment Quarterly, pages 1–28, 2025. ↗
  71. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer, 2021. URL https://arxiv.org/abs/2010.11934. ↗
  72. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. A new dataset and method for automatically grading esol texts. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 180–189, 2011.
  73. Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatistics, 7, 2005. ↗
  74. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. ↗
  75. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023.
  76. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/ abs/2311.07911. ↗
  77. Runtao Zhou, Guangya Wan, Saadia Gabriel, Sheng Li, Alexander J Gates, Maarten Sap, and Thomas Hartvigsen. Disparities in llm reasoning accuracy and explanations: A case study on african american english. arXiv preprint arXiv:2503.04099, 2025. ↗
  78. Xuhui Zhou. Challenges in automated debiasing for toxic language detection. University of Washington, 2020.
  79. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. Value: Understanding dialect disparity in nlu. arXiv preprint arXiv:2204.03031, 2022. ↗
  80. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. Multi-value: A framework for cross-dialectal english nlp. arXiv preprint arXiv:2212.08011, 2022. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2025-trans-env-framework-evaluating-linguistic/