Elamur Research⌕
☰
home / papers / 2026 / DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

Paper · 2026 · Jason S Lucas, Matt Murtagh White, Ali Al-Lawati et al.

Summary

Detectors of harmful content, especially disinformation classifiers, are usually built and tested on Standard American English (SAE). It is unclear whether they keep working when the same false claim appears in another English dialect. The authors built DIA-HARM, a benchmark that tests this across 50 English varieties from the U.S., Britain, Africa, the Caribbean and the Asia-Pacific region. They used Multi-VALUE's rule-based morphosyntactic transformations to convert 9 existing SAE disinformation datasets into dialectal versions. The result is the D-CUBE corpus, filtered by a quality pipeline called D-PURIFY. They evaluated 16 detectors (fine-tuned encoders, classic deep learning models and zero-shot LLMs) in four settings:

  • SAE-only training
  • dialect-aware training
  • cross-dialect transfer
  • architecture comparison

Dialectal rewriting of human-written content consistently lowers detection quality, while AI-generated content is barely affected. Zero-shot LLMs perform far worse than fine-tuned models. The work matters because it shows that speakers of non-SAE varieties may be less protected from disinformation, or more often wrongly flagged.

Key points

  • D-CUBE corpus: about 195K samples covering 50 dialects, derived from 9 benchmarks (e.g., GossipCop, PolitiFact, CoAID). The corpus, D-PURIFY, evaluation code and checkpoints are released. The adversarial generation pipeline is withheld.
  • Fine-tuned models: F1 drops by 1.4–3.6% on human-written dialectal text. Zero-shot LLMs can lose up to 27%. Some models degrade by more than 33% on mixed human/AI content.
  • Best-case dialect F1: 96.6% for fine-tuned encoders versus 78.3% for Mistral-7B in zero-shot mode. Zero-shot LLMs often refused to answer, with abstention rates of 58–98%.
  • Cross-dialect transfer: across 2,450 dialect pairs, mDeBERTa averaged 97.2% F1, while RoBERTa and XLM-RoBERTa broke down on dialectal inputs.
  • Hardest and most robust dialects: Maltese, Australian Vernacular and Southeast England English caused the largest drops. Ghanaian and Manx English stayed robust.
  • Main limitation: the metrics are SAE-trained, which may bias quality filtering; for example, Chicano English had a 71.4% pass rate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🎓Pennsylvania State UniversityState College · United States56 works · 5,360 citations🔬MIT Lincoln LaboratoryLexington · United States4 works

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksBias, fairness & safety · 517 works

Cite

Cite as Jason S Lucas, Matt Murtagh White, Ali Al-Lawati, Uchendu Uchendu, Adaku Uchendu and Dongwon Lee. 2026. DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.144

@inproceedings{lucas2026dia,
  title = {{DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects}},
  author = {Jason S Lucas, Matt Murtagh White, Ali Al-Lawati, Uchendu Uchendu, Adaku Uchendu and Dongwon Lee},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.144},
  url = {https://doi.org/10.18653/v1/2026.acl-long.144}
}

References

51 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdullah Marish Ali, Fuad A Ghaleb, Bander Ali Saleh Al-Rimy, Fawaz Jaber Alsolami, and Asif Irshad Khan. 2022. Deep ensemble fake news detection model using sequential deep learning technique. Sensors, 22(18):6970. ↗
  2. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics. ↗
  3. Nida Aslam, Irfan Ullah Khan, Farah Salem Alotaibi, Lama Abdulaziz Aldaej, and Asma Khaled Al-dubaikil. 2021. Fake detect: A deep learning ensemble model for fake news detection. Complexity, 2021:1–8. ↗
  4. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  5. Su Lin Blodgett and Jacob Eisenstein. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computational Linguistics.
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  7. Limeng Cui and Dongwon Lee. 2020. Coaid: Covid-19 healthcare misinformation dataset. Preprint, arXiv:2006.00885. ↗
  8. Limeng Cui, Haeseung Seo, Maryam Tabar, Fenglong Ma, Suhang Wang, and Dongwon Lee. 2020. DE-TERRENT: Knowledge guided graph attention network for detecting healthcare misinformation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 492–502, New York, NY, USA. Association for Computing Machinery. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  11. Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. DIALECTBENCH: An NLP benchmark for dialects, varieties, and closely-related languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14412–14454, Bangkok, Thailand. Association for Computational Linguistics. find
  12. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. The Innovation. find
  13. Benjamin D. Horne, Jeppe Nørregaard, and Sibel Adalı. 2019. Robust fake news detection over time and attack. ACM Transactions on Intelligent Systems and Technology, 11(1). ↗
  14. Nirosh Jayakody, Azeem Mohammad, and Malka N Halgamuge. 2022. Fake news detection using a decentralized deep learning model and federated learning. In IECON 2022 – 48th Annual Conference of the IEEE Industrial Electronics Society, pages 1–6. IEEE. find
  15. Anjali Kantharuban, Ivan Vulić, and Anna Korhonen. 2023. Quantifying the dialect gap and its correlates across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7226–7245, Singapore. Association for Computational Linguistics. find
  16. Vijay Keswani and L Elisa Celis. 2021. Dialect diversity in text summarization on twitter. In Proceedings of the web conference 2021, pages 3802–3814. find
  17. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. 2020a. eWAVE. find
  18. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret. 2020b. eWAVE: The electronic world atlas of varieties of English. In The Handbook of English Linguistics, pages 613–635. John Wiley & Sons. find
  19. Yichuan Li, Bohan Jiang, Kai Shu, and Huan Liu. 2020. Mm-covid: A multilingual and multimodal data repository for combating covid-19 disinformation. Preprint, arXiv:2011.04088. ↗
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  21. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the logical reasoning ability of ChatGPT and GPT-4. arXiv preprint arXiv:2304.03439. ↗
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  23. Jason Lucas, John Chen, Ali Al-Lawati, Mahjabin Nahar, and Mahnoosh Mehrabani. 2025. Chain-of-interactions: Multi-step iterative ICL framework for abstractive task-oriented dialogue summarization of conversational AI interactions. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 3560–3599, Suzhou, China. Association for Computational Linguistics. find
  24. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8018–8025. Association for the Advancement of Artificial Intelligence.
  25. Jason Lucas, Adaku Uchendu, Michiharu Yamashita, Jooyoung Lee, Shaurya Rohatgi, and Dongwon Lee. 2023. Fighting fire with fire: The dual role of LLMs in crafting and detecting elusive disinformation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14279–14305, Singapore. Association for Computational Linguistics. find
  26. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2025. Natural language processing for dialects of a language: A survey. ACM Computing Surveys, 57(6):1–37. ↗
  27. Jason S. Lucas, Barani Maung Maung, Maryam Tabar, Keegan McBride, and Dongwon Lee. 2024a. The longtail impact of generative ai on disinformation: Harmonizing dichotomous perspectives. IEEE Intelligent Systems, 39(5):12–19. ↗
  28. Jason S Lucas, Barani Maung Maung, Maryam Tabar, Keegan McBride, and Dongwon Lee. 2024b. The longtail impact of generative ai on disinformation: Harmonizing dichotomous perspectives. IEEE Intelligent Systems, 39(5):12–19. ↗
  29. Jing Ma, Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation structure via kernel learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 708–717, Vancouver, Canada. Association for Computational Linguistics. ↗
  30. Barani Maung Maung, Keegan McBride, Jason S. Lucas, Maryam Tabar, and Dongwon Lee. 2024. Generative ai disproportionately harms long tail users. Computer, 57(11):82–85. ↗
  31. Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Maarten Sap. 2025. Rejected dialects: Biases against african american language in reward models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7468–7487. find
  32. Van-Hoang Nguyen, Kazunari Sugiyama, Preslav Nakov, and Min-Yen Kan. 2020. FANG: Leveraging social context for fake news detection using graph representation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, pages 1165–1174, New York, NY, USA. Association for Computing Machinery. find
  33. Ebuka Okpala, Long Cheng, Nicodemus Mbwambo, and Feng Luo. 2022. Aaebert: Debiasing bert-based hate speech detection models via adversarial learning. In 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1606–1612. IEEE. find
  34. Aidan Peppin, Julia Kreutzer, Alice Schoenauer Sebag, Kelly Marchisio, Beyza Ermis, John Dang, Samuel Cahyawijaya, Shivalika Singh, Seraphina Goldfarb-Tarrant, Viraat Aryabumi, and 1 others. 2025. The multilingual divide and its impact on global ai safety. arXiv preprint arXiv:2505.21344. ↗
  35. Matúš Pikuliak, Ivan Srba, Robert Moro, Timo Hro-madka, Timotej Smoleň, Martin Melišek, Ivan Vykopal, Jakub Simko, Juraj Podroužek, and Maria Bielikova. 2023. Multilingual previously fact-checked claim retrieval. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16477–16500, Singapore. Association for Computational Linguistics. Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. find
  36. Kevin Scannell. 2020. Universal Dependencies for Manx Gaelic. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 152–157, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  37. Qiang Sheng, Juan Cao, H. Russell Bernard, Kai Shu, Jintao Li, and Huan Liu. 2022. Characterizing multi-domain false news and underlying user effects on Chinese Weibo. Information Processing & Management, 59(4):102959. ↗
  38. Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019. dEFEND: Explainable fake news detection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, pages 395–405, New York, NY, USA. Association for Computing Machinery. find
  39. Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big Data, 8(3):171–188. ↗
  40. Cagri Toraman, Oguzhan Ozcelik, Furkan Sahinuc, and Fazli Can. 2024. MiDe22: An annotated multi-event tweet dataset for misinformation detection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11283– 11295, Torino, Italia. ELRA and ICCL. find
  41. Bibek Upadhayay and Vahid Behzadan. 2022. Hybrid deep learning model for fake news detection in social networks (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 13067–13068. find
  42. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver, Canada. Association for Computational Linguistics.
  43. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc.
  44. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476. ↗
  45. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, volume 32. ↗
  46. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics.
  47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTscore: Evaluating text generation with BERT. In International Conference on Learning Representations. ↗
  48. Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023. Synthetic lies: Understanding AI-generated misinformation and evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery. find
  49. Runtao Zhou, Guangya Wan, Saadia Gabriel, Sheng Li, Alexander J Gates, Maarten Sap, and Thomas Hartvigsen. 2025. Disparities in llm reasoning accuracy and explanations: A case study on african american english. arXiv preprint arXiv:2503.04099. ↗
  50. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics. ↗
  51. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A framework for cross-dialectal English NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lucas-2026-dia-harm-dialectal-disparities-harmful/