Elamur Research⌕
☰
home / papers / 2025 / DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation

DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation

Paper · 2025 · Yu Zhou, Sohyun An, Haikang Deng et al.

Summary

Text-to-image and text-to-video models are trained mostly on Standard American and British English. This paper asks how they cope when a prompt uses another English dialect. The authors built DialectGen, a benchmark covering SAE plus five other dialects: AAE, BrE, ChE, InE and SgE.

How the benchmark was built

  • Dialect words were taken from regional dictionaries and each was matched to an SAE synonym.
  • GPT4o wrote scene prompts around the SAE word, and the dialect word was then swapped in.
  • Dialect speakers, checked by self-report and a quiz, kept only pairs that were exactly synonymous, natural in the dialect, and unambiguous.

What they found

  • Across 17 generators, replacing a single word with its dialect form cut performance by roughly a third to nearly a half.
  • Prompt rewriting and fine-tuning recovered little.

Their fix

  • The authors retrain only the text encoder with three objectives: learning dialect words, keeping SAE senses of polysemous words, and a KL term that keeps general behaviour stable.
  • On Stable Diffusion 1.5, dialect scores rise to SAE level while SAE scores stay almost unchanged.

This matters because dialect speakers get worse results from generative tools.

Key points

  • Dataset construction:
    • 1126 dialect lexemes were collected from dictionaries, including the OED regional database, DARE and dictionaries of Singlish, Indian English and AAE.
    • 6552 prompts were generated in Concise (≤6 words) and Detailed (≥9 words) versions.
    • Speaker filtering removed 35.9% of them, leaving 4,200 validated prompts.
  • Measured drops: 17 image and video models lose 32.26%–48.17% when one dialect word appears in the prompt. Open-weights models drop up to 38.63% for images and 48.17% for video.
  • Baselines:
    • Prompt rewriting and decoder fine-tuning improve dialect scores by under 7%.
    • Fine-tuning also damages SAE and polysemy results.
  • Proposed method on SD1.5:
    • Average dialect score reaches 77.68%, against an SAE baseline of 77.91% (+34.4% overall).
    • SAE performance on MSCOCO falls by less than 1%.
    • It was also tested on SDXL and Wan 2.1.
  • Polysemy Control loss: adding it raises polysemy performance by 17.43% with Text KL Reg. and 17.76% with Image KL Reg.

Interesting ideas

  • I1. Build test pairs that differ in exactly one lexeme, so any change in output can be blamed on that word alone.
  • I2. Accept a pair only if two independently qualified judges both approve it on validity and both rule out ambiguity, and qualify judges by a self-report plus a quiz that must agree.
  • I3. While teaching a new sense of a word, pin the representation of sentences that use its standard sense to the frozen original model, so the old meaning is not overwritten.
  • I4. Turn continuous embeddings into a probability distribution by scoring them against a fixed set of reference anchors, then constrain drift with KL instead of plain cosine distance; the paper shows plain cosine regularization recovers far less.
  • I5. Evaluate every input in a terse low-context version and a richer high-context version, measuring how much context rescues an unknown word.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Feed readers fable or test texts where one word is swapped for a BrE/AAE/InE synonym and check that the answers stay identical.
  • I2 (learn): Apply the same double-agreement gate to silver data, admitting a parse only when separately validated witness parsers concur.
  • I3 (frames): When adding dialect senses (e.g., "whip" = car) to role or frame lexicons, keep a held set of standard-sense sentences whose frame output must not change.
  • I4 (learn): When retraining gparse, compare similarity distributions over a fixed reference sentence set between old and new encoders to detect and penalize general regressions.
  • I5 (tokens): Test whether the tagger and lemmatizer resolve unfamiliar dialect words better with richer context, which can show where a dialect-to-SAE lemma map is needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang and Nanyun Peng. 2025. DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation. arXiv:2510.14949.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.14949 · arXiv 2510.14949

@misc{zhou2025dialectgen,
  title = {{DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation}},
  author = {Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang and Nanyun Peng},
  year = {2025},
  doi = {10.48550/arxiv.2510.14949},
  eprint = {2510.14949},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.14949}
}

References

60 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nur Aeni, Like Raskova Octaberlina, Nenni Dwi Aprianti Lubis, et al. A literature review of English Language Variation on Sociolinguistics. OSF, 2021. find
  2. Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. ↗
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610–623, 2021. find
  4. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn.openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. ↗
  5. Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023. ↗
  6. Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024. ↗
  7. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. Twitter universal dependency parsing for african-american and mainstream american english. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1415– 1425, 2018.
  8. Frederic G Cassidy. Dictionary of American Regional English: IO. Belknap Press of Harvard University Press, 1985. find
  9. Jack K Chambers and Peter Trudgill. Dialectology. Cambridge University Press, 1998. find
  10. Zijie Chen, Lichao Zhang, Fangsheng Weng, Lili Pan, and Zhenzhong Lan. Tailored visions: Enhancing text-to-image generation with personalized prompt rewriting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7727–7736, 2024. find
  11. Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint arXiv:2304.09151, 2023. ↗
  12. Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400, 2023. ↗
  13. David Crystal. English as a Global Language. Cambridge University Press, Cambridge, 2 edition, 2003. ISBN 9780521823470. ↗
  14. Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. ↗
  15. Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, and Jiaming Song. Personalized preference fine-tuning of diffusion models. arXiv preprint arXiv:2501.06655, 2025. ↗
  16. Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phúc Lê Khac, Luke Melas, and Ritobrata Ghosh. Dall·e mini, 7 2021. URL https://github.com/ borisdayma/dalle-mini. ↗
  17. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL https://arxiv.org/abs/2403.03206, 2, 2024. ↗
  18. Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858–79885, 2023. ↗
  19. VA Fromkin, Robert Rodman, and V Hyams. An introduction to language 6e. Orlando, FL: Hartcourt Brace College Publishers, 1998. find
  20. Scott Geng, Cheng-Yu Hsieh, Vivek Ramanujan, Matthew Wallingford, Chun-Liang Li, Pang Wei W Koh, and Ranjay Krishna. The unmet promise of synthetic training images: Using retrieved real images performs better. Advances in Neural Information Processing Systems, 37:7902–7929, 2024. find
  21. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. ↗
  22. Suchin Gururangan, Dallas Card, Sarah K Dreier, Emily K Gade, Leroy Z Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A Smith. Whose language counts as high quality? measuring language ideologies in text data selection. arXiv preprint arXiv:2201.10474, 2022. ↗
  23. Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36:66923–66939, 2023. find
  24. Jennifer KN Heinmiller. Compiling the oxford dictionary of african american english: A progress report. Dictionaries: Journal of the Dictionary Society of North America, 44(1):91–104, 2023. ↗
  25. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Joseph Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Conference on Empirical Methods in Natural Language Processing, 2021. ↗
  26. Dirk Hovy and Shannon L Spruit. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 591–598, 2016. ↗
  27. Richard A Hudson. Sociolinguistics. Cambridge university press, 1996. find
  28. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. ↗
  29. Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, et al. Openclip. Zenodo, 2021. find
  30. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. Challenges of studying and processing dialects in social media. In Proceedings of the workshop on noisy user-generated text, pp. 9–18, 2015. find
  31. Jack Tsen-Ta Lee. A Dictionary of Singlish and Singapore English. Lee, Jack Tsen-Ta, 2004. Accessed 2025-05-16. find
  32. Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36: 69981–70011, 2023. find
  33. Ran Li, Xiaomeng Jin, et al. Real: Realism evaluation of text-to-image generation models for effective data augmentation. arXiv preprint arXiv:2502.10663, 2025. ↗
  34. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014. ↗
  35. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366–384. Springer, 2024. find
  36. John Nerbonne. Data-driven dialectology. Language and linguistics compass, 3(1):175–198, 2009. ↗
  37. OpenAI. Introducing 4o image generation. https://openai.com/index/ introducing-4o-image-generation/, 2025. Accessed: 2025-05-19. ↗
  38. Karl Pearson. Notes on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240–242, 1895. doi: 10.1098/rspl.1895.0041. ↗
  39. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. ↗
  40. Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023. ↗
  41. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021a. ↗
  42. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021b. ↗
  43. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Dall·e 2: A new ai system that can create realistic images and art from a description in natural language. https://openai.com/research/dall-e-2, 2022. ↗
  44. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022. ↗
  45. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278–25294, 2022.
  46. Eyal Segalis, Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. A picture is worth a thousand words: Principled recaptioning improves image generation. ArXiv, abs/2310.16656, 2023. URL https://api.semanticscholar.org/CorpusID:266003242. ↗
  47. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. ↗
  48. V. Subhash. Dictionary of Indian English. V. Subhash, Hyderabad, 2020. ISBN 9789354374487. find
  49. Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228–8238, 2024. find
  50. Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang. Survey of bias in text-to-image generation: Definition. Evaluation, and Mitigation, 2024. find
  51. Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. ↗
  52. Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. Promptcharm: Text-to-image generation through multi-modal prompting and refinement. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1–21, 2024. find
  53. Ronald Wardhaugh and Janet M Fuller. An introduction to sociolinguistics. John Wiley & Sons, 2021. find
  54. Noah Webster. An American dictionary of the English language. Merriam, 1869. find
  55. Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. ↗
  56. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975–11986, 2023.
  57. Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836–3847, 2023. ↗
  58. Yu Zhou, Bingxuan Li, Mohan Tang, Xiaomeng Jin, Te-Lin Wu, Kuan-Hao Huang, Heng Ji, Kai-Wei Chang, and Nanyun Peng. Contrastive visual data augmentation. In ICML 2025, 2025. ↗
  59. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Brooke Anderson, and Diyi Yang. Value: Understanding dialect disparity in nlu. In Annual Meeting of the Association for Computational Linguistics, 2022.
  60. Caleb Ziems, William B. Held, Jingfeng Yang, and Diyi Yang. Multi-value: A framework for cross-dialectal english nlp. ACL 2023, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2025-dialectgen-benchmarking-improving-dialect-robustness/