Elamur Research⌕
☰
home / papers / 2026 / Improving Dialect Robustness in Large Language Models via LoRA and Mixture-of-Experts

Improving Dialect Robustness in Large Language Models via LoRA and Mixture-of-Experts

Summary

Large language models are largely trained on mainstream, mostly Standard American, English. As a result, they perform worse on regional varieties with different vocabulary, grammar and style. Earlier work mainly measured this gap. This paper tries to reduce it with two parameter-efficient fine-tuning schemes, tested on the BESSTIE dataset. BESSTIE covers Australian, British and Indian English, with sentiment and sarcasm labels drawn from Google Places reviews and Reddit.

  • DialectFusion trains one LoRA adapter per variety and then combines them. It either learns per-layer mixing weights (CAT) or prunes the adapters and merges them by sign agreement (TIES).
  • DialectMoE builds on MixLoRA. It places several LoRA experts inside the feed-forward blocks behind a top-k router, so that dialect specialisation can emerge within a single model.

The authors evaluate five open instruct models in zero- and few-shot settings. Both schemes outperform plain instruction-tuned models and LoRA trained on pooled data without dialect labels. This matters for deployment: one merged or routed model can handle mixed or unidentified dialect input without choosing an adapter at inference time.

Key points

  • Models: Mistral-v0.1 7B Instruct, Gemma2-9B Instruct and Phi-3 Medium Instruct (mostly English-trained), plus Llama 3.1 8B Instruct and Qwen 2.5 7B Instruct (multilingual). Each is evaluated against six methods.
  • Method comparison:
    • CAT gave the most stable gains among the merging methods.
    • DialectMoE was strongest in zero-shot, especially for Qwen 2.5, Llama 3.1 and Mistral-7B, but weakened in few-shot.
    • LoRA Separate (one adapter per dialect) was robust in few-shot but requires knowing the dialect in advance.
  • Implementation: the authors used the MoE-PEFT library with a learning rate of 1e-5 on A100 GPUs (40GB and 80GB). Code is available on GitHub (Sanjh-Maheshwari/LLM-Dialect-Robustness).
  • Limitations: error analysis shows all methods lean on surface lexical cues and miss culture-specific intent, such as a PubG reference or Indian English slang. DialectMoE also needs custom code for unsupported models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.vardial-1.24

@inproceedings{maheshwari2026improving,
  title = {{Improving Dialect Robustness in Large Language Models via LoRA and Mixture-of-Experts}},
  author = {Maheshwari, Sanjh and Rajpoot, Aniket Singh and Cocarascu, Oana and Mamta},
  booktitle = {Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2026},
  doi = {10.18653/v1/2026.vardial-1.24},
  url = {https://doi.org/10.18653/v1/2026.vardial-1.24}
}

References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. arcee-ai. 2024. The-tome: A curated dataset for instruction-following language models. https://huggingface.co/datasets/arcee-ai/The-Tome. Accessed: 2025-12-10. ↗
  2. Fei Ding and Baiqiao Wang. 2025. Improved supervised fine-tuning for large language models to mitigate catastrophic forgetting. arXiv e-prints, pages arXiv– 2506. find
  3. Abdin et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Technical Report MSR-TR-2024-12, Microsoft. ↗ our card
  4. In this paper, we evaluated the performance of five open-source LLMs across Australian, British, Fahim Faisal and Antonios Anastasopoulos. 2024. Data-augmentation-based dialectal adaptation for LLMs. and Indian English dialects, and proposed two In Proceedings of the Eleventh Workshop on NLP parameter-efficient frameworks based on LoRA for Similar Languages, Varieties, and Dialects (Var-adapters and mixture of experts to enhance their Dial 2024), pages 197–208, Mexico City, Mexico. robustness. Our first approach, DialectFusion in-Association for Computational Linguistics. 301 find
  5. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  6. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  7. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2025. Natural language processing for dialects of a language: A survey. ACM Comput. Surv., 57(6). ↗
  8. Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. 2024. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. CoRR, abs/2404.15159. ↗
  9. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, and Furu Wei. 2025. Assessing dialect fairness and robustness of large language models in reasoning tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6317–6342, Vienna, Austria. Association for Computational Linguistics. ↗
  10. Yanchen Liu, William Held, and Diyi Yang. 2023. DADA: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793, Singapore. Association for Computational Linguistics. find our card
  11. Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. 2025. Lora soups: Merging loras for practical skill composition tasks. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 644–655. ↗
  12. Dipankar Srirag, Aditya Joshi, Jordan Painter, and Diptesh Kanojia. 2025a. BESSTIE: A benchmark for sentiment and sarcasm classification for varieties of English. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8413–8429, Vienna, Austria. Association for Computational Linguistics. find
  13. Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi. 2025b. Evaluating dialect robustness of language models via conversation understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 24–38, Abu Dhabi. Association for Computational Linguistics. ↗
  14. Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann. 2023. Dialect-robust evaluation of generated text. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6010–6028, Toronto, Canada. Association for Computational Linguistics. ↗
  15. Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118. ↗
  16. Sowmya Vajjala and Shwetali Shimangaud. 2025. Text classification in the LLM era - where do we stand? CoRR, abs/2502.11830. ↗
  17. Gido M. van de Ven, Nicholas Soures, and Dhireesha Kudithipudi. 2025. Continual learning and catastrophic forgetting, page 153–168. Elsevier. find
  18. Sajana Weerawardhena, Paul Kassianik, Blaine Nelson, Baturay Saglam, Anu Vellore, Aman Priyanshu, Supriti Vijay, Massimo Aufiero, Arthur Goldblatt, Fraser Burch, Ed Li, Jianliang He, Dhruv Kedia, Kojin Oshiba, Zhouran Yang, Yaron Singer, and Amin Karbasi. 2025. Llama-3.1-foundationai-securityllm-8b-instruct technical report. CoRR, abs/2508.01059. ↗
  19. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems, volume 36, pages 7093–7115. Curran Associates, Inc. find our card
  20. An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, and 9 others. 2025. Qwen2.5-1m technical report. CoRR, abs/2501.15383. ↗
  21. Yazhou Zhang, Chunwang Zou, Zheng Lian, Prayag Tiwari, and Jing Qin. 2025. Sarcasmbench: Towards evaluating large language models on sarcasm understanding. IEEE Transactions on Affective Computing, 16(4):2560–2578. ↗
  22. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A framework for cross-dialectal English NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/maheshwari-2026-improving-dialect-robustness-large-language/