Elamur Research⌕
☰
home / papers / 2025 / Are Dialects Better Prompters? A Case Study on Arabic Subjective Text Classification

Are Dialects Better Prompters? A Case Study on Arabic Subjective Text Classification

Paper · 2025 · Leila Moudjari and Farah Benamara

Summary

The authors ask whether instructing LLMs in a regional Arabic dialect helps them classify subjective text better than instructions in English or Modern Standard Arabic (MSA). Earlier work found that English prompts usually win for low-resource languages, and that Arabic LLMs struggle with nuanced tasks. The study covers four tasks: sentiment, emotion, irony and crisis-related tweets. It uses eight public datasets in MSA, Algerian, Tunisian, Levantine, Gulf and mixed varieties. Twelve open LLMs (general multilingual, Arabic-centric and affect-specialised) are tested in zero-shot mode and again after QLoRA fine-tuning, with BERT-style Arabic encoders as baselines. Prompts vary along two axes: language (English, MSA, Levantine, Algerian, Tunisian) and script (Arabic or Arabizi). After fine-tuning, dialectal prompts in Arabic script or Arabizi generally give the best macro-F1, and Arabic-centric models are especially effective. Fine-tuned LLMs beat the smaller encoders. However, moving to another dataset, or to a different prompt variety at inference, hurts results noticeably. The work suggests that low-resource prompting should not default to English or translated instructions.

Key points

  • Scale: 12 open LLMs (e.g. SOLAR 10.7B, Mistral 7b, LLaMA3 8b, Jais, AceGPT, EmoLlama) are compared against AraBERTv2, CAMeLBERT and DarijaBERT. The evaluation spans 4 tasks and 8 datasets, including OCA, Sem17, Sem18, Twifil-sent/emo, TSAC, IDAT and Flood.
  • Prompt variants: prompts come in English, MSA (ChatGPT-translated, then checked by a native speaker), and hand-translated dialect versions.
    • Levantine is tested in both Arabic script and Arabizi.
    • Algerian is tested in Arabizi only, and Tunisian in Arabic script only.
  • Evaluation design: four settings are used: in-dataset, a dialect-only in-dataset subset filtered by ALDi score, cross-dataset transfer, and fine-tuning transfer. Scores are macro-F1 averaged over three runs.
  • Script matters: the best prompt tends to follow the dataset's dominant script.
    • Sem18 is about 83% Latin script, and Levantine-Arabizi prompting worked best there.
    • On Twifil-emo, Levantine-Arabizi reached 35.81 after fine-tuning.
  • Caveats and release: ALDi proved unreliable on Arabizi text, and possible data contamination is acknowledged for OCA and TSAC. The fine-tuned models are released on Hugging Face as DialectPrompting/Dialectal_LLMs.

Interesting ideas

  • I1. Writing the instruction in the same script and register as the input data, instead of a "prestige" standard, can improve a fine-tuned model's alignment with noisy, informal text.
  • I2. A "fine-tuning transfer" test changes only the prompt variety at inference time. This exposes how tightly a tuned model is bound to its training conditions.
  • I3. The authors manually checked whether an automatic dialect score (ALDi) really matched the data, and found it misleading on Arabizi. This is a reminder to validate any routing metric before relying on it.
  • I4. Restricting outputs to the label set, with a fallback (neutral if available, otherwise a random wrong label), makes abstentions cost something rather than silently inflating scores.
  • I5. A supplication asking for rain was labelled sadness under English/MSA prompts but trust under the dialect prompt. This shows that religious or formulaic speech carries culturally specific stance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher prompts for silver data could be phrased in the register of the source text (informal web, children's tales) to see whether teacher labels become more consistent.
  • I2 (learn): add evaluations where only the input register or style shifts, as a negative control showing whether learned repair rules are robust or overfitted.
  • I3 (tree): before choosing a parser or applying UD dialect conversion based on an automatic register or variety detector, spot-check the detector on the hard cases.
  • I4 (answer): school-test readers could have refusals and off-label answers scored explicitly as errors, so accuracy cannot be gamed by not answering.
  • I5 (frames): treat prayers and blessings as a distinct speech act whose default emotional stance is hopeful, so they are not misread from surface words of need or hardship.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🏛️Agency for Science, Technology and ResearchSingapore · Singapore30 works · 14,235 citations🔬Institut de Recherche en Informatique de ToulouseToulouse · France35 works · 1,872 citations🎓Université de ToulouseToulouse · France16 works · 1,999 citations🔬Image and Pervasive Access LaboratorySingapore · Singapore2 works · 3 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Leila Moudjari and Farah Benamara. 2025. Are Dialects Better Prompters? A Case Study on Arabic Subjective Text Classification. Findings of the Association for Computational Linguistics: ACL 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.892

@inproceedings{moudjari2025are,
  title = {{Are Dialects Better Prompters? A Case Study on Arabic Subjective Text Classification}},
  author = {Leila Moudjari and Farah Benamara},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.892},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.892}
}

References

43 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, et al. 2024. Larabench: Benchmarking arabic ai with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520. ↗
  2. AhmedElmogtaba Abdelmoniem Ali Abdelaziz, Ashraf Hatim Elneima, and Kareem Darwish. 2024. LLM-based MT data creation: Dialectal to MSA translation shared task. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 112–116, Torino, Italia. ELRA and ICCL. find
  3. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics. ↗
  4. AI@Meta. 2024. Llama 3 model card. find
  5. Alaa Alharbi and Mark Lee. 2019. Crisis detection from arabic tweets. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 72–79.
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  7. Patrick Bareiß, Roman Klinger, and Jeremy Barnes. 2024. English prompts are better for nli-based zero-shot emotion classification than target-language prompts. In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 1318–1326, New York, NY, USA. Association for Computing Machinery.
  8. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, virtual+Dublin. Association for Computational Linguistics. ↗
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  10. Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. LLMs are few-shot in-context low-resource language learners. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 405–433, Mexico City, Mexico. Association for Computational Linguistics. ↗
  11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  13. Kamel Gaanoun, Abdou Mohamed Naira, Anass Allak, and Imade Benelallam. 2024. Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics, pages 1–13. ↗
  14. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat@fire2019: Overview of the track on irony detection in arabic tweets. In Mehta P., Rosso P., Majumder P., Mitra M. (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019). CEUR Workshop Proceedings. In: CEUR-WS.org, Kolkata, India, December 12-15, pages 10– 13. find
  15. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023a. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365– 12394, Singapore. Association for Computational Linguistics.
  16. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  17. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023b. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  19. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  20. Amr Keleg, Walid Magdy, and Sharon Goldwater. 2024. Estimating the level of dialectness predicts inter-annotator agreement in multi-dialect Arabic datasets. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778–789, Bangkok, Thailand. Association for Computational Linguistics.
  21. Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2023. Solar 10.7b: Scaling large language models with simple yet effective depth upscaling. Preprint, arXiv:2312.15166. ↗
  22. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics ACL 2024, pages 5622–5640, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. ↗
  23. Xuan-Phi Nguyen, Mahani Aljunied, Shafiq Joty, and Lidong Bing. 2024. Democratizing LLMs for low-resource languages by leveraging their English dominant abilities with linguistically-diverse prompts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3501–3516, Bangkok, Thailand. Association for Computational Linguistics.
  24. Pingsheng Liu, Zhengjie Huang, Xiechi Zhang, Linlin Wang, Gerard de Melo, Xin Lin, Liang Pang, and Liang He. 2023. A disentangled-attention based framework with persona-aware prompt learning for dialogue generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13255–13263. find
  25. Viktória Ondrejová and Marek Šuppa. 2024. Can LLMs handle low-resource dialects? a case study on translation and common sense reasoning in šariš. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 130–139, Mexico City, Mexico. Association for Computational Linguistics. find
  26. Zhiwei Liu, Kailai Yang, Qianqian Xie, Tianlin Zhang, and Sophia Ananiadou. 2024. Emollms: A series of emotional large language models and annotation tools for comprehensive affective analysis. KDD ’24, page 5487–5496, New York, NY, USA. Association for Computing Machinery. find
  27. Benedikt Perak, Slobodan Beliga, and Ana Meštrović. 2024. Incorporating dialect understanding into LLM using RAG and prompt engineering techniques for causal commonsense reasoning. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 220–229, Mexico City, Mexico. Association for Computational Linguistics. find
  28. Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. 2024. Are emergent abilities in large language models just in-context learning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5098–5139, Bangkok, Thailand. Association for Computational Linguistics. ↗
  29. Salima Medhaffar, Fethi Bougares, Yannick Estève, and Lamia Hadrich-Belguith. 2017. Sentiment analysis of tunisian dialects: Linguistic ressources and experiments. In Proceedings of the third Arabic natural language processing workshop, pages 55–61.
  30. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.
  31. Leila Moudjari, Karima Akli-Astouati, and Farah Benamara. 2020. An Algerian corpus and an annotation platform for opinion and emotion analysis. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1202–1210, Marseille, France. European Language Resources Association. find
  32. Leila Moudjari, Farah Benamara, and Karima Akli-Astouati. 2021. Multi-level embeddings for processing arabic social media contents. Computer Speech & Language, 70:101240. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. ↗
  33. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, UAE. Association for Computational Linguistics.
  34. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502–518, Vancouver, Canada. Association for Computational Linguistics.
  35. Mohammed Rushdi-Saleh, M Teresa Martín-Valdivia, L Alfonso Ureña-López, and José M Perea-Ortega. 2011. Oca: Opinion corpus for arabic. Journal of the American Society for Information Science and Technology, 62(10):2045–2054. ↗
  36. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  37. Bhavani Shankar, Preethi Jyothi, and Pushpak Bhattacharyya. 2024. In-context mixing (ICM): Code-mixed prompts for multilingual LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4162–4176, Bangkok, Thailand. Association for Computational Linguistics. find
  38. Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005, Singapore. Association for Computational Linguistics.
  39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  40. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research. Survey Certification. ↗
  41. Mohammad Ali Yaghan. 2008. Arabizi: A contemporary style of arabic slang. Design issues, 24:39–52. ↗
  42. Chiyu Zhang, Khai Doan, Qisheng Liao, and Muhammad Abdul-Mageed. 2023. The skipped beat: A study of sociopragmatic understanding in LLMs for 64 languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2630–2662, Singapore. Association for Computational Linguistics. find
  43. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024. Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881–3906, Mexico City, Mexico. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moudjari-2025-are-dialects-better-prompters-case/