Elamur Research⌕
☰
home / papers / 2025 / From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher

From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher

Paper · 2025 · Renad Al-Monef, Hassan Alhuzali, Nora Alturayeif et al.

Summary

The paper examines how well current large language models handle Saudi Arabic dialects and the cultural knowledge carried in them. Most Arabic evaluation focuses on Modern Standard Arabic, so regional speech is rarely tested. The authors build Absher, a benchmark of 18,564 multiple-choice questions. The questions are drawn from a curated collection of dialectal words, phrases and proverbs from five Saudi regions, plus a general category. Items were produced by prompting LLMs to write questions, and native speakers from each region then reviewed them in several stages. The questions fall into six formats, covering literal meaning, usage in context, cultural reading and regional origin. Six open models were tested zero-shot. Qwen scored best overall, while ALLaM did best on proverbs. Every model was weaker on Southern and Eastern material and on True/False items that give no context. The results show that strong general performance does not guarantee dialect grounding. They also show that evaluation needs to be sliced by region and by type of linguistic unit.

Key points

  • Absher contains 18,564 MCQs split across six question types: identifying an expression's meaning, judging statements as true or false, filling gaps in sentences, choosing the right usage in context, interpreting cultural sense, and recognizing which region an expression comes from.
  • Source items are words, phrases and proverbs from the Central, Western, Southern, Eastern and Northern regions. Native speakers from each region validated them.
  • Six models were evaluated zero-shot: ALLaM, LLaMA, Jais, Mistral, Qwen and AceGPT.
    • Qwen led overall and reached about 63% on word-level items.
    • ALLaM led on proverb interpretation with about 48%.
  • Multilingual models (Qwen, Mistral) beat Arabic-native ones on general tasks. Arabic-native models had an edge on culturally loaded expressions.
  • Absher is positioned against SaudiCulture, which has only 441 questions, mostly in MSA. The dataset is promised for public release upon acceptance.

Interesting ideas

  • I1. Each result is broken down along three axes at once (unit size × question format × region), so a single average cannot hide a weak slice.
  • I2. The same source item is probed through several formats, such as gloss, binary judgment, cloze and use in context; context-free yes/no questions turned out to be especially fragile.
  • I3. A cheap generator drafts the questions, and region-specific native speakers then vet them in stages; this scales a benchmark without giving up authenticity.
  • I4. Proverbs are tested separately from literal word meaning because their intended sense diverges from their surface; this gap is exactly where models scored lowest.
  • I5. Asking a model where an expression comes from treats variety identification as a task in its own right.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Tag fable and school-test QA items by unit (word, clause, whole story) and question type, so failures can be traced to the stage that caused them.
  • I2 (learn): Turn each test fact into several question formats and treat disagreement between formats as a consistency check, alongside negative controls.
  • I3 (learn): Let an LLM teacher tier draft candidate QA or frame annotations, and accept them only after human or multi-witness review.
  • I4 (frames): Build an English idiom/proverb test set that checks whether implicature output gives the figurative reading rather than the literal frame.
  • I5 (tree): Add a variety/register label (e.g. dialectal or informal English) to treebank sentences, so parser accuracy can be reported per variety and guide UD dialect conversion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Umm al-Qura UniversityMakkah al Mukarramah · Saudi Arabia21 works · 274 citations🎓Imam Abdulrahman Bin Faisal UniversityDammam · Saudi Arabia7 works · 95 citations🎓King Khalid UniversityAbhā · Saudi Arabia4 works · 16 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Renad Al-Monef, Hassan Alhuzali, Nora Alturayeif and Ashwag Alasmari. 2025. From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher. arXiv:2507.10216.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.10216 · arXiv 2507.10216

@misc{almonef2025words,
  title = {{From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher}},
  author = {Renad Al-Monef, Hassan Alhuzali, Nora Alturayeif and Ashwag Alasmari},
  year = {2025},
  doi = {10.48550/arxiv.2507.10216},
  eprint = {2507.10216},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.10216}
}

References

26 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners (2020)
  2. OpenAI: GPT-4 Technical Report (2023). https://openai.com/research/gpt-4 ↗
  3. Mousi, B., Durrani, N., Ahmad, F., Hasan, M.A., Hasanain, M., Kabbani, T., Dalvi, F., Chowdhury, S.A., Alam, F.: AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs, 4186–4218 (2025)
  4. Alwajih, F., Mekki, A.E., Magdy, S.M., Elmadany, A.A., Nacar, O., Nagoudi, E.M.B., Abdel-Salam, R., Atwany, H., Nafea, Y., Yahya, A.M., Alhamouri, R., Alsayadi, H.A., Zayed, H., Shatnawi, S., Sibaee, S., Ech-Chammakhy, Y., Al-Dhabyani, W., Ali, M.M., Jarraya, I., El-Shangiti, A.O., Alraeesi, A., Al-Ghrawi, M.A., Al-Batati, A.S., Mohamed, E., Elgindi, N.T., Saeed, M., Atou, H., Yahia, I.A., Bouayad, A., Machrouh, M., Makouar, A., Alkawi, D., Mohamed, M., Abdelfadil, S.T., Ounnoughene, A.Z., Anfel, R., Assi, R., Sorkatti, A., Tourad, M.C., Koubaa, A., Berrada, I., Jarrar, M., Shehata, S., Abdul-Mageed, M.: Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32871–32894. Association for Computational Linguistics, Vienna, Austria (2025). https://doi.org/10.18653/v1/2025. acl-long.1579 . https://aclanthology.org/2025.acl-long.1579/ ↗
  5. Al-Twairesh, N., Al-Matham, R., Madi, N., Almugren, N., Al-Aljmi, A.-H., Alshalan, S., Alshalan, R., Alrumayyan, N., Al-Manea, S., Bawazeer, S., Al-Mutlaq, N., Almanea, N., Huwaymil, W.B., Alqusair, D., Alotaibi, R., Al-Senaydi, S., Alfutamani, A.: Suar: Towards building a corpus for the saudi dialect. Procedia Computer Science 142, 72–82 (2018) https://doi.org/10.1016/j.procs.2018.10.462 . Arabic Computational Linguistics ↗
  6. Ayash, L., Alhuzali, H., Alasmari, A., Aloufi, S.: Saudiculture: A benchmark for evaluating large language models’ cultural competence within saudi arabia. Journal of King Saud University Computer and Information Sciences 37(6), 123 (2025) https://doi.org/10.1007/s44443-025-00137-9 ↗
  7. Al-Laith, A., Kebdani, R.: Evaluating calibration of Arabic pre-trained language models on dialectal text. In: Ezzini, S., Alami, H., Berrada, I., Benlahbib, A., El Mahdaouy, A., Lamsiyah, S., Derrouz, H., Haddad Haddad, A., Jarrar, M., El-Haj, M., Mitkov, R., Rayson, P. (eds.) Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pp. 68–76. Association for Computational Linguistics, Abu Dhabi, UAE (2025). https://aclanthology.org/2025.wacl-1.8/ ↗
  8. Alharbi, A.o., Alsuhaibani, A., Alalawi, A.A., Naseem, U., Jameel, S., Kanhere, S., Razzak, I.: Evaluating large language models on health-related claims across Arabic dialects. In: El-Haj, M. (ed.) Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pp. 95–103. Association for Computational Linguistics, Abu Dhabi, UAE (2025). https://aclanthology.org/2025.abjadnlp-1.11/ ↗
  9. Alwajih, F., Bhatia, G., Abdul-Mageed, M.: Dallah: A dialect-aware multimodal large language model for Arabic, 320–336 (2024) https://doi.org/10.18653/v1/2024.arabicnlp-1.27 ↗
  10. Hasanaath, A., Alansari, A., Ashraf, A., Salmane, C., Luqman, H., Ezzini, S.: Arareasoner: Evaluating reasoning-based llms for arabic nlp. arXiv preprint arXiv:2506.08768 (2025) ↗
  11. Magdy, S.M., Kwon, S.Y., Alwajih, F., Abdelfadil, S.T., Shehata, S., Abdul-Mageed, M.: JAWAHER: A multidialectal dataset of Arabic proverbs for LLM benchmarking, 12320–12341 (2025) https://doi.org/10.18653/v1/2025. naacl-long.613 ↗
  12. Guo, G., Naous, T., Wakaki, H., Nishimura, Y., Mitsufuji, Y., Ritter, A., Xu, W.: Care: Aligning language models for regional cultural awareness. arXiv preprint arXiv:2504.05154 (2025) ↗
  13. Alwajih, F., Nagoudi, E.M.B., Bhatia, G., Mohamed, A., Abdul-Mageed, M.: Peacock: A family of Arabic multimodal large language models and benchmarks, 12753–12776 (2024) https://doi.org/10.18653/v1/2024.acl-long.689 ↗
  14. Kadaoui, K., Atwany, H., Al-Ali, H., Mohamed, A., Mekky, A., Tilga, S., Fedorova, N., Artemova, E., Aldarmaki, H., Kementchedjhieva, Y.: Jeem: Vision-language understanding in four arabic dialects. arXiv preprint arXiv:2503.21910 (2025) ↗
  15. Bhatia, G., Nagoudi, E.M.B., El Mekki, A., Alwajih, F., Abdul-Mageed, M.: Swan and ArabicMTEB: Dialect-aware, Arabic-centric, cross-lingual, and cross-cultural embedding models and benchmarks, 4654–4670 (2025) https://doi.org/10.18653/v1/2025.findings-naacl.263 ↗
  16. Robinson, N.R., Abdelmoneim, S., Marchisio, K., Ruder, S.: AL-QASIDA: Analyzing LLM quality and accuracy systematically in dialectal Arabic, 22048–22065 (2025) https://doi.org/10.18653/v1/2025.findings-acl.1137 ↗
  17. Abdelaziz, A.A.A., Elneima, A.H., Darwish, K.: LLM-based MT data creation: Dialectal to MSA translation shared task. In: Al-Khalifa, H., Darwish, K., Mubarak, H., Ali, M., Elsayed, T. (eds.) Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pp. 112–116. ELRA and ICCL, Torino, Italia (2024). https://aclanthology.org/2024.osact-1.14/ ↗
  18. Sajjad, H., Abdelali, A., Durrani, N., Dalvi, F.: AraBench: Benchmarking dialectal Arabic-English machine translation. In: Scott, D., Bel, N., Zong, C. (eds.) Proceedings of the 28th International Conference on Computational Linguistics, pp. 5094–5107. International Committee on Computational Linguistics, Barcelona, Spain (Online) (2020). https://doi.org/10.18653/v1/2020. coling-main.447 . https://aclanthology.org/2020.coling-main.447/ ↗
  19. Yamani, A., Alziyady, R., AlYami, R., Albelali, S., Albelali, L., Almulhim, J., Alsulami, A., Alfarraj, M., Al-Zaidy, R.: The kind dataset: A social collaboration approach for nuanced dialect data collection. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 32–43 (2024) find
  20. Touvron, H., Lavril, T., Izacard, G., et al.: LLaMA 3: Open Foundation and Instruction-Tuned Models. Meta AI. (2024) find
  21. Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Afzal, O.M., Kamboj, S., Pandit, O., Pal, R., Pradhan, L., Mujahid, Z.M., Baali, M., Fikri Aji, A., Liu, Z., Hock, A., Feldman, A., Lee, J., Jackson, A., Nakov, P., Baldwin, T., Xing, E.: Jais-13b. Hugging Face. Accessed: 2025-08-1 (2023). https://huggingface.co/inceptionai/jais-13b ↗
  22. Bari, M.S., Alnumay, Y., Alzahrani, N.A., Alotaibi, N.M., Alyahya, H.A., AlRashed, S., Mirza, F.A., Alsubaie, S.Z., Alahmed, H.A., Alabduljabbar, G., Alkhathran, R., Almushayqih, Y., Alnajim, R., Alsubaihi, S., Mansour, M.A., Hassan, S.A., Alrubaian, D.M., Alammari, A., Alawami, Z., Al-Thubaity, A., Abdelali, A., Kuriakose, J., Abujabal, A., Al-Twairesh, N., Alowisheq, A., Khan, H.: ALLam: Large language models for arabic and english. In: The Thirteenth International Conference on Learning Representations (2025). https://openreview.net/forum?id=MscdsFVZrN ↗
  23. Jiang, A., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D., Bressand, F., Lengyel, G., Lample, G., Lavaud, L.R., Saulnier, L., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., El Sayed, W.: Mistral-7B-v0.1. Hugging Face. Accessed: 2025-8-1 (2023). https://huggingface.co/mistralai/Mistral-7B-v0.1 ↗
  24. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., Qiu, Z., et al.: Qwen2.5: A Party of Foundation Models (2024). https://qwenlm.github.io/blog/qwen2.5/ ↗
  25. Huang, H., Yu, F., Zhu, J., Sun, X., Cheng, H., Dingjie, S., Chen, Z., Alharthi, M., An, B., He, J., Liu, Z., Chen, J., Li, J., Wang, B., Zhang, L., Sun, R., Wan, X., Li, H., Xu, J.: AceGPT, localizing large language models in Arabic, 8139–8163 (2024) https://doi.org/10.18653/v1/2024.naacl-long.450 ↗
  26. Nazi, Z.A., Hossain, M.R., Mamun, F.A.: Evaluation of open and closed-source llms for low-resource language with zero-shot, few-shot, and chain-of-thought prompting. Natural Language Processing Journal 10, 100124 (2025) https://doi.org/10.1016/j.nlp.2024.100124 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-monef-2025-words-proverbs-evaluating-llms-linguistic/