Elamur Research⌕
☰
home / papers / 2025 / JAWAHER: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking

JAWAHER: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking

Paper · 2025 · Samar Mohamed Magdy, Sang Yun Kwon, Fakhraddin Alwajih et al.

Summary

Many LLMs still lean toward Anglo-American culture and handle English better than other languages. This makes culturally loaded figurative language, such as proverbs, especially hard for them. The authors present Jawaher, a manually curated benchmark of Arabic proverbs from 20 Arabic varieties spread across the Arab world. Each proverb comes with an idiomatic or literal English translation and explanations in both English and Arabic. Native-speaker annotators gathered the proverbs from public online sources and checked where each one actually comes from. The authors also mapped the collection's themes using multilingual embeddings.

They tested open- and closed-source LLMs zero-shot on translation and on explanation, scoring the outputs with automatic metrics and human judges. Models usually translate proverbs idiomatically, but their explanations are weak: they miss cultural background, origin stories, dialect-specific usage and deeper figurative meaning. Closed models do better than open ones, yet none fully closes the gap. The resource gives a concrete way to measure cultural competence in figurative understanding, beyond surface translation.

Key points

  • Jawaher holds 10,037 proverbs covering 20 Arabic varieties, grouped as:
    • Gulf
    • Levantine
    • North African
    • Arabian Peninsula
    • Nile Basin
  • Four native annotators (two from Egypt, one from Mauritania, one from Morocco) curated the data, consulting speakers from Jordan, Syria and the UAE.
  • Thematic analysis used multilingual-e5-large embeddings with UMAP. Clusters appeared around food, mother, children and eyes, with body parts as a recurring motif.
  • Human evaluation of explanations used four criteria: clarity, correctness, depth & details, and cultural relevance and sensitivity.
    • In English explanations, GPT-4o led on clarity (4.65) and correctness (4.25).
    • Claude 3.5 Sonnet led on depth (4.04) and cultural relevance (3.11).
    • Among open models (Gemma2-9B-it, Llama family), the best clarity was Gemma2-9B-it at 3.15.
  • No model's explanation recounted the story behind a proverb. Open models gave short explanations with little cultural background. The evaluation was zero-shot only.

Interesting ideas

  • I1. Each figurative item carries two separate targets, a translation and an explanation of its intended meaning, so a system can be right about the wording yet wrong about the point.
  • I2. Results showed translation can succeed while explanation fails, which shows that paraphrase skill is a poor proxy for comprehension.
  • I3. Explanations were scored on several separate human-judged axes rather than one overall number, which shows where exactly a system falls short.
  • I4. The paper frames proverbs through structure mapping theory: a source concept, such as a snake's offspring, is mapped onto a target concept, such as inherited bad character.
  • I5. Embedding the whole collection and visualising clusters was used as a cheap audit of thematic coverage and of culture-specific motifs.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): For proverbs and idioms, keep a literal frame alongside a separate implicature frame, and check each independently so a correct literal parse is not mistaken for understanding.
  • I2 (answer): In QA on fables and tales, test the moral or lesson as its own question, distinct from retelling or paraphrasing questions.
  • I3 (learn): When an LLM teacher or human grades Mova's explanations, use separate scores for correctness, clarity, depth and context, so repair targets the weakest axis.
  • I4 (world): Represent a proverb or fable moral as an explicit mapping between source-domain entities and states and target-domain ones, so the analogy stays explainable and can be queried.
  • I5 (learn): Use the same clustering offline, outside the pipeline, to check whether silver or test sets over- or under-represent certain topics before training or evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Samar Mohamed Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Taher Abdelfadil, Shady Shehata and Muhammad Abdul-Mageed. 2025. JAWAHER: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.613

@inproceedings{magdy2025jawaher,
  title = {{JAWAHER: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking}},
  author = {Samar Mohamed Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Taher Abdelfadil, Shady Shehata and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.613},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.613}
}

References

71 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  3. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling" culture" in llms: A survey. arXiv preprint arXiv:2403.15412. https://alliancecan.ca https://arc.ubc.ca/ubc-arc-sockeye ↗
  4. Salwa Ahmed. 2005. Educational and social values expressed by proverbs in two cultures: knowledge and use of proverbs in Sudan and in England. Ph.D. thesis, Berlin, Techn. ↗
  5. Juhaina Maen Al Issawi. 2021. Cultural-bound meaning of animal names in arabic. English Linguistics Research, 10(1):42–55. ↗
  6. Manar M Almanea. 2021. Automatic methods and neural networks in arabic texts diacritization: a comprehensive survey. IEEE Access, 9:145012–145032. ↗
  7. Walid Aransa. 2015. Statistical machine translation of the Arabic language. Ph.D. thesis, Université du Maine. ↗
  8. Muhammad Hasan Bakalla. 2023. Arabic culture: through its language and literature. Taylor & Francis. find
  9. Mona Baker. 2018. In other words: A coursebook on translation. Routledge. find
  10. Riadh Belkebir and Nizar Habash. 2021. Automatic error type annotation for arabic. In Conference on Computational Natural Language Learning. ↗
  11. Hezi Brosh. 2013. Proverbs in the arabic language classroom. International Journal of Humanities and Social Science, 3(5):19–29. find
  12. Lucília Chacoto. 1994. Estudo e formalização das propriedades léxico-sintácticas das expressões fixas proverbiais. find
  13. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  14. Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apidianaki, and Smaranda Muresan. 2023. I spy a metaphor: Large language models and diffusion models co-create visual metaphors. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7370–7388, Toronto, Canada. Association for Computational Linguistics. find
  15. Weijie Chen, Yongzhu Chang, Rongsheng Zhang, Jiashu Pu, Guandan Chen, Le Zhang, Yadong Xi, Yijiang Chen, and Chang Su. 2022. Probing simile knowledge from pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5875–5887, Dublin, Ireland. Association for Computational Linguistics. ↗
  16. Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. 2024. Can large language models be trusted for evaluation? scalable meta-evaluation of llms as evaluators via agent debate. arXiv preprint arXiv:2401.16788. ↗
  17. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. find
  18. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Ireland. Association for Computational Linguistics.
  19. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  20. Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388. ↗
  21. Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. 2023. EtiCor: Corpus for analyzing LLMs for etiquettes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6921–6931, Singapore. Association for Computational Linguistics. find
  22. Tayyara A El-Rahman. 2020. The practicability of proverbs in teaching arabic language and culture. Language Teaching Research. ↗
  23. NOUH SABRI Elmitwally and Ahmed Alsayat. 2020. Classification and construction of arabic corpus: figurative and literal. Journal of Theoretical and Applied Information Technology, 98(19). find
  24. Mohammed Farghal. 2021. Animal proverbs in jordanian popular culture: A thematic and translational analysis. Journal of English Literature and Language, 2(1):1–8. find
  25. Charles A Ferguson. 2003. Diglossia. In The bilingualism reader, pages 71–86. Routledge. find
  26. Susan Fussell and Mallie Moss. 2008. Figurative language in emotional communication. find
  27. Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155– 170. ↗
  28. Bahman Gorjian. 2024. Translating english proverbs into persian: A case of comparative linguistics. Comparative Linguistics. Ph.D. dissertation. find
  29. Akshat Gupta, Xiaoyang Song, and Gopala Anumanchipalli. 2023. Investigating the applicability of self-assessment tests for personality measurement of large language models. arXiv preprint arXiv:2309.08163. ↗
  30. Hady J Hamdan, Hanan Al-Madanat, and Wael Hamdan. 2023. Connotations of animal metaphors in the jordanian context. Psycholinguistics, 33(1):132– 166. ↗
  31. Qianyu He, Sijie Cheng, Zhixu Li, Rui Xie, and Yanghua Xiao. 2022. Can pre-trained language models interpret similes as smart as human? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7875–7887, Dublin, Ireland. Association for Computational Linguistics. find
  32. Clive Holes. 2004. Modern Arabic: Structures, functions, and varieties. Georgetown University Press. find
  33. Hyewon Jang, Qi Yu, and Diego Frassinelli. 2023. Figurative language processing: A linguistically informed feature analysis of the behavior of language models and humans. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9816–9832. find
  34. Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O’Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, et al. 2024. Emma-500: Enhancing massively multilingual adaptation of large language models. arXiv preprint arXiv:2409.17892. ↗
  35. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785. ↗
  36. Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023. Multi-lingual and multi-cultural figurative language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8269–8284, Toronto, Canada. Association for Computational Linguistics. ↗
  37. Raymond W Gibbs and Herbert L Colston. 2012. Interpreting figurative meaning. Cambridge University Press. ↗
  38. Jihen Karoui, Farah Benamara, Véronique Moriceau, Nathalie Aussenac-Gilles, and Lamia Hadrich Belguith. 2015. Towards a contextual pragmatic model to detect irony in tweets. In 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), volume 2, pages 644–650. ACL: Association for Computational Linguistics.
  39. Mª González Rey. 2002. Isabel, la phraséologie du français. find
  40. Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870. ↗
  41. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. Technical report, University of Pennsylvania ScholarlyCommons. find
  42. EV Kuhareva. 2008. Arabic proverbs and sayings. Dictionary with lexical and phraseological comments. Moscow: AST: Vosto. Zapad. find
  43. Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2023. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint arXiv:2309.08591. ↗
  44. W. Mieder. 2004. Proverbs: A Handbook. Greenwood Folklore Handbooks. Bloomsbury Academic. find
  45. Wolfgang Mieder. 2007. Proverbs as cultural units or items of folklore. na. find
  46. Wolfgang Mieder. 2021. From innovative anti-proverbs to modern proverbs. The Discoursal Use of Phraseological Units, page 61. find
  47. Suzanne Mpouli. 2017. Annotating similes in literary texts. In Proceedings of the 13th Joint ISO-ACL Workshop on Interoperable Semantic Annotation (ISA-13). find
  48. Vlad Niculae and Cristian Danescu-Niculescu-Mizil. 2014. Brighter than gold: Figurative language in user generated comparisons. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 2008–2018. find
  49. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. ↗
  50. Ben Prystawski, Paul Thibodeau, Christopher Potts, and Noah D Goodman. 2022. Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models. arXiv preprint arXiv:2209.08141. ↗
  51. Zhaozhi Qian, Faroq Altam, Muhammad Saleh Saeed Alqurishi, and Riad Souissi. 2024. Cameleval: Advancing culturally aligned arabic language models and benchmarks. arXiv preprint arXiv:2409.12623. ↗
  52. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. ↗
  53. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. ↗
  54. Aida Rinasovna Fattakhova, Alfiya Albertovna Gi-madeeva, Emma Rishatovna Galimova, and Timur Akzamovich Shaihullin. 2019. Proverbs in arabic: Definition, classification, outlook reflection. Journal of Research in Applied Linguistics, 10(Proceedings of the 6th International Conference on Applied Linguistics Issues (ALI 2019) July 19-20, 2019, Saint Petersburg, Russia):521–528. ↗
  55. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2024. V-flute: Visual figurative language understanding with textual explanations. arXiv preprint arXiv:2405.01474. ↗
  56. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. ↗
  57. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. ↗
  58. Ekaterina Shutova. 2011. Computational approaches to figurative language. find
  59. Vered Shwartz and Ido Dagan. 2019. Still a pain in the neck: Evaluating text representations on lexical composition. Transactions of the Association for Computational Linguistics, 7:403–419. ↗
  60. Pushpdeep Singh, Mayur Patidar, and Lovekesh Vig. 2024. Translating across cultures: Llms for intralingual cultural adaptation. arXiv preprint arXiv:2406.14504. ↗
  61. Minghuan Tan and Jing Jiang. 2021. Does BERT understand idioms? a probing-based empirical study of BERT encodings of idioms. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1397–1407, Held Online. INCOMA Ltd. find
  62. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. ↗
  63. Faezdzhon Meliboevich Tursunov. 2022. The influence of the cultural aspect on the translation of proverbs and idioms (a case study of the tajiki/persian, russian and english languages). Filologicheskie nauki. Voprosy teorii i praktiki, 15(2):616–620. find
  64. Laurens van der Maaten and Geoffrey Hinton. 2008. Vi-ualizing data using t-sne. Journal of Machine Learning Research, 9:2579–2605. find
  65. Kees Versteegh. 2014. Arabic language. Edinburgh University Press. find
  66. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
  67. Shelia Webster. 2000. Arabic proverbs and related forms. De Proverbio: An Electronic Journal of International Proverb Studies, 6(2). find
  68. Ron Yosef, Yonatan Bitton, and Dafna Shahaf. 2023. IRFL: Image recognition of figurative language. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1044–1058, Singapore. Association for Computational Linguistics. find
  69. Jiali Zeng, Linfeng Song, Jinsong Su, Jun Xie, Wei Song, and Jiebo Luo. 2020. Neural simile recognition with cyclic multitask learning and local attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9515–9522. find
  70. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2023. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641. ↗
  71. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/magdy-2025-jawaher-multidialectal-dataset-arabic-proverbs/