Elamur Research⌕
☰
home / papers / 2025 / Towards Inclusive Arabic LLMs: A Culturally Aligned Benchmark in Arabic Large Language Model Evaluation

Towards Inclusive Arabic LLMs: A Culturally Aligned Benchmark in Arabic Large Language Model Evaluation

Paper · 2025 · Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed et al.

Summary

Arabic LLMs are often scored on translated Western benchmarks, so their grasp of Arab cultural, religious and social context is never really tested. This work audits the Arabic MMLU, using both the GPT-3.5-Turbo machine translation and OpenAI's human-translated version, to see how well it fits Arabic-speaking users. Eleven Arabic-language experts reviewed more than 2,500 questions. Each topic had three annotators, and the team held consensus meetings. Ratings covered fluency, adequacy, cultural appropriateness, bias, religious sensitivity and social norms, and were paired with automatic translation metrics. The audit found serious problems in morally and religiously loaded topics, and sorted the issues into three groups: linguistic, cultural, and methodological/structural. The authors then dropped Western-centric subjects, added five topics rooted in Arab culture, wrote annotation guidelines and published the refined dataset. They also launched the ILMAAM leaderboard, which scores Arabic LLMs on topics outside the sensitive set. The study shows that good accuracy on translated benchmarks does not prove a model is culturally suitable for its users.

Key points

  • Six MMLU topics were marked as critically misaligned: world religions, philosophy, professional psychology, human sexuality, and the two morality subjects (disputes and scenarios). Over 700 Western-centric questions (e.g., European and US history, US Foreign Policy) were set aside as unsuitable.
  • Five new Arab-relevant topics were added: Islamic religion, Old Arab history, Islamic history, Arabic ethics and Arabic educational methodologies.
  • Fluency and adequacy were rated on a 1–5 scale. Four cultural metrics were added on top. Automatic metrics were BLEU, ROUGE, METEOR, chrF, BERTScore and COMET.
  • The ILMAAM leaderboard covers around 30 Arabic LLMs. Qwen/Qwen2.5-72B-Instruct leads at 73.45, followed by CohereForAI/aya-expanse-32b at 63.87; aya-expanse-8b scores 51.79. Larger and instruction-tuned models tend to rank higher.
  • The refined dataset (ILMAAM-Arabic-Culturally-Aligned-MMLU) and the leaderboard are hosted on Hugging Face.

Interesting ideas

  • I1. Topics in a benchmark are first triaged into general, critically misaligned, excluded and newly added groups, and the headline score uses only the non-critical part, so culture-loaded items do not distort the accuracy figure.
  • I2. Human rating keeps form (fluency) and faithfulness (adequacy) apart from content judgments (appropriateness, bias, norms), so one weak dimension cannot hide behind a strong one.
  • I3. Error findings are organized into a three-way taxonomy: linguistic, cultural, and structural/source problems. One example is terms left untranslated or Arabized inconsistently even when a standard native word exists.
  • I4. Several independent annotators cover each topic, and periodic consensus meetings refine the shared interpretation of the guidelines, which reduces subjectivity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Split school-course tests and fable QA into culture-neutral and culture-dependent items, and report them separately so reasoning failures are not confused with missing background knowledge.
  • I2 (learn): When grading LLM-teacher silver data, score grammatical well-formedness and semantic faithfulness to the world model on separate axes, and accept data only if both pass.
  • I3 (back): During translation through the meta-world, check output terminology against a canonical lexicon so each concept always gets the same established target word.
  • I4 (learn): Mirror this in the witness-parser setup by using disagreements to drive periodic revisions of the annotation conventions, not just majority voting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S. Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader and Anis Koubaa. 2025. Towards Inclusive Arabic LLMs: A Culturally Aligned Benchmark in Arabic Large Language Model Evaluation. Proceedings of the First Workshop on Language Models for Low-Resource Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nacar2025inclusive,
  title = {{Towards Inclusive Arabic LLMs: A Culturally Aligned Benchmark in Arabic Large Language Model Evaluation}},
  author = {Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S. Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, Mohamed Abdelkader and Anis Koubaa},
  booktitle = {Proceedings of the First Workshop on Language Models for Low-Resource Languages},
  year = {2025},
  url = {https://aclanthology.org/2025.loreslm-1.29/}
}

References

10 references, 10 of them with a card of ours. ↗ leads to the work.

  1. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  2. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2021. Systematic inequalities in language technology performance across the world’s languages. arXiv preprint arXiv:2110.06733. ↗
  3. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arXiv preprint arXiv:2303.17466. ↗
  4. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507. This study provides a comprehensive evaluation of the Arabic MMLU Benchmark, highlighting critical issues in linguistic coherence and cultural Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, alignment that hinder its effectiveness for Arabic.
  5. Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under-Results reveal cultural misalignments stemming standing. arXiv preprint arXiv:2009.03300. from an over-reliance on Western concepts and a lack of clear Arabic source references, all of which Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao reduce the benchmark’s cultural relevance and us-Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, et al. 2023. Acegpt, ability. Furthermore, the large volume of questions localizing large language models in arabic. arXiv across varied topics poses a challenge for thorough preprint arXiv:2309.12053. cultural review, as addressing this comprehensively requires a larger team and extended time commit- Yuu Jinnai. 2024. Does cross-cultural alignment change the commonsense morality of language models? ment. These insights underscore the need for a arXiv preprint arXiv:2406.16316. refined benchmark with culturally aligned topics. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Dough-Future work should focus on evaluating Arabic man, Abdelrahman Boda Sadallah, Aisha Alraeesi, LLMs on culturally tailored benchmarks to assess Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, their performance when engaging with content that Shady Shehata, et al. 2024. Arabicmmlu: Assessing resonates with Arabic social, historical, and ethical massive multitask language understanding in arabic. perspectives. arXiv preprint arXiv:2402.12840. 394 ↗
  6. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. A.1 Evaluation Criteria Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. arXiv preprint arXiv:2006.07264. Evaluators assess translations based on two primary categories: Translation Metrics and Culture Alignment Metrics. The Culture Alignment Metrics apply only to topics requiring additional cultural sensitivity (CA-marked topics). ↗
  7. Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. arXiv preprint arXiv:2309.12342. A.1.1 Translation Metrics These metrics evaluate the linguistic quality of the translation to ensure accuracy and naturalness in the target language. ↗
  8. Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1–21. • Fluency: Measures grammatical accuracy and ease of reading. Ratings range from: – 1 – Incomprehensible – 2 – Poor fluency with many grammatical errors and unnatural phrasing – 3 – Understandable but contains some awkward language – 4 – Good fluency with minor errors and natural phrasing – 5 – Native-level fluency, flawless grammar, and exceptionally natural language Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. pages 311–318.
  9. Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395.
  10. Zhaozhi Qian, Faroq Altam, Muhammad Saleh Saeed Alqurishi, and Riad Souissi. 2024. Cameleval: Advancing culturally aligned arabic language models and benchmarks. arXiv preprint arXiv:2409.12623. • Adequacy: Assesses how accurately the translation conveys the meaning, intent, and nuances of the source text. Ratings are: Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nacar-2025-inclusive-arabic-llms-culturally-aligned/