Elamur Research⌕
☰
home / papers / 2026 / Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

Summary

Most cultural benchmarks for LLMs check what a model knows. This paper instead measures how a model behaves when it gives advice, opinions or guidance in Arabic. The authors built AraBehave, a benchmark of open-ended prompts. The prompts come mainly from user-flagged chat logs, plus tester-written seeds and expert curation, and are grouped into eight sensitivity categories. Native speakers from several Arab regions rated six models on a 1–5 scale and had to justify each score in writing. The authors also trained a regression scorer on these labels.

The analysis of the annotators' rationales shows that cultural appropriateness splits into two mostly independent parts:

  • Normative stance: whether the answer adopts the community's expected framing.
  • Grounded accuracy: for example, correct religious citations.

The top general-purpose and Arabic-centric models tie overall but fail on opposite parts. Stance can be switched on or off with a single system-prompt sentence. Grounding depends on model scale and on Arabic alignment data. General safety benchmarks miss this dimension entirely, so stance and grounding should be reported separately.

Key points

  • Resource: AraBehave has 1,623 prompts and 29,214 judgments (6 models × 3 annotators each), with an inter-annotator ICC of 0.58. The scoring model reaches Pearson r = 0.74 when each model is held out in turn. The prompts, scores, rationales and scorer are all released.
  • Same score, different failures: Gemini 3.1 scores 3.84 and Fanar 2.0 scores 3.83.
    • General-purpose models lose points for secular or "both-sides" framing (28–33% of low-score rationales).
    • Fanar loses points for fabricated hadith and misquoted verses (29%).
  • Prompt effects:
    • One culturally oriented instruction raises Gemini to 4.57 and GPT-5 to 4.15.
    • A generic "be objective" prompt drops Allam-7B by 0.68.
    • When the identical prompts are posed in English, 5 of the 6 models receive lower ratings.
  • Scale and tuning:
    • Within a model family, scores rise with size (Qwen3.5 goes from 1.98 at 0.8B to 3.77 at 27B).
    • Replacing the instruction tuning with Arabic-translated UltraChat cuts Fanar-1 from 3.35 to 2.69, which is below instruction-tuned Gemma2-9B at 2.91.
  • Safety benchmarks: AIxamine safety scores sit in a narrow band (89.4–98.7), while cultural scores range from 2.71 to 3.84. The correlation between them is r = −0.58, which is not significant with only six models.

Interesting ideas

  • I1. Making every rater write a justification turns a single number into a coded failure taxonomy, which revealed two failure types that the averages hid. — For Mova (learn): Mova's evaluation could require each error on a test item to carry a short cause tag (wrong tokenization, wrong attachment, wrong frame role, missing world fact), then report error rates per tag rather than one accuracy figure.
  • I2. Two systems with equal totals can be broken in disjoint ways, so the paper argues for reporting separate axes; this applies to any multi-stage system. — For Mova (answer): Mova's school-test and fable-QA results could be split into "understood the text but lacked knowledge" versus "had the knowledge but misread the text", which points directly at the stage that needs fixing.
  • I3. The scorer is validated by leaving out one whole system at a time, which tests transfer to unseen producers rather than to unseen items. — For Mova (learn): when a judge or agreement model is built over witness parsers for silver data, an entire parser should be held out during validation to check that the judge does not just memorize one parser's habits.
  • I4. A capability is located by ablation: the culture-aware tuning data is swapped for a neutral corpus, and the resulting drop shows where the skill comes from. — For Mova (learn): the same negative control could remove one knowledge source (for example, VerbNet classes or a set of repair rules) and measure the drop, to confirm that the gain really comes from that component.
  • I5. Items where raters split evenly between extremes, each with coherent reasons, are treated as genuine pluralism rather than label noise. — For Mova (frames): for irony or implicature with no single correct reading, Mova could store several weighted interpretations, each with its own justification, instead of forcing one gold reading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.16006 · arXiv 2609.16006

@misc{altinisik2026beyond,
  title = {{Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models}},
  author = {Altinisik, Enes and Mubarak, Hamdy and Fatehkia, Masoomali and Sencar, Husrev\_Taha\_Sencar Husrev Taha},
  year = {2026},
  doi = {10.48550/arxiv.2609.16006},
  eprint = {2609.16006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.16006}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2026. Karnak: A Depth-Extended Arabic-English LLM. [https://huggingface.co/Applied-Innovation-Center/Karnak](https://huggingface.co/Applied-Innovation-Center/Karnak). ↗
  2. Adilazuarda, M. F.; Mukherjee, S.; Lavania, P.; Singh, S. S.; Aji, A. F.; O’Neill, J.; Modi, A.; and Choudhury, M. 2024. Towards measuring and modeling “culture” in LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 15763–15784. find
  3. Alghamdi, E. A.; Masoud, R. I.; Alnuhait, D.; Alomairi, A. Y.; Ashraf, A.; and Zaytoon, M. 2024. AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic. arXiv preprint arXiv:2403.09017. ↗
  4. AlKhamissi, M.; Xiao, Y.; AlKhamissi, B.; and Diab, M. 2026. Hire your anthropologist! rethinking culture benchmarks through an anthropological lens. In Findings of the Association for Computational Linguistics: EACL 2026, 1218– 1235. find
  5. Altakrori, M. H.; Habash, N.; Freihat, A. A.; Samih, Y.; Chirkunov, K.; AbuOdeh, M.; Florian, R.; Lynn, T.; Nakov, P.; and Aji, A. F. 2026. Dialectalarabicmmlu: Benchmarking dialectal capabilities in arabic and multilingual language models. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), volume 11, 3199– 3219. find our card
  6. Alwajih, F.; El Mekki, A.; Magdy, S. M.; Elmadany, A. A.; Nacar, O.; Nagoudi, E. M. B.; Abdel-Salam, R.; Atwany, H.; Nafea, Y.; Yahya, A. M.; Alhamouri, R.; Alsayadi, H. A.; Zayed, H.; Shatnawi, S.; Sibaee, S.; Ech-chammakhy, Y.; Al-Dhabyani, W.; Ali, M. M.; Jarraya, I.; El-Shangiti, A. O.; Alraeesi, A.; AL-Ghrawi, M. A.; Al-Batati, A. S.; Mohamed, E.; Elgindi, N. T.; Saeed, M.; Atou, H.; Yahia, I. A.; Bouayad, A.; Machrouh, M.; Makouar, A.; Alkawi, D.; Mohamed, M.; Abdelfadil, S. T.; Ounnoughene, A. Z.; Rouabhia, A.; Assi, R.; Sorkatti, A.; Tourad, M. C.; Koubaa, A.; Berrada, I.; Jarrar, M.; Shehata, S.; and Abdul-Mageed, M. 2025. Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 32871–32894. Vienna, Austria: Association for Computational Linguistics. ISBN 979-8-89176-251-0. find
  7. Anwar, M.; Freihat, A.; Ibrahim, G.; Awad, M.; Sadallah, A. A. M. A.; Gosal, G.; Ramakrishnan, G.; Chandran, S.; Mishra, B.; Joshi, R.; Frikha, A.; Goffinet, E.; Maiti, A.; El Filali, A.; Al Barri, S.; Ghosh, S.; Pal, R.; Mullah, P.; Shukla, A.; Siddiki, S.; Kamboj, S.; Pandit, O.; Sahu, S.; El Badawy, A.; Mohamed, A.; Chamma, A.; Dufraisse, E.; Bounhar, A.; Bouch, D.; Abdine, H.; Shang, G.; Koto, F.; Wang, Y.; Xie, Z.; Mekky, A.; Elbadry, R. H. E.; Ahmad, S.; Ahsan, M.; El-Herraoui, O. E. M.; Orel, D.; Iqbal, H.; Elzeky, K. M. N. A. F.; Abassy, M.; Ali, K.; Eletter, S.; Atif, F.; Mukhituly, N.; Li, H.; Han, X.; Singh, A.; Quraishi, Z.; Sengupta, N.; Murray, L.; Sheinin, A.; Hestness, J.; Vassilieva, N.; Ren, H.; Liu, Z.; Vazirgiannis, M.; and Nakov, P. 2025. Jais 2: A Family of Arabic-Centric Open Large Language Models. Technical report, IFM. find our card
  8. Arora, A.; Kaffee, L.-a.; and Augenstein, I. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the first workshop on cross-cultural considerations in NLP (C3NLP), 114–130. find our card
  9. Ashraf, Y.; Wang, Y.; Gu, B.; Nakov, P.; and Baldwin, T. 2025. Arabic Dataset for LLM Safeguard Evaluation. In ACL. ACL. find
  10. Bari, M. S.; Alnumay, Y.; Alzahrani, N. A.; Alotaibi, N. M.; Alyahya, H. A.; AlRashed, S.; Mirza, F. A.; Alsubaie, S. Z.; Alahmed, H. A.; Alabduljabbar, G.; et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗ our card
  11. Bulté, B.; and Terryn, A. R. 2026. Llms and cultural values: the impact of prompt language and explicit cultural framing. Computational Linguistics, 1–88. find
  12. Chaves, A. P.; and Gerosa, M. A. 2021. How should my chatbot interact? A survey on social characteristics in human– chatbot interaction design. International Journal of Human– Computer Interaction, 37(8): 729–758. find
  13. Chiu, Y. Y.; Jiang, L.; Lin, B. Y.; Park, C. Y.; Li, S. S.; Ravi, S.; Bhatia, M.; Antoniak, M.; Tsvetkov, Y.; Shwartz, V.; et al. 2024. Culturalbench: A robust, diverse, and challenging cultural benchmark by human-ai culturalteaming. arXiv preprint arXiv:2410.02677. ↗
  14. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1): 37–46. find
  15. Deniz, F.; Popovic, D.; Boshmaf, Y.; Jeong, E.; Ahmad, M.; Chawla, S.; and Khalil, I. 2025. aiXamine: Simplified LLM Safety and Security. arXiv preprint arXiv:2504.14985. ↗
  16. Ding, N.; Chen, Y.; Xu, B.; Qin, Y.; Zheng, Z.; Hu, S.; Liu, Z.; Sun, M.; and Zhou, B. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. find our card
  17. Fatehkia, M.; Altinisik, E.; Osman, M.; and Sencar, H. T. 2025. PAM: Training Policy-Aligned Moderation Filters at Scale. arXiv preprint arXiv:2505.19766. ↗
  18. Fatehkia, M.; Altinisik, E.; and Sencar, H. T. 2026. FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 7848–7869. find
  19. Haoyue, L. L.; and Cho, H. 2025. Factors influencing intention to engage in human–chatbot interaction: examining user perceptions and context culture orientation. Universal Access in the Information Society, 24(1): 607–620. find
  20. Hershcovich, D.; Frank, S.; Lent, H.; De Lhoneux, M.; Abdou, M.; Brandl, S.; Bugliarello, E.; Piqueras, L. C.; Chalkidis, I.; Cui, R.; et al. 2022. Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6997–7013. find our card
  21. Ji, J.; Liu, M.; Dai, J.; Pan, X.; Zhang, C.; Bian, C.; Zhang, C.; Sun, R.; Wang, Y.; and Yang, Y. 2023. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. arXiv preprint arXiv:2307.04657. ↗ our card
  22. Johnson, R. L.; Pistilli, G.; Menédez-González, N.; Duran, L. D. D.; Panai, E.; Kalpokiene, J.; and Bertulfo, D. J. 2022. The Ghost in the Machine has an American accent: value conflict in GPT-3. arXiv preprint arXiv:2203.07785. ↗
  23. Kharchenko, J.; Roosta, T.; Chadha, A.; and Shah, C. 2024. How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions. arXiv preprint arXiv:2406.14805. ↗
  24. Koto, F.; Li, H.; Shatnawi, S.; Doughman, J.; Sadallah, A.; Alraeesi, A.; Almubarak, K.; Alyafeai, Z.; Sengupta, N.; Shehata, S.; et al. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, 5622–5640. find our card
  25. Magdy, S. M.; Kwon, S. Y.; Alwajih, F.; Abdelfadil, S. T.; Shehata, S.; and Abdul-Mageed, M. 2025. JAWAHER: A multidialectal dataset of Arabic proverbs for LLM benchmarking. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 12320–12341. find
  26. Mousi, B.; Durrani, N.; Ahmad, F.; Hasan, M. A.; Hasanain, M.; Kabbani, T.; Dalvi, F.; Chowdhury, S. A.; and Alam, F. 2024. AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs. find
  27. Mubarak, H.; Mohamed, A.; and Hawasly, M. 2025. AraSafe: Benchmarking Safety in Arabic LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. find
  28. Naous, T.; Ryan, M. J.; Ritter, A.; and Xu, W. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 16366–16393. find our card
  29. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. ↗
  30. Qian, Z.; Altam, F.; Alqurishi, M.; and Souissi, R. 2024. Cameleval: Advancing culturally aligned arabic language models and benchmarks. arXiv preprint arXiv:2409.12623. ↗ our card
  31. Sadallah, A.; Tonga, J. C.; Almubarak, K.; Almheiri, S.; Atif, F.; Qwaider, C.; Kadaoui, K.; Shatnawi, S.; Alesh, Y.; and Koto, F. 2025. Commonsense Reasoning in Arab Culture. arXiv:2502.12788. ↗
  32. Sengupta, N.; Sahu, S. K.; Jia, B.; Katipomu, S.; Li, H.; Koto, F.; Marshall, W.; Gosal, G.; Liu, C.; Chen, Z.; et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗ our card
  33. Shrout, P. E.; and Fleiss, J. L. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin, 86(2): 420. find
  34. TEAM, F.; Abbas, U.; Ahmad, M. S.; Ahmad, M.; Al-Homaid, A.; Al-Nuaimi, A.; Altinisik, E.; Asgari, E.; Chawla, S.; Chowdhury, S.; Dalvi, F.; Darwish, K.; Durrani, N.; Elfeky, M.; Elmagarmid, A.; Eltabakh, M.; Ersoy, A.; Fatehkia, M.; Hashim, M. Q.; Hawasly, M.; Hefeeda, M.; Husaini, M.; Isufaj, K.; Jung, S.-G.; Lachemat, H.; Lucas, J. K.; Mohamed, A.; Mohiuddin, T.; Mousi, B.; Mubarak, H.; Musleh, A.; Ouzzani, M.; Sadeghi, A.; Sencar, H. T.; Shinoy, M.; Sinan, O.; and Zhang, Y. 2026. Fanar 2.0: Arabic Generative AI Stack. arXiv:2603.16397. ↗
  35. Team, F.; Abbas, U.; Ahmad, M. S.; Alam, F.; Altinisik, E.; Asgari, E.; Boshmaf, Y.; Boughorbel, S.; Chawla, S.; Chowdhury, S.; et al. 2025. Fanar: An Arabic-Centric Multimodal Generative AI Platform. arXiv preprint arXiv:2501.13944. ↗ our card
  36. Team, F.-L. 2025. Falcon-Arabic: A Breakthrough in Arabic Language Models. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/altinisik-2026-cultural-knowledge-evaluating-arabic-cultural/