Elamur Research⌕
☰
home / papers / 2026 / SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

Summary

Safety testing of Arabic language models has mostly relied on English-centric benchmarks or on Modern Standard Arabic (MSA) alone. Most Arabic users, however, write in regional dialects. SalamahBench addresses this gap. It contains 8,270 human-verified harmful prompts organised by the ML Commons hazard categories. Each prompt is expressed in MSA and in five regional varieties (Egyptian, Syrian, Saudi, Lebanese, Moroccan), giving 49,620 paired items in which the harmful intent stays the same and only the variety changes.

The authors propose two measures built on attack success rate (ASR):

  • Dialect Shift (DS) captures a model's overall change from MSA to a dialect.
  • Category-Specific Dialect Deviation (CSDD) flags harm categories whose change differs from that overall shift.

Before using any automatic safeguard as a judge, they check it against human gold labels. Qwen3Guard in strict mode performs best and is chosen. Testing Fanar 2, ALLaM 2 and Karnak 1 shows that dialect robustness depends heavily on the model. MSA results do not predict dialect behaviour, and averaged scores can hide failures in specific categories.

Key points

  • The benchmark has 8,270 prompts × 6 varieties = 49,620 instances, labelled with the ML Commons taxonomy. It is available from the corresponding author on request.
  • Qwen3Guard (strict) reached 84.7% response-safety accuracy against human labels. Some other safeguards barely beat an "always safe" baseline. In one case, a guard scored 1.4 points above a 57.6% baseline while recalling only 6.7% of unsafe responses. Llama Guard 4 recalled just 12.1% of unsafe responses on Sex-Related Crimes.
  • ALLaM's average shift across the five dialects is statistically indistinguishable from zero. Yet its Intellectual Property ASR rises in every dialect.
  • The Arabic models were also tested as prompted judges on 1,440 responses (four-class accuracy):
    • Karnak 1 scored 59.3%, ALLaM 2 51.3% and Fanar 2 47.0%.
    • Fanar 2 reached 65.6% on Karnak's outputs but only 38.3% on its own.
  • Limitations: native-speaker validation has so far covered only the Egyptian variants, and the evaluated models answered mostly in MSA.

Interesting ideas

  • I1. Keep the content of a test item fixed and vary only the linguistic variety, so any behaviour change can be attributed to the form of the text rather than its meaning. — For Mova (answer): Rewrite school tests and maths word problems in informal or regional English registers, then measure how much answer accuracy drops for each pair.
  • I2. Measure each category's change relative to the model's overall change (the CSDD idea), so a regression in one category is not hidden by stability elsewhere. — For Mova (tree): After a UD dialect conversion or parser update, report each dependency relation's LAS change minus the global LAS change, to catch relations that quietly degrade.
  • I3. Score every automatic classifier against a trivial majority-class baseline and report recall, because high accuracy can coexist with almost no detection of the rare class. — For Mova (learn): Add "always predict the majority label" as a mandatory negative control for every rule or classifier, and require a clear margin over it.
  • I4. How well a judge performs depends on whose output it grades, and models can be unusually good or bad at grading themselves. — For Mova (learn): When LLM teachers or witness parsers grade silver data, prevent self-grading, cross-assign judges, and track accuracy per source.
  • I5. Validate any automatic judge on a small human-labelled gold subset before trusting it for large-scale analysis. — For Mova (learn): Calibrate each LLM teacher tier on a hand-checked sample of frames or trees before its labels are allowed into training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining · Semitic languages · Bias, fairness & safety

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.04410 · arXiv 2603.04410

@misc{abdelnasser2026salamahbench,
  title = {{SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models}},
  author = {Abdelnasser, Omar and Alharbi, Fatemah and Khasawneh, Khaled and Alouani, Ihsen and Fouda, Mohammed E.},
  year = {2026},
  doi = {10.48550/arxiv.2603.04410},
  eprint = {2603.04410},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.04410}
}

References

47 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ben Mann, Nick Ryder, Melanie Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1(3):3, 2020. ↗
  2. Walter F Wiggins and Ali S Tejani. On the opportunities and risks of foundation models for natural language processing in radiology. Radiology: Artificial Intelligence, 4(4):e220119, 2022. find our card
  3. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. ↗ our card
  4. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. ↗
  5. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. ↗ our card
  6. Ziwei et al. Ji. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023. find our card
  7. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. All languages matter: On the multilingual safety of llms. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5865–5877, 2024. find
  8. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944, 2025. ↗ our card
  9. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan AlRashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390, 2024. ↗ our card
  10. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507, 2021. find our card
  11. Ashraf Elnagar, Sane M Yagi, Ali Bou Nassif, Ismail Shahin, and Said A Salloum. Systematic literature review of dialectal arabic: identification and detection. IEEE Access, 9:31010–31042, 2021. find
  12. Shaona Ghosh, Heather Frase, Adina Williams, Sarah Luger, Paul Röttger, Fazl Barez, Sean McGregor, Kenneth Fricklas, Mala Kumar, Kurt Bollacker, et al. Ailuminate: Introducing v1. 0 of the ai risk and reliability benchmark from mlcommons. arXiv preprint arXiv:2503.05731, 2025. ↗
  13. HUMAIN. HUMAIN and Qualcomm launch global AI inferencing initiative, 2025. URL https://www.humain.com/en/news/humain-qualcomm-global-inferencing.html. ↗
  14. Capacity Global. Stargate UAE data centre, 2025. URL https://capacityglobal.com/news/stargate-uae-data-centre/. ↗
  15. FANAR Team. Fanar 2.0: Arabic generative AI stack, 2026. URL https://arxiv.org/abs/2603.16397. ↗
  16. Mohamed Anwar, Abdelhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Atef Mohamed Ali Sadallah, Gurpreet Gosal, Gokul Ramakrishnan, Sarath Chandran, Biswajit Mishra, Rituraj Joshi, Ahmed Frikha, Etienne Goffinet, Abhishek Maiti, Ali El Filali, Sarah Al Barri, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Awantika Shukla, Sajid Siddiki, Samta Kamboj, Onkar Pandit, Sunil Sahu, Abelrahman El Badawy, Amr Mohamed, Ahmad Chamma, Evan Dufraisse, Abdelaziz Bounhar, Dani Bouch, Hadi Abdine, Guokan Shang, Fajri Koto, Yuxia Wang, Zhuohan Xie, Ali Mekky, Rania Hossam Elmohamady Elbadry, Sarfraz Ahmad, Momina Ahsan, Omar Emad Mohamed El-Herraoui, Daniil Orel, Hasan Iqbal, Kareem Mohamed Naguib Abdelmohsen Fahmy Elzeky, Mervat Abassy, Kareem Ali, Saadeldine Eletter, Farah Atif, Nurdaulet Mukhituly, Haonan Li, Xudong Han, Aaryamonvikram Singh, Zain Quraishi, Neha Sengupta, Larry Murray, Avraham Sheinin, Joel Hestness, Natalia Vassilieva, Hector Ren, Zhengzhong Liu, Michalis Vazirgiannis, and Preslav Nakov. Jais 2: A family of Arabic-centric open large language models. Technical report, IFM, December 2025. find our card
  17. Iheb Chaabane, Puneesh Khanna, Suhail Mohmad, Slim Frikha, Shi Hu, Abdalgader Abubaker, Reda Alami, Mikhail Lubinets, Mohamed El Amine Seddik, Hakim Hacid, et al. Falcon-h1r: Pushing the reasoning frontiers with a hybrid model for efficient test-time scaling. arXiv preprint arXiv:2601.02346, 2026. ↗
  18. Applied Innovation Center. Karnak: A depth-extended arabic-english llm. https://huggingface.co/Applied-Innovation-Center/Karnak, 2026. ↗
  19. Yasser Ashraf, Yuxia Wang, Bin Gu, Preslav Nakov, and Timothy Baldwin. Arabic dataset for llm safeguard evaluation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5529–5546, 2025. find
  20. Rawan Nasser Almatham, Kareem Mohamed Darwish, Raghad Al-Rasheed, Waad Thuwaini Alshammari, Muneera Alhoshan, Amal Almazrua, Asma Al Wazrah, Mais Alheraki, Firoj Alam, Preslav Nakov, et al. Balsam: A platform for benchmarking arabic large language models. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 258–277, 2025. find
  21. Emad A Alghamdi, Reem Masoud, Deema Alnuhait, Afnan Y Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. Aratrust: An evaluation of trustworthiness for llms in arabic. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8664–8679, 2025. find
  22. Lama Ayash, Hassan Alhuzali, Ashwag Alasmari, and Sultan Aloufi. Saudiculture: A benchmark for evaluating large language models’ cultural competence within saudi arabia. Journal of King Saud University Computer and Information Sciences, 37(6):123, 2025. find
  23. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. ↗ our card
  24. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. ↗ our card
  25. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models, 2020. URL https://arxiv.org/abs/2009.11462. ↗ our card
  26. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL https://arxiv.org/pdf/2307.15043. ↗ our card
  27. Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment, 2023. find
  28. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387, 2023. ↗
  29. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. ↗ our card
  30. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023. ↗ our card
  31. Paul et al. Christiano. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017. find
  32. Meta AI. Llama guard 4 technical report, 2025. URL https://huggingface.co/meta-llama/Llama-Guard-4-12B. Technical description of Llama Guard 4 safety classifier. ↗
  33. Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, and Maarten Sap. Polyguard: A multilingual safety moderation tool for 17 languages. arXiv preprint arXiv:2504.04377, 2025. ↗
  34. Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, et al. Safeguarding large language models: A survey. Artificial intelligence review, 58(12):382, 2025. find our card
  35. Qwen. Qwen3guard technical report. arXiv preprint arXiv:2510.14276, 2025. ↗
  36. vLLM Project. HaluGate Sentinel: Prompt-level fact-check switch for hallucination gatekeepers, 2024. URL https://github.com/vllm-project/semantic-router. ↗
  37. Adrian de Wynter, Ishaan Watts, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Nektar Ege Altıntoprak, Lena Baur, Samantha Claudet, Pavel Gajdušek, Qilong Gu, et al. RTP-LX: Can LLMs evaluate toxicity in multilingual scenarios? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27940–27950, 2025. find
  38. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024. URL https://arxiv.org/abs/2406.18495. ↗ our card
  39. Hamdy Mubarak, Abubakr Mohamed, and Majd Hawasly. Arasafe: Benchmarking safety in arabic llms. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 9976–9992, 2025. find
  40. Zhiyuan Ning, Tianle Gu, Jiaxin Song, Shixin Hong, Lin Li, Haoran Liu, Jie Li, Yixu Wang, Meng Lingyu, Yan Teng, and Yingchun Wang. LinguaSafe: A comprehensive multilingual safety benchmark for large language models. arXiv preprint arXiv:2508.12733, 2025. ↗
  41. Oskar Hollinsworth, Ian McKenzie, Tom Tseng, and Adam Gleave. Clearharm: A more challenging jailbreak dataset. FAR.AI Research Webpage, 2025. URL https://www.far.ai/research/clearharm-a-more-challenging-jailbreak-dataset. ↗
  42. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024. ↗ our card
  43. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. URL https://arxiv.org/abs/2411.15594. ↗
  44. Anthropic. Introducing Claude Sonnet 4.5. claude-sonnet-4-5, 2025. find
  45. OpenAI. Gpt-5 system card. OpenAI Blog / System Card, 2025. URL https://openai.com/blog/gpt-5-system-card/. ↗ our card
  46. Qwen Team. Qwen3.8. https://qwen.ai/blog?id=qwen3.8, 2026. Open weights: https://huggingface.co/Qwen/Qwen3.8-27B. ↗
  47. Masoomali Fatehkia, Enes Altinisik, and Husrev Taha Sencar. Fanarguard: A culturally-aware moderation filter for arabic language models. arXiv preprint arXiv:2511.18852, 2025. https://www.anthropic.com/news/ ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdelnasser-2026-salamahbench-dialect-category-level-safety/