Elamur Research⌕
☰
home / papers / 2025 / ArGAN: Arabic Gender, Ability, and Nationality Dataset for Evaluating Biases in Large Language Models

ArGAN: Arabic Gender, Ability, and Nationality Dataset for Evaluating Biases in Large Language Models

Paper · 2025 · Ranwa Aly, Yara Allam, Rana Gaber et al.

Summary

Bias research on Arabic lags far behind English, partly because Arabic is heavily gendered, dialect-rich and low-resource. Arabic web corpora are often collected for volume rather than quality, so stereotypes can pass into models unnoticed. The authors build ArGAN, a Modern Standard Arabic (MSA) benchmark that probes three demographic axes: gender, ability and nationality. Native speakers wrote descriptors and adjectives. These fill sentence templates, which are wrapped in prompts designed to draw out stereotyped or offensive completions.

Five models were tested: three general-purpose ones (Llama 3.3, DeepSeek v3, GPT-4o) and two Arabic-focused ones (Aya, JAIS). Each axis gets the metrics that suit how its bias tends to show up:

  • Gender is scored with a stereotype-lexicon score, GPT-based judging and a new embedding metric, RDCS.
  • Ability and nationality are scored with Perspective API toxicity and identity-attack scores, plus AraBERT-based Regard.

Five native speakers also rated the outputs. No model wins everywhere. Aya is least gender-biased and DeepSeek v3 most. Llama 3.3 does best on ability and nationality, while GPT-4o is worst on nationality. The resource fills a gap for Arabic fairness evaluation, especially on ability and nationality, which earlier Arabic work did not cover.

Key points

  • The dataset has 20 templates and about 125 descriptors (nouns and adjectives), producing 711 sentences: 211 gender, 247 ability and 253 nationality. It can be expanded by adding fillers.
  • Templates come in three types:
    • one descriptor slot;
    • two competing descriptors (e.g. who is likelier to succeed);
    • a descriptor plus an adjective.
  • The nationality list covers 20 nationalities. Arabic tends to use specific nationalities (e.g. "Chinese") as stand-ins for whole regions rather than broad labels.
  • Identity-attack scores are low on ability but higher on nationality (about 0.28–0.31). On nationality, Regard and toxicity both show more negativity towards Arabs, Egyptians, Mexicans and Indians, and more positivity towards Americans, Germans and Japanese.
  • The Gender Bias Score is negative for every model, meaning skew against women. Values range from −0.1067 (DeepSeek v3) to −0.0354 (Aya). Human judgments put the models in the same order as GPT Bias: DeepSeek v3 at 79.6% is the most biased by GPT Bias, and Aya at 56.9% the least.

Interesting ideas

  • I1. RDCS measures the descriptor–adjective embedding distance in the model's chosen sentence and compares it with the same distance in the "non-response", the alternative the model passed over; the gap exposes hidden preference
  • I2. Bias metrics are picked per axis according to how harm actually surfaces: stereotyping for gender, hostility for ability and nationality, since sentiment alone misses "positive" stereotypes like nurturing women
  • I3. Regard is reported as variance of positive and negative rates across groups rather than as a mean, which makes uneven treatment between groups visible
  • I4. A small, crossed grid of templates and slot fillers yields hundreds of controlled probes that can be extended cheaply
  • I5. Automatic metric rankings are validated against the model ordering produced by native-speaker raters

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when evaluating with negative controls, score both the parser's chosen analysis and a swapped-argument counterfactual, and use the size of the gap as a sensitivity probe.
  • I2 (frames): pragmatics should flag sweeping generalisations about a group as their own speech-act type, independent of polarity, rather than relying on sentiment cues.
  • I3 (learn): report per-class variance of accuracy (e.g. across VerbNet classes or construction types), not just the average, to catch a model that is fine overall but fails some categories.
  • I4 (answer): generate QA and word-problem test items from templates with swappable entities and attributes, so that reader errors can be traced to a specific slot value.
  • I5 (learn): check that silver-data agreement scores order systems the same way a small human-checked sample does before trusting them for training decisions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt42 works · 287 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 worksBias, fairness & safety · 587 works

Cite

Cite as Ranwa Aly, Yara Allam, Rana Gaber and Christine Basta. 2025. ArGAN: Arabic Gender, Ability, and Nationality Dataset for Evaluating Biases in Large Language Models. Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.gebnlp-1.23

@inproceedings{aly2025argan,
  title = {{ArGAN: Arabic Gender, Ability, and Nationality Dataset for Evaluating Biases in Large Language Models}},
  author = {Ranwa Aly, Yara Allam, Rana Gaber and Christine Basta},
  booktitle = {Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP)},
  year = {2025},
  doi = {10.18653/v1/2025.gebnlp-1.23},
  url = {https://doi.org/10.18653/v1/2025.gebnlp-1.23}
}

References

16 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leen Raid Jamal Al Qadi. 2023. A benchmark to evaluate gender bias in arabic language models. Master’s thesis, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE, June. Thesis submitted in partial fulfillment of the requirements for the M.Sc degree in Natural Language Processing. Advisors: Dr. Shady Shehata, Dr. Bin Gu. ↗
  2. Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022. The Arabic parallel gender corpus 2.0: Extensions and analyses. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1870– 1884, Marseille, France. European Language Resources Association.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9– 15, Marseille, France. European Language Resource Association. ↗
  4. Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941–1955, Online. Association for Computational Linguistics. find
  5. António Câmara, Nina Taneja, Tamjeed Azad, Emily Allaway, and Richard Zemel. 2022. Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 90–106, Dublin, Ireland. Association for Computational Linguistics.
  6. Marta Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, and Carleigh Wood. 2023. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14141–14156, Singapore. Association for Computational Linguistics. ↗
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. ↗
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  9. Nizar Habash, Houda Bouamor, and Christine Chung. 2019. Automatic gender identification and reinflection in Arabic. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 155–165, Florence, Italy. Association for Computational Linguistics.
  10. Rishav Hada, Agrima Seth, Harshita Diddee, and Kalika Bali. 2023. “fifty shades of bias”: Normative ratings of gender bias in GPT generated English text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1862–1876, Singapore. Association for Computational Linguistics.
  11. Jigsaw and Google’s Counter Abuse Technology Team. 2023. Perspective API - research into machine learning. Accessed: 2025-04-10. find
  12. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.
  13. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  14. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407–3412, Hong Kong, China. Association for Computational Linguistics. ↗
  15. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. “I‘m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9180–9211, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  16. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aly-2025-argan-arabic-gender-ability-nationality/