Elamur Research⌕
☰
home / papers / 2025 / AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

Paper · 2025 · Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait et al.

Summary

Most safety and trustworthiness benchmarks for LLMs are built in English or Chinese, or are machine-translated. Such resources miss Arabic linguistic and cultural specifics, including the strong influence of Islam on social norms. The authors present AraTrust, which they describe as the first broad trustworthiness benchmark written natively in Arabic. It contains 522 human-authored multiple-choice items in Modern Standard Arabic, each with a single correct option. Native-speaker authors drew the items from exams, websites and existing datasets, or wrote them themselves, and cross-reviewed every item for quality.

Five models were tested: GPT-4, GPT-3.5 Turbo, AceGPT 7B, AceGPT 13B and Jais 13B. Each was run under zero-shot, one-shot, five-shot and zero-shot CoT prompting, and the authors graded the answers by hand. The proprietary OpenAI models clearly beat the Arabic-centric open models, and GPT-4 ranked highest overall. This gap shows that Arabic-focused LLMs still need substantial safety and alignment work. AraTrust provides a cheap, automatable way to track that progress.

Key points

  • Benchmark scope: AraTrust spans 8 categories split into 34 sub-categories.
    • The categories cover truthfulness, ethics and morality, offensive language, unfairness, illegal activities and privacy, plus two health categories, one physical and one mental.
    • Truthfulness, which tests commonsense knowledge, has 78 questions.
  • Data sources: 18 items were adapted from the Arabic Hate Speech Twitter dataset (Mubarak et al., 2023). The full set is published on Hugging Face as asas-ai/AraTrust.
  • Top results: GPT-4 achieved the best average, 84% in the one-shot setting, versus 81% zero-shot and 82% with CoT. GPT-3.5 Turbo reached 73–79% across settings.
  • Open-model results: AceGPT 7B averaged only 54–59%, AceGPT 13B 65–72% and Jais 13B 58–65%.
  • Prompting effects: Prompts in Arabic gave more accurate answers than prompts in English. Few-shot examples did not help the GPT models. CoT improved GPT-4 mainly on health and ethics questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations🎓Alexandria UniversityAlexandria · Egypt42 works · 287 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 works

Cite

Cite as Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf and Mohamed Zaytoon. 2025. AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alghamdi2025aratrust,
  title = {{AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic}},
  author = {Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf and Mohamed Zaytoon},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.579/}
}

References

24 references, 18 of them with a card of ours. ↗ leads to the work.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  2. Rishabh Bhardwaj and Soujanya Poria. 2023. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662. ↗
  3. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335. ↗
  4. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Orca: A challenging benchmark for arabic language understanding. Preprint, arXiv:2212.10758. ↗
  5. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  6. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗
  7. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 220–247, Singapore. Association for Computational Linguistics.
  8. Trustworthiness benchmarks are crucial for recognizing potential risks linked to LLMs. Given the increasing worry about the accuracy of current benchmarks in measuring and pinpointing particu- Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large lan-lar safety concerns, our focus is on enhancing the guage models are zero-shot reasoners. Advances in security and safety of Arabic LLMs. This is ac-neural information processing systems, 35:22199– complished by evaluating prominent Arabic LLMs 22213. using multiple-choice questions about different di-Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and mensions of Trustworthiness. We hope that our Yangqiu Song. 2023. Multi-step jailbreaking privacy work will trigger more efforts to create more safety attacks on chatgpt. arXiv preprint arXiv:2304.05197. benchmarks, including newer tasks in what could Percy Liang, Rishi Bommasani, Tony Lee, Dimitris be a virtuous cycle. Regarding data involved in Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian AraTrust, we develop the benchmark manually or Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku-using data from the public domain. For this reason, mar, et al. 2022. Holistic evaluation of language we do not have serious concerns about privacy. models. arXiv preprint arXiv:2211.09110. 8670 ↗
  9. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374. ↗
  10. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1–35. ↗
  11. Lingbo Mo, Boshi Wang, Muhao Chen, and Huan Sun. 2023. How trustworthy are open-source llms? an assessment under malicious demonstrations shows their vulnerabilities. arXiv preprint arXiv:2311.09447. ↗
  12. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect arabic offensive language and hate speech. Natural Language Engineering, 29(6):1436–1457. ↗
  13. Tarek Naous, Michael J Ryan, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456. ↗
  14. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155, 13. ↗
  15. Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022. Cultural incongruencies in artificial intelligence. arXiv preprint arXiv:2211.13069. ↗
  16. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021. Alue: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 173–184.
  17. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  18. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. 2023b. All languages matter: On the multilingual safety of large language models. arXiv preprint arXiv:2310.00905. ↗
  19. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023c. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387. ↗
  20. Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, et al. 2023a. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705. ↗
  21. Liang Xu, Kangkang Zhao, Lei Zhu, and Hang Xue. 2023b. Sc-safety: A multi-round open-ended question adversarial safety benchmark for large language models in chinese. arXiv preprint arXiv:2310.05818. ↗
  22. Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. 2021. Synthbio: A case study in human-ai collaborative curation of text datasets. arXiv preprint arXiv:2111.06467. ↗
  23. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045. ↗
  24. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alghamdi-2025-aratrust-evaluation-trustworthiness-llms-arabic/