Elamur Research⌕
☰
home / papers / 2025 / MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks

MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks

Summary

LLMs are increasingly considered for clinical use, but there has been almost no way to check how they handle medical content in Arabic. Existing medical benchmarks are mostly English, and the few Arabic resources focus narrowly on question answering. The authors build MedArabiQ, a benchmark of seven Arabic medical tasks in Modern Standard Arabic (MSA). It totals about 700 samples drawn from past medical exams and from real patient-doctor consultations in the public AraMed dataset. Formats include multiple-choice items, fill-in-the-blank items and open patient-doctor Q&A, and the tasks also touch on ethical decision making. Eight proprietary and open LLMs, among them GPT-4, Deepseek v3 and Gemini 1.5, are tested zero-shot. Models are grouped by two factors: whether their training data is documented and whether they are general multilingual or Arabic-centric. The evaluation shows factual hallucination in open-ended answers and susceptibility to bias in clinical decisions. The benchmark is publicly released, giving a starting point for fairer medical AI in Arabic and a template for other underserved languages.

Key points

  • MedArabiQ has seven datasets and roughly 700 samples, all in MSA. Dialects such as Gulf, Maghreb, Egyptian and Levantine are left for future work.
  • The data comes from two sources: private structured medical exams and notes, and AraMed dialogues, which are real consultations from the Altibbi telemedicine platform.
  • Eight LLMs were evaluated zero-shot, including GPT-4, Deepseek v3 and Gemini 1.5 Pro. Higher training compute (FLOPs) generally went with better scores, especially for closed models.
  • Medical students rated the Q&A subset on a 1–5 scale:
    • relevance 4.99
    • factuality 4.97
    • accuracy 4.88
    • clarity 4.89
  • The dataset and evaluation code are released at github.com/nyuad-cai/MedArabiQ.

Interesting ideas

  • I1. Models are sorted by whether their training corpora are documented, so contamination risk can be judged where it is knowable and flagged where it is not. — For Mova (learn): Tag each LLM teacher tier by corpus transparency and give less weight to "wins" from opaque teachers on public test sets.
  • I2. Items taken from a public dataset were deliberately altered so that pure memorization would fail while real understanding would still succeed. — For Mova (answer): Rewrite school-test and fable QA items with renamed characters, shuffled options and changed numbers as a negative control, confirming that readers reason from the world model rather than recall.
  • I3. The benchmark checks whether answers shift unfairly when patient profile details change, treating personalization and prejudice as separable. — For Mova (world): Vary irrelevant entity attributes in maths word problems and stories and assert the derived answer stays invariant, exposing spurious state dependencies.
  • I4. It mixes closed-format exam items with open dialogue answers, and humans grade the open part on separate axes (relevance, factuality, clarity) instead of one score. — For Mova (back): Score generated text from the meaning graph on distinct factuality and clarity axes so that fluent but unfaithful output is not rewarded.
  • I5. Linguistically specialized models are compared against broad multilingual ones, on the reasoning that exposure to a language is not the same as competence in it. — For Mova (tree): When choosing witness parsers or gparse training mixes, test single-dialect treebank models against multi-treebank ones on the target text type rather than assuming broader data helps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

22 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.03427 · arXiv 2505.03427

@misc{daoud2025medarabiq,
  title = {{MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks}},
  author = {Daoud, Mouath Abu and Abouzahir, Chaimae and Kharouf, Leen and Al-Eisawi, Walid and Habash, Nizar and Shamout, Farah E.},
  year = {2025},
  doi = {10.48550/arxiv.2505.03427},
  eprint = {2505.03427},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.03427}
}

References

59 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. ↗ our card
  2. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. Aramed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT), pages 50–56, 2024. find our card
  3. Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar Habash. Advancements in Arabic grammatical error detection and correction: An empirical investigation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6430–6448, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.396. URL https://aclanthology.org/2023.emnlp-main.396. ↗ our card
  4. Yousef M Aljawarneh, Rabiha Seboussi, and Gregory L Blatch. Advancements in Health Sciences. Springer, 2024. find
  5. Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial Intelligence in Medicine, 155:102938, September 2024. ISSN 0933-3657. doi: 10.1016/j.artmed.2024.102938. URL http://dx.doi.org/10.1016/j. artmed.2024.102938. ↗
  6. Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran AQ Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, et al. Cidar: Culturally relevant instruction dataset for arabic. arXiv preprint arXiv:2402.03177, 2024. ↗
  7. Anthropic. Claude 3.5 sonnet model card addendum: Enhanced capabilities and evaluations for reasoning, coding, and visual processing. Available at https://www.anthropic.com, 2024. ↗
  8. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Association for Computing Machinery, 2021. find
  9. Trista M Benítez, Yueyuan Xu, J Donald Boudreau, Alfred Wei Chieh Kow, Fernando Bello, Le Van Phuoc, Xiaofei Wang, Xiaodong Sun, Gilberto Ka-Kit Leung, Yanyan Lan, Yaxing Wang, Davy Cheng, Yih-Chung Tham, Tien Yin Wong, and Kevin C Chung. Harnessing the potential of large language models in medical education: promise and pitfalls. Journal of the American Medical Informatics Association, 31(3):776–784, 2023. URL https://academic.oup.com/jamia/article/31/3/776/7588721. ↗
  10. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions, 2025. URL https://arxiv.org/abs/2402.18060. ↗
  11. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, and Quoc V. Le. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. URL https://arxiv.org/abs/2210.11416. ↗
  12. Deng Chunyuan, Zhao Yilun, Tang Xiangru, Gerstein Mark, and Cohan Arman. Investigating data contamination in modern benchmarks for large language models. arXiv preprint arXiv:2311.09783, 2024. ↗ our card
  13. Amin Dada, Marie Bauer, Amanda Butler Contreras, Osman Alperen Koraş, Constantin Marc Seibold, Kaleb E Smith, and Jens Kleesiek. Does biomedical training lead to better medical performance?, 2024. URL https://arxiv.org/abs/2404.04067. ↗
  14. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938, 2024. ↗
  15. Weihua Du, Yiming Yang, and Sean Welleck. Optimizing temperature for language models with multi-sample inference. 2025. URL https://arxiv.org/abs/2502.05234. ↗
  16. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. Ethnologue: Languages of the world, 2024. URL http://www.ethnologue.com. Online resource, accessed on [your access date]. ↗
  17. Hela Fehri, Sondes Dardour, and Kais Haddar. Armed question answering system. Concurrency and Computation: Practice and Experience, 34(21):e7054, 2022. doi: https://doi.org/10.1002/cpe.7054. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.7054. ↗
  18. Agasthya Gangavarapu. Introducing l2m3, a multilingual medical large language model to advance health equity in low-resource regions. ArXiv Preprint, 2024. URL https://arxiv.org/abs/2404.08705. ↗
  19. Yanjun Gao, Dmitriy Dligach, Timothy Miller, John Caskey, Brihat Sharma, Matthew M. Churpek, and Majid Afshar. Dr.bench: Diagnostic reasoning benchmark for clinical natural language processing. Journal of Biomedical Informatics, 138:104286, 2023. ISSN 1532-0464. doi: https://doi.org/10.1016/j.jbi.2023.104286. URL https://www.sciencedirect.com/science/article/pii/S1532046423000072. ↗
  20. Hezam Gawbah. Ahd: Arabic healthcare dataset, 2024. find
  21. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and Akhil Mathur et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. ↗
  22. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge. 2025. URL https://arxiv.org/abs/2411.15594. ↗
  23. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, and Deyi Xiong. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736, 2023. ↗ our card
  24. Nizar Habash and David Palfreyman. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.9/. ↗
  25. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300. ↗ our card
  26. Shadi Iskander, Nachshon Cohen, and Zohar Karnin. Quality matters: Evaluating synthetic data for tool-using llms. arXiv preprint arXiv:2409.16341, 2024. URL https://arxiv.org/abs/2409.16341. ↗
  27. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020. URL https://arxiv.org/abs/2009.13081. ↗ our card
  28. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. ArXiv Preprint, 2023. URL https://arxiv.org/abs/2307.10169. ↗ our card
  29. Praveen K. Kanithi, Clément Christophe, Marco A. F. Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. Medic: Towards a comprehensive framework for evaluating llms in clinical applications. arXiv preprint arXiv:2409.07314, 2024. ↗
  30. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022. find our card
  31. Ruoyu Liu, Kui Xue, Xiaofan Zhang, and Shaoting Zhang. Interactive evaluation for medical LLMs via task-oriented dialogue system. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 4871–4896, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.325/. ↗
  32. Xiangbin Meng, Xiangyu Yan, Kuo Zhang, Da Liu, Xiaojuan Cui Yaodong Yang, Muhan Zhang, Chunxia Cao, Jingjia Wang, Xuliang Wang, Jun Gao, Yuan-Geng-Shuo Wang, Jia ming Ji, Zifeng Qiu, Muzi Li, Cheng Qian, Tianze Guo, Shuangquan Ma, Zeying Wang, Zexuan Guo, Youlan Lei, Chunli Shao, Wenyao Wang, Haojun Fan, and Yi-Da Tang. The application of large language models in medicine: A scoping review. Iscience, 2024. URL https://www.sciencedirect.com/science/article/pii/S2589004224009350?via%3Dihub. ↗
  33. Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. The first QALB shared task on automatic text correction for Arabic. In Nizar Habash and Stephan Vogel, editors, Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39–47, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-3605. URL https://aclanthology.org/W14-3605/. ↗
  34. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects, 2024. URL https://arxiv.org/abs/2501.00559. ↗ our card
  35. Preslav Nakov, Lluís Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, James Glass, and Bilal Randeree. Semeval-2016 task 3: Community question answering, 2019. URL https://arxiv.org/abs/1912.01972. ↗ our card
  36. Zabir Al Nazi and Wei Peng. Large language models in healthcare and medical domain: A review. Informatics, 11(3):57, 2023. doi: 10.3390/informatics11030057. URL https://www.mdpi.com/2227-9709/11/3/57. ↗
  37. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. Camel tools: An open source python toolkit for arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, 2020. find our card
  38. Mahmud Omar, Shelly Soffer, Reem Agbareia, Nicola Luigi Bragazzi, Donald U Apakama, Carol R Horowitz, Alexander W Charney, Robert Freeman, Benjamin Kummer, Benjamin S Glicksberg, et al. Sociodemographic biases in medical decision-making by large language models: a large-scale multi-model analysis. medRxiv, pages 2024–10, 2024. find
  39. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, ..., and Irwan Bello. Gpt-4 technical report. 2024. URL https://arxiv.org/abs/2303.08774. ↗ our card
  40. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR, 2022. find our card
  41. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023a. ↗
  42. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023b. URL https://arxiv.org/abs/2306.01116. ↗
  43. Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards building multilingual language model for medicine. Nature Communications, 2024. URL https://www.nature.com/articles/s41467-024-52417-z. ↗
  44. Sandeep Reddy, Sonia Allan, Simon Coghlan, and Paul Cooper. A governance model for the application of ai in health care. Journal of the American Medical Informatics Association, 27(3):491 – 497, 2020. doi: 10.1093/jamia/ocz192. URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-85078084667&doi=10.1093%2fjamia%2focz192&partnerID=40&md5=cd03274d7d5a043dbb46cc82edbc5a16. Cited by: 379; All Open Access, Green Open Access. ↗
  45. Rick Rejeleene, Xiaowei Xu, and John Talburt. Towards trustable language models: Investigating information quality of large language models. arXiv preprint arXiv:2401.13086, 2024. URL https://arxiv.org/pdf/2401.13086. ↗
  46. Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid, and Behrang Mohit. The second QALB shared task on automatic text correction for Arabic. In Nizar Habash, Stephan Vogel, and Kareem Darwish, editors, Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 26–35, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.18653/v1/W15-3204. URL https://aclanthology.org/W15-3204/. ↗
  47. Nada Saadi, Tathagata Raha, Clément Christophe, Marco AF Pimentel, Ronnie Rajan, and Praveen K Kanithi. Bridging language barriers in healthcare: A study on arabic llms, 2025. URL https://arxiv.org/abs/2501.09825. ↗ our card
  48. Mohammad Salameh, Houda Bouamor, and Nizar Habash. Fine-grained arabic dialect identification. In Proceedings of the ACL Conference, pages 113–122, 2018. URL https://aclanthology.org/C18-1113/. ↗ our card
  49. Malik Sallam and Dhia Mousa. Evaluating chatgpt performance in arabic dialects: A comparative study showing defects in responding to jordanian and tunisian general health prompts. Mesopotamian Journal of Artificial Intelligence in Healthcare, 2024. find
  50. Samuel Schmidgall, Carl Harris, Ime Essien, Daniel Olshvang, Tawsifur Rahman, Ji Woong Kim, Rojin Ziaei, Jason Eshraghian, Peter Abadir, and Rama Chellappa. Addressing cognitive bias in medical language models. arXiv preprint arXiv:2402.08113, 2024. ↗
  51. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models, 2023. find our card
  52. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, and Damien Vincent et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. find our card
  53. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. 2025. URL https://arxiv.org/abs/2406.12624. ↗
  54. Kim Theodos and Scott Sittig. Health information privacy laws in the digital age: Hipaa doesn’t apply. Perspectives in health information management, 18(Winter):1l, 2020. find
  55. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen2.5 technical report. 2025. URL https://arxiv.org/abs/2412.15115. ↗
  56. Rui Yang, Ting Fang Tan, Wei Lu, Arun James Thirunavukarasu, Daniel Shu Wei Ting, and Nan Liu. Large language models in health care: Development, applications, and challenges. Health Care Systems Journal, 2:61, 2024. doi: 10.1002/hcs2.61. URL https://onlinelibrary.wiley.com/doi/10.1002/hcs2.61. ↗
  57. Shuo Zhang, Lin Dong, Xiaoqing Li, Sheng Zhang, Xiaolong Sun, Song Wang, and Guoyin Wang. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023. URL https://arxiv.org/abs/2308.10792. ↗ our card
  58. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr. ↗
  59. Chao Zhou, Cheng Qiu, Lizhen Liang, and Daniel E. Acuna. Paraphrase identification with deep learning: A review of datasets and methods. 2024. URL https://arxiv.org/abs/2212.06933. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/daoud-2025-medarabiq-benchmarking-large-language-models/