Elamur Research⌕
☰
home / papers / 2026 / MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark

MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark

Summary

Arabic has few openly available medical NLP resources, so it is hard to measure how well LLMs handle clinical content in this language. MedAraBench addresses this gap. The authors gathered scanned paper exams that had been shared on student-run platforms of regional medical schools. Professional typists transcribed them, and five NLP researchers removed malformed items by hand. The result is an Arabic multiple-choice benchmark of 24,883 questions. Each question is tagged with its medical specialty (19 in total) and a difficulty level tied to the year of medical study (Y1–Y5). The data is split into training and test sets, stratified by specialty.

Quality was checked in two ways: two senior clinicians reviewed a statistically sized sample, and an LLM-as-a-judge analysis was also run. Proprietary, open-source, Arabic-centric and medical LLMs were tested zero-shot. The strongest model reached 0.765 accuracy, which is still short of expert level. The authors argue the benchmark is harder than earlier ones and that correct answers alone say little about real clinical reasoning.

Key points

  • 24,883 MCQs covering 19 specialties, from Anatomy to Surgery, with either 4 or 5 options. The 9 six-option items were dropped. The split is 80/20, stratified by specialty.
  • Difficulty is heavily skewed toward early years: Y1 is 56% and Y2 is 22.89%, while Y4 is only 3.38% and Y5 5.14%.
  • Expert review rated medical accuracy, clinical relevance, difficulty and item construction. Construction covered clarity, option homogeneity, single best answer and no clueing. The sample size came from Cochran's formula (95% confidence, ±5 points, p = 0.5) with a finite population correction.
  • Evaluated models include GPT-5, Gemini 2.0 Flash and Claude 4-Sonnet. The abstract mentions eight models, while the contributions list says 16 across three categories. The best accuracy reported is 0.765.
  • The dataset and evaluation scripts are released at github.com/nyuad-cai/MedAraBench. The data is text-only and in Modern Standard Arabic.

Interesting ideas

  • I1. The size of the human audit sample is computed with Cochran's formula plus a finite population correction, rather than picked arbitrarily, so the reported quality rate comes with a known margin of error.
  • I2. Each question carries a difficulty tier tied to the year of study, which lets results be reported per stage of the curriculum instead of as one overall number.
  • I3. The authors note that a model can reach high MCQ accuracy by ruling out implausible options through learned word associations, without real reasoning, so the rationale should be scored as well as the choice.
  • I4. Building the benchmark from sources that were never digitized lowers the risk that the test items appeared in model pretraining data.
  • I5. Item quality is judged with medical-education rules for MCQs: a single best answer, plausible distractors of the same type, and no option that gives away another.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Size manual spot-checks of silver UD trees or teacher-labelled frames with the same formula, so every "x% correct" claim about silver data carries a confidence bound.
  • I2 (answer): Tag school-course test items by grade and report accuracy per grade, to show exactly where the reader stops coping.
  • I3 (answer): Score the explanation chain (tree → frames → world facts used) next to the selected option, and add negative controls such as answering with the stem removed to catch distractor-only shortcuts.
  • I4 (learn): Assemble held-out evaluation texts from material that is not on the web, such as transcribed print or newly written items, so comparisons with LLM teachers are not inflated by memorization.
  • I5 (back): Apply these rules as automatic checks when Mova generates comprehension questions and distractors from the meaning graph of a fable or lesson.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Question answering Evaluation & benchmarks Semitic languages

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates45 works · 2,055 citations

Also on the byline: Cleveland Clinic Abu Dhabi.

Cite

Cite as Mouath Abu-Daoud, Leen Kharouf, Omar El Hajj, Dana El Samad, Mariam Al-Omari, Jihad Mallat, Khaled Saleh, Nizar Habash and Farah E. Shamout. 2026. MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark. arXiv:2602.01714.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2602.01714 · arXiv 2602.01714

@misc{abudaoud2026medarabench,
  title = {{MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark}},
  author = {Mouath Abu-Daoud, Leen Kharouf, Omar {El Hajj}, Dana {El Samad}, Mariam Al-Omari, Jihad Mallat, Khaled Saleh, Nizar Habash and Farah E. Shamout},
  year = {2026},
  doi = {10.48550/arxiv.2602.01714},
  eprint = {2602.01714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2602.01714}
}

References

47 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Umar Abbas. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944, 2025. ↗ our card
  2. Mariam Essam Abdelaziz, Mohanad A. Deif, Shabbab Ali Algamdi, and Rania Elgohary. A benchmark arabic dataset for arabic question classification using aafaq framework. Scientific Data, 2025. find
  3. Mouath Abu Daoud, Chaimae Abou Zahir, Leen Kharouf, Walid AlEisawi, Nizar Habash, and Farah Shamout. Medarabiq: Benchmarking large language models on arabic medical tasks. PMLR, 2025. our card
  4. Mohammed Al-Rukban. Guidelines for the construction of multiple choice questions tests. Journal of Family Community Medicine, 2006. URL https://pmc.ncbi.nlm.nih.gov/articles/PMC3410060/. ↗
  5. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. Aramed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT), pp. 50–56, 2024. our card
  6. Yazeed AlMoaiad. Challenges in natural arabic language processing. Edelweiss Applied Science and Technology, 2024. find
  7. Yazeed Alnumay. Command r7b arabic: A small, enterprise focused, multilingual, and culturally aware arabic llm. arXiv preprint arXiv:2503.14603v1, 2025. ↗
  8. Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial Intelligence in Medicine, 155:102938, September 2024. ISSN 0933-3657. doi: 10.1016/j.artmed.2024.102938. URL http://dx.doi.org/10.1016/j.artmed.2024.102938. ↗
  9. Khatra Amar. Arabic morphological productivity in the translation of medical terminology. PhD Thesis - UNIVERSITI SAINS MALAYSIA, 2022. find
  10. Anthropic. Claude opus 4 & claude sonnet https://www.anthropic.com/claude-4-system-card, 2025. ↗
  11. Yasser Mohamed Attia. The challenges of achieving dynamic equivalence in the arabic translation of medical terminology for heart and brain diseases: Difficulties and recommendations. Journal of Faculty of Arts, Port Said University, 30(30):14–31, 2024. find
  12. M Saiful Bari. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390, 2024. ↗ our card
  13. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, — system card. Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. URL https://arxiv.org/abs/2108.07258. ↗ our card
  14. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxiv.org/abs/2005.14165. ↗ our card
  15. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions, 2025. URL https://arxiv.org/abs/2402.18060. ↗
  16. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https://arxiv.org/abs/2204.02311. ↗ our card
  17. Clement Christophe. Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142, 2024. ↗
  18. William Gemmell Cochran. Sampling techniques. john wiley & sons, 1977. find
  19. DeepSeek-AI. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. ↗
  20. Ali Farghali and Khaled Shaalan. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 2009. find
  21. Google Cloud. Gemini 2.0 flash — generative ai on vertex ai. https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-0-flash, May 2025. ↗
  22. A. Grattafiori et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024a. ↗
  23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and Akhil Mathur et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024b. ↗
  24. Nizar Habash. Introduction to arabic natural language processing. ACL Tutorial, 2005. find
  25. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300. ↗ our card
  26. Pedram Hosseini. A benchmark for long-form medical question answering. arXiv preprint, 2024. find
  27. Shadi Iskander, Nachshon Cohen, and Zohar Karnin. Quality matters: Evaluating synthetic data for tool-using llms. arXiv preprint arXiv:2409.16341, 2024. URL https://arxiv.org/abs/2409.16341. ↗
  28. Luyi Jiang, Jiayuan Chen, Lu Lu, Xinwei Peng, Lihao Liu, Junjun He, and Jie Xu. Benchmarking chinese medical llms: A medbench-based analysis of performance gaps and hierarchical optimization strategies, 2025. URL https://arxiv.org/abs/2503.07306. ↗
  29. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020. URL https://arxiv.org/abs/2009.13081. ↗ our card
  30. Di Jin, Yansong Pan, Tao Ouyang, and Pascale Fung. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081, 2021. URL https://arxiv.org/abs/2009.13081. Submitted to AAAI 2021. ↗ our card
  31. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567–2577, 2019. our card
  32. Praveen K. Kanithi, Clément Christophe, Marco A. F. Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. Medic: Towards a comprehensive framework for evaluating llms in clinical applications. arXiv preprint arXiv:2409.07314, 2024. ↗
  33. Tobi Olatunji, Charles Nimo, Abraham Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, and Chinemelu Aka. Afrimed-qa: A pan-african, multi-specialty, medical question-answering benchmark dataset. arXiv preprint arXiv:2411.15640, 2024. URL https://arxiv.org/html/2411.15640. ↗
  34. OpenAI. Gpt-4 technical report. https://openai.com/index/gpt-4-1/, April 2025a. ↗ our card
  35. OpenAI. GPT-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf, 2025b. ↗ our card
  36. OpenAI. Openai o3 and o4-mini system card. https://cdn.openai.com/pdf/o3-and-o4-mini-system-card.pdf, 2025c. ↗
  37. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 248–260. PMLR, 2022. our card
  38. Sara Pieri, Sahal Shaji Mullappilly, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Timothy Baldwin, and Hisham Cholakkal. Bimedix: Bilingual medical mixture of experts llm. arXiv preprint arXiv:2402.13253, 2024. ↗
  39. Rick Rejeleene, Xiaowei Xu, and Johm Talburt. Towards trustable language models: Investigating information quality of large language models. arXiv preprint arXiv:2401.13086, 2024. URL https://arxiv.org/abs/2401.13086. ↗
  40. Andrew Sellergren. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025. ↗
  41. Shivalika Singh. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025. our card
  42. Felix Stahlberg. Neural machine translation: A review and survey, 2020. URL https://arxiv.org/abs/1912.02047. ↗
  43. Bin Wang, Chengwei Wei, Zhengyuan Liu, Geyu Lin, and Nancy Chen. Resilience of large language models for noisy instructions. EMNLP, 2024. find
  44. Cunxiang Wang, Pai Liu, and Yue Zhang. Can generative pre-trained language models serve as knowledge bases for closed-book qa?, 2021. URL https://arxiv.org/abs/2106.01561. ↗
  45. Xidong Wang. Apollo: A lightweight multilingual medical llm towards democratizing medical ai to 6b people. arXiv preprint arXiv:2403.03640, 2024. ↗
  46. An Yang and et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. URL https://arxiv.org/abs/2412.15115. ↗
  47. Han Yang, Mingchen Li, Huixue Zhou, Yongkang Xiao, Qian Fang, and Rui Zhang. Large language model synergy for ensemble learning in medical question answering: Design and evaluation study. J Med Internet Res, 2025. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abu-daoud-2026-medarabench-large-scale-arabic-medical/