Elamur Research⌕
☰
home / papers / 2025 / Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks

Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks

Summary

The paper asks how well current large language models handle medical questions written in Modern Standard Arabic, a setting with far fewer benchmarks than English. The authors used the 700-item evaluation data of the AraHealthQA shared task (MedArabiQ2025 track), plus a later 200-item test release. The data has two parts. One covers multiple-choice and fill-in-the-blank items with options, scored by exact-answer accuracy. The other covers open-ended answers, scored against expert references with BERTScore on XLM-RoBERTa-Large.

They queried proprietary models (GPT, Gemini, Claude, Grok, Deepseek, Llama) and open Arabic-oriented models (Falcon3, Fanar, Allam, Falcon Arabic). All runs were zero-shot and deterministic, with several prompt variants. Models with explicit reasoning, GPT o3 and the two Gemini 2.5 models, clearly led on multiple choice. A majority vote over these three reached 77% accuracy and won the challenge. On open-ended items, short answers in standard Arabic scored best, up to a BERTScore of 0.8644.

The work matters because it shows that even top models still lack medical knowledge in Arabic. It also shows that the common similarity metric barely separates good answers from weak ones.

Key points

  • The data has 700 MSA clinical items in two groups:
    • Multiple choice: 100 standard questions, 100 bias-injected questions and 100 fill-in-the-blank items with options.
    • Open-ended: 100 blanks without options, 100 patient–doctor exchanges from AraMed, 100 grammar-corrected Q&As and 100 LLM-rewritten Q&As meant to counter memorization.
  • Multiple-choice accuracy by model:
    • Gemini Pro 2.5 was the best single model at 76%, with Gemini Flash 2.5 at 74% and GPT o3 at 72–74%.
    • Llama 4 Maverick reached 63%, Grok 3 60%, Deepseek v3 56%, and Claude Opus and GPT-4o-mini 49% each.
    • Falcon Arabic scored only 38%.
  • The majority vote of GPT o3 and both Gemini 2.5 models gave 77% and first place overall.
  • A prompt asking for step-by-step reasoning generally beat the plain prompt on multiple-choice questions.
  • On open-ended questions, a prompt demanding concise, explanation-free MSA answers lifted all three tested models above 0.85 BERTScore. The authors note that BERTScore values cluster closely and hide differences in meaning.

Interesting ideas

  • I1. Pooling the outputs of three strong but differently built models by simple majority beat every single member, so diversity among voters was worth more than one better model. — For Mova (learn): The same vote could decide frame and role labels in silver data, not just parse arcs, with a label accepted only when most witnesses agree.
  • I2. The benchmark adds rewritten copies of questions so that a model cannot score by recalling seen text, which separates real reasoning from memorization. — For Mova (learn): Paraphrased or reworded variants of test items could act as negative controls, where a large accuracy drop on rewrites would flag that a component is matching surface strings.
  • I3. The same blank-filling task is offered both with and without answer options, which separates recognizing a right answer from producing it. — For Mova (answer): School-test and fable QA readers could be scored in both modes, to tell whether failures come from missing world knowledge or from the generation step.
  • I4. Asking for short answers without explanations raised similarity scores, which shows the metric partly rewards brevity rather than correctness. — For Mova (back): When judging generated text from the meaning graph, metrics should be checked for length bias, preferably by comparing graphs rather than surface embeddings.
  • I5. A dedicated subset where grammar errors in the source text were corrected lets one measure how sensitive a system is to inflectional noise. — For Mova (tokens): Pairs of raw and corrected sentences could test whether the tagger and lemmatizer give stable analyses when the input contains spelling or agreement mistakes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.15797 · arXiv 2508.15797

@misc{aldahoul2025benchmarking,
  title = {{Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks}},
  author = {AlDahoul, Nouar and Zaki, Yasir},
  year = {2025},
  doi = {10.48550/arxiv.2508.15797},
  eprint = {2508.15797},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.15797}
}

References

26 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. 2024. Aramed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OS-ACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation@ LREC-COLING 2024, pages 50–56. find our card
  2. Nouar AlDahoul, Talal Rahwan, and Yasir Zaki. 2024a. Polytc: a novel bert-based classifier to detect political leaning of youtube videos based on their titles. Journal of Big Data, 11(1):80. find
  3. Nouar AlDahoul, Myles Joshua Toledo Tan, Har-ishwar Reddy Kasireddy, and Yasir Zaki. 2024b. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123. ↗ our card
  4. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗ our card
  5. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676. ↗ our card
  6. QCRI. 2025. Fanar-1-9b-instruct. https://huggingface.co/QCRI/Fanar-1-9B-Instruct. Accessed: 2025-07-13. ↗
  7. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv preprint arXiv:2505.03427. ↗ our card
  8. Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Towards building multilingual language model for medicine. Nature Communications, 15(1):8384. find
  9. DeepSeek. 2024. Deepseek chat. https://www.deepseek.com. Accessed: July 13, 2025. ↗
  10. Falcon-LLM Team. 2025a. Falcon-arabic: A breakthrough in arabic language models. find
  11. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23. find our card
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗ our card
  13. Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. 2025. O1 replication journey– part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.06458. ↗
  14. Koray Kavukcuoglu. 2025. Gemini 2.5: Our most intelligent ai model. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking. ↗
  15. Chen Wei Kuo, Kevin Chu, Nouar AlDahoul, Hazem Ibrahim, Talal Rahwan, and Yasir Zaki. 2025. Neutralizing the narrative: Ai-powered debiasing of online news articles. arXiv preprint arXiv:2504.03520. ↗
  16. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗ our card
  17. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. arXiv preprint arXiv:2501.00559. ↗ our card
  18. Zabir Al Nazi and Wei Peng. 2024. Large language models in healthcare and medical domain: A review. In Informatics, volume 11, page 57. MDPI. find
  19. OpenAI. 2025. Introducing o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. Accessed: 2025-07-13. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. Fanar: An arabic-centric multimodal generative ai platform. ↗ our card
  20. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗ our card
  21. Meta Team. 2025b. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. find
  22. TII Team. 2024. The falcon 3 family of open models. find
  23. Technology Innovation Institute (TII) UAE. 2024. Falcon3-7b-instruct. https://huggingface.co/ tiiuae/Falcon3-7B-Instruct. xAI. 2025. Grok 3 beta — the age of reasoning agents. ↗
  24. Rui Yang, Ting Fang Tan, Wei Lu, Arun James Thirunavukarasu, Daniel Shu Wei Ting, and Nan Liu. 2023. Large language models in health care: Development, applications, and challenges. Health Care Science, 2(4):255–263. find
  25. Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al. 2022. A large language model for electronic health records. NPJ digital medicine, 5(1):194. find
  26. Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aldahoul-2025-benchmarking-medical-understanding-reasoning-large/