Elamur Research⌕
☰
home / papers / 2025 / NYUAD at AraHealthQA Shared Task: Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks

NYUAD at AraHealthQA Shared Task: Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks

Paper · 2025 · Nouar AlDahoul and Yasir Zaki

Summary

The paper asks how well current large language models handle medical questions written in Arabic, a domain where most evaluation resources are English-only. The authors entered the AraHealthQA shared task (MedArabiQ2025 track), which uses an MSA benchmark of 700 clinical items. A further 200 items, half MCQs and half open-ended, were released later for testing. They queried a broad set of models zero-shot with deterministic decoding:

  • proprietary systems (GPT-4o, GPT o3, Gemini Flash 2.5 and Pro 2.5, Claude 3 Opus, Grok 3, Deepseek v3, Llama 4 Maverick);
  • open Arabic-centric models (Falcon3, Fanar, ALLaM).

Several prompt variants were compared. For multiple-choice items, reasoning-oriented models clearly led. A simple majority vote over GPT o3 and the two Gemini 2.5 models reached 77% accuracy and took first place in the challenge. For open-ended answers, scored with BERTScore, the best result was 0.8644. However, scores clustered tightly, which shows the metric's weak discrimination. The open Arabic models trailed far behind. The work matters as a reference point showing that even top models have clear gaps in Arabic medical knowledge.

Key points

  • The dataset contains seven 100-item subsets:
    • plain MCQs and bias-injected MCQs;
    • fill-in-the-blank with choices and without choices;
    • AraMed patient-doctor Q&As;
    • grammar-corrected Q&As;
    • LLM-modified Q&As meant to counter memorization.
  • All runs used zero-shot prompting with temperature 0 and top_p 1. A chain-of-thought MCQ prompt (Prompt 2) generally beat the plain one.
  • MCQ accuracy ranged widely:
    • Gemini Pro 2.5 scored 76%, GPT o3 and Gemini Flash 2.5 74%, and the majority vote 77%.
    • Llama 4 Maverick scored 63%, while GPT-4o-mini and Claude 3 Opus scored 49%.
    • Falcon Arabic, tested manually through its web chat, scored 38%.
  • Open-ended answers were scored with BERTScore on XLM-RoBERTa-Large:
    • Gemini Pro 2.5 reached 0.8644 and GPT-4o-mini 0.8581.
    • The winning prompt demanded MSA, medical correctness and brevity.
  • Among the open Arabic models, ALLaM led on MCQs (39%; Falcon3 36%, Fanar 31%). Falcon3 led on BERTScore (0.8493).

Interesting ideas

  • I1. A plain majority vote among a few strong but different models beat every single model, at almost no engineering cost.
  • I2. The benchmark adds perturbed copies of items (bias-injected, LLM-rewritten, grammar-corrected) to tell real reasoning apart from memorized answers.
  • I3. Asking the same fill-in-the-blank question with and without answer options separates recognizing an answer from producing one.
  • I4. The authors admit that BERTScore packed all systems into a narrow band (about 0.81–0.86), so it barely ranks answers.
  • I5. Telling the model to answer briefly, without explanations, raised alignment with reference answers. Graders reward concise outputs that match the reference's form.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): this supports building silver UD trees and frames from agreement between independent witness parsers. Disagreements can be routed to an LLM teacher tier rather than trusted.
  • I2 (learn): rewritten paraphrases of school tests and fables can act as negative controls. A reader that only passes the original wording is relying on surface cues.
  • I3 (answer): test readers in both modes. A gap between them reveals whether a failure comes from building the world of the text or from verbalizing the answer.
  • I4 (answer): grade free-form answers by matching frames or entities in the meaning graph instead of embedding similarity. This keeps scores discriminative and explainable.
  • I5 (back): when generating answers from the graph, emit the minimal phrase that satisfies the question. Justification should be a separate explainable trace, not padding in the answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 works

Cite

Cite as Nouar AlDahoul and Yasir Zaki. 2025. NYUAD at AraHealthQA Shared Task: Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.19

@inproceedings{aldahoul2025nyuad,
  title = {{NYUAD at AraHealthQA Shared Task: Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks}},
  author = {Nouar AlDahoul and Yasir Zaki},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.19},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.19}
}

References

20 references, 13 of them with a card of ours. ↗ leads to the work.

  1. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. ↗
  2. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. 2024. Aramed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OS-ACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation@ LREC-COLING 2024, pages 50–56.
  3. Nouar AlDahoul, Talal Rahwan, and Yasir Zaki. 2024a. Polytc: a novel bert-based classifier to detect political leaning of youtube videos based on their titles. Journal of Big Data, 11(1):80. ↗
  4. Nouar AlDahoul, Myles Joshua Toledo Tan, Har-ishwar Reddy Kasireddy, and Yasir Zaki. 2024b. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123. ↗
  5. Hassan Alhuzali, Farah Shamout, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Walid Al-Eisawi, Renad Al-Monef, Ali Alqahtani, Lama Ayash, et al. 2025. Arahealthqa 2025 shared task description paper. arXiv preprint arXiv:2508.20047. ↗
  6. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867. ↗
  7. Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323. ↗
  8. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390. ↗
  9. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676. ↗
  10. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv preprint arXiv:2505.03427. ↗
  11. Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. 2025. O1 replication journey– part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.06458. ↗
  12. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗
  13. Chen Wei Kuo, Kevin Chu, Nouar AlDahoul, Hazem Ibrahim, Talal Rahwan, and Yasir Zaki. 2025. Neutralizing the narrative: Ai-powered debiasing of online news articles. arXiv preprint arXiv:2504.03520. ↗
  14. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  15. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. ↗
  16. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. arXiv preprint arXiv:2501.00559. ↗
  17. Zabir Al Nazi and Wei Peng. 2024. Large language models in healthcare and medical domain: A review. In Informatics, volume 11, page 57. MDPI. ↗
  18. Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Towards building multilingual language model for medicine. Nature Communications, 15(1):8384. ↗
  19. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. ↗
  20. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aldahoul-2025-nyuad-arahealthqa-shared-task-benchmarking/