Elamur Research⌕
☰
home / papers / 2026 / Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks

Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks

Paper · 2026 · Chaimae Abouzahir, Congbo Ma, Nizar Habash et al.

Summary

The paper investigates why large language models do worse on Arabic medical questions than on English ones. The authors ask whether the cause is the language itself, weak medical adaptation, or design choices in the models. They use MedAraBench, a benchmark of Modern Standard Arabic exam questions, and a Google-Translate English copy of it, so the medical content stays fixed while the language changes. Six open-source models are tested: three general-purpose (DeepSeek-V3.2, LLaMA 3.3 70B, Mistral-Small-3.2-24B) and three medical (Meditron 3 70B, Med42-70B, MedGemma-27B). The analysis looks at:

  • question length, difficulty and specialty,
  • letter-choice versus full-answer-text scoring,
  • tokenizer fragmentation,
  • whether self-reported confidence and explanations track correctness.

Arabic consistently scores lower, and the gap grows as tasks get harder or the output becomes free-form. Arabic medical text is split into many more subword pieces. Model confidence and explanations turn out to be poor signals of correctness. The authors conclude that fixing such gaps needs language-aware choices across tokenization, evaluation and calibration, not just bigger or more specialised models.

Key points

  • MedAraBench has 19 specialties and 4–6 answer options per item. Each question is tagged with a difficulty level by year of study (Y1–Y5). The split is 19,894 train and 4,989 test items, with matched specialty distributions.
  • Decoding is greedy with fixed generation budgets: 4 tokens for letter answers, 15 for answer text and 70 for explanations. The 70B models run in 4-bit quantization.
  • Scoring uses two modes: soft matching (choose an option letter) and hard matching (reproduce the exact answer text). This separates instruction-following and surface-form problems from reasoning errors.
  • Arabic is fragmented far more by subword tokenizers. Multilingual tokenizers reduce this somewhat, and Arabic-focused ones reduce it further. Longer effective inputs plausibly explain the sharper Arabic decline on complex questions.
  • Self-reported confidence relates negatively to correctness. Asking for explanations shifts model behaviour in ways that depend on the model and the language, rather than improving accuracy.

Interesting ideas

  • I1. Translating a benchmark while keeping its gold labels isolates the effect of language from the effect of domain knowledge, since the content stays constant across conditions.
  • I2. Scoring the same answer two ways, by option letter and by exact text, shows how much of a model's error is format or alignment noise rather than wrong reasoning.
  • I3. Measuring how many subword pieces domain terms break into, and relating that to accuracy, gives a structural explanation for performance drops instead of just an aggregate score.
  • I4. Breaking accuracy down by input length, difficulty level and topic exposes weaknesses that a single average hides.
  • I5. Self-confidence and generated rationales should not be trusted as reliability signals without checking them against correctness.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could run the same school tests or fables through paraphrased or simplified English variants with fixed answers, to see whether failures come from surface form or from understanding.
  • I2 (answer): Mova's readers could report both "picked the right option" and "produced the right answer phrase", so generation slips are not counted as comprehension failures.
  • I3 (tokens): tracking fragmentation of rare or technical words in the en tokenizer and lemmatizer could reveal where morphology analysis breaks before tagging and parsing errors spread.
  • I4 (learn): evaluation reports for gparse or the perceptron parser could be stratified by sentence length and genre, alongside the negative controls already in use.
  • I5 (learn): when LLM teacher tiers supply silver labels, their stated confidence or explanations should be calibrated against agreement among the witness parsers rather than taken at face value.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Chaimae Abouzahir, Congbo Ma, Nizar Habash and Farah E. Shamout. 2026. Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks. Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.healing-1.13

@inproceedings{abouzahir2026cross,
  title = {{Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks}},
  author = {Chaimae Abouzahir, Congbo Ma, Nizar Habash and Farah E. Shamout},
  booktitle = {Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.healing-1.13},
  url = {https://doi.org/10.18653/v1/2026.healing-1.13}
}

References

38 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mouath Abu-Daoud, Leen Kharouf, Omar El Hajj, Dana El Samad, Mariam Al-Omari, Jihad Mallat, Khaled Saleh, Nizar Habash, and Farah E. Shamout. 2026. Medarabench: Large-scale arabic medical question answering dataset and benchmark. Preprint, arXiv:2602.01714. ↗
  2. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. 2024. AraMed: Arabic Medical Question Answering using Pretrained Transformer Language Models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 50–56, Torino, Italia. ELRA and ICCL.
  3. Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. MedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering. Artificial Intelligence in Medicine, 155:102938. ↗
  4. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. Majed Alrubaian, Ali Alammari, Zaki Alawami, and 7 others. 2025. AL-LaM: Large Language Models for Arabic and English. In The Thirteenth International Conference on Learning Representations. Med42-v2: A Suite of Clinical LLMs. Preprint, arXiv:2408.06142. ↗
  5. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E. Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. Preprint, arXiv:2505.03427. ↗
  6. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. Deepseek-v3.2: Pushing the Frontier of Open Large Language Models. Preprint, arXiv:2512.02556. ↗
  7. Epoch AI. 2024. Data on Notable AI Models. https://epoch.ai/data/notable-ai-models. Accessed: 2025-05-11. ↗
  8. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22. ↗
  9. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  10. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  11. Daniel P Jeong, Saurabh Garg, Zachary Chase Lipton, and Michael Oberst. 2024. Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12143–12170, Miami, Florida, USA. Association for Computational Linguistics. find
  12. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What Disease Does This Patient Have? A Large-Sale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences, 11(14). find
  13. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, and Benyou Wang. 2025. Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14552–14573, Vienna, Austria. Association for Computational Linguistics. find
  14. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567– 2577, Hong Kong, China. Association for Computational Linguistics. ↗
  15. Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. find
  16. Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. 2024. find
  17. Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 2627–2638, New York, NY, USA. Association for Computing Machinery. Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260. PMLR. find
  18. Vignesh Kumar Kembu, Pierandrea Morandini, Marta Bianca Maria Ranzini, and Antonino Nocera. 2025. Are LLMs Truly Multilingual? Exploring Zero-Shot Multilingual Capability of LLMs for Information Retrieval: An Italian Healthcare Use Case. Preprint, arXiv:2512.04834. ↗
  19. Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Towards building multilingual language model for medicine. Nature Communications, 15(1):8384. ↗
  20. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5848–5864, Bangkok, Thailand. Association for Computational Linguistics.
  21. Meta AI. 2024. Llama 3.3 70b instruct. https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct. Hugging Face model card. ↗
  22. Mistral AI. 2025. Mistral-small-3.2-24b-instruct-2506. https://huggingface.co/mistralai/ Mistral-Small-3.2-24B-Instruct-2506. Hugging Face model card. ↗
  23. Moshe Munk and Dror G. Feitelson. 2022. When Are Names Similar Or the Same? Introducing the Code Names Matcher Library. Preprint, arXiv:2209.03198. ↗
  24. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023a. Capabilities of GPT-4 on Medical Challenge Problems. Preprint, arXiv:2303.13375. ↗
  25. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. 2023b. Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine. Preprint, arXiv:2311.16452. ↗
  26. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  27. OpenMeditron Initiative. 2024. Llama-3.1 meditron-3 [70b]. https://huggingface.co/OpenMeditron/ Meditron3-70B. Hugging Face model card; publication forthcoming. ↗
  28. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A Large-scale Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, and 62 others. 2025. MedGemma Technical Report. Preprint, arXiv:2507.05201. ↗
  29. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models. Preprint, arXiv:2308.16149. ↗
  30. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics.
  31. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H. Chen, Nigam H. Shah, Sami Lachgar, Philip Andrew Mansfield, and 16 others. 2025. Toward Expert-Level Medical Question Answering with Large Language Models. Nature Medicine, 31(3):943–950. find
  32. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. 2025. Fanar: An Arabic-Centric Multimodal Generative AI Platform. arXiv:2501.13944. ↗
  33. Preprint, Juraj Vladika, Mahdi Dhaini, and Florian Matthes. 2025. Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 9161–9174, Suzhou, China. Association for Computational Linguistics. ↗
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  35. Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, Yihan Cao, Hui Ren, Xiang Li, Xiaoxiao Li, and Yuyin Zhou. 2025. MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs. Preprint, arXiv:2504.00993. ↗
  36. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking Retrieval-Augmented Generation for Medicine. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, Bangkok, Thailand. Association for Computational Linguistics.
  37. Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, and Haizhou Li. 2023. HuatuoGPT, Towards Taming Language Model to Be a Doctor. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10859–10885, Singapore. Association for Computational Linguistics. ↗
  38. Xiao Zhang, Ji Wu, Zhiyang He, Xien Liu, and Ying Su. 2018. Medical Exam Question Answering with Large-Scale Reading Comprehension. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abouzahir-2026-cross-lingual-empirical-evaluation-large/