Elamur Research⌕
☰
home / papers / 2025 / Bridging Language Barriers in Healthcare: A Study on Arabic LLMs

Bridging Language Barriers in Healthcare: A Study on Arabic LLMs

Summary

The authors, from M42 Health, ask why open LLMs that handle English medical questions well perform much worse when the same questions are posed in Arabic. They also ask how to close that gap. They first benchmark several open model families (Qwen2.5, Llama3.1, Mistral, Pangea, Silma, Med42, Meditron3, BiMedix) on Arabic-translated versions of PubMedQA, MedMCQA, MedQA and Medical MMLU. For this they adapt the Harness framework to Arabic script and score whole answer strings. Next they test a translate-to-English pipeline using dedicated MT systems and general LLMs as translators. Finally they fine-tune Llama 3.1 (8B and 70B, base and instruct) on original and synthetic Arabic medical data mixed with English in different proportions. The best language ratio turns out to depend on the task. Larger models favour Arabic-only data more uniformly, and an already-aligned 70B instruct model barely improves. The authors conclude that fine-tuning alone is a weak way to add a language and that heavier pretraining may still be required. The work matters because medical AI should serve communities whose language has little in-domain data.

Key points

  • Llama3.1 shows a large drop from English to Arabic on MedQA. The 8B model falls from 62.0 to 29.5 and the 70B model from 78.2 to 56.6. Qwen2.5 holds up comparatively better in Arabic.
  • The Arabic benchmarks come from BiMediX's semi-automated translation process, in which LLM drafts are refined by humans. Question and answer format is kept intact so the two languages can be compared directly.
  • In the translate-then-answer setup, general LLMs outperform specialised MT models as translators. For example, on MedQA the Qwen2.5-72B translator reaches 71.3, compared with 31.2 for the Flores 101 model. The drawback is that the pipeline needs at least three model calls.
  • Fine-tuning used two epochs with a cosine LR from 1e-5 to 1e-6, 8192-token packed chunks, loss computed only on response tokens, and 4 H100 nodes.
  • The best data mix varies by task:
    • PubMedQA is best with Arabic-only data (71.2).
    • MedMCQA is best with a strong Arabic majority (35.1).
    • MedQA is best with an English majority (29.8).
    • MMLU is best with English-only data (42.4, against a zero-shot score of 39.8).

Interesting ideas

  • I1. Multiple-choice items are scored by the likelihood of the full answer text rather than of the option letter, which reveals more about what the model actually understands. — For Mova (answer): School-test readers could rank each option by how well its complete content fits the world of the text, instead of picking a label, so every choice gets an explainable justification.
  • I2. Translation quality is measured indirectly, by how well a fixed strong model answers questions on the translated material, instead of by surface similarity to a reference. — For Mova (back): Translation through the meta-world could be judged by whether Mova's own readers still answer correctly after the text → graph → text round trip, which tests meaning preservation directly.
  • I3. The authors sweep several language ratios in the training mix and find that each task has a different optimum, so no single "right" blend exists. — For Mova (learn): When combining silver parses from witness parsers with gold treebanks, the ratio could be tuned separately for each downstream stage rather than fixed once globally.
  • I4. Loss is applied only to response tokens, so the model learns to produce answers rather than to reproduce prompts. — For Mova (learn): When distilling from LLM teacher tiers, supervision could be restricted to the target structure (frames, roles) and not to the context the teacher was given.
  • I5. Every fine-tuned variant is compared against its untouched starting model, and the authors report honestly that the aligned 70B instruct model rarely improved. — For Mova (learn): Any new training run or self-induced repair rule should be required to beat the frozen baseline, which works as a negative control and catches changes that only appear to help.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.09825 · arXiv 2501.09825

@misc{saadi2025bridging,
  title = {{Bridging Language Barriers in Healthcare: A Study on Arabic LLMs}},
  author = {Saadi, Nada and Raha, Tathagata and Christophe, Clement and Pimentel, Marco AF and Rajan, Ronnie and Kanithi, Praveen K},
  year = {2025},
  doi = {10.48550/arxiv.2501.09825},
  eprint = {2501.09825},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.09825}
}

References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alyafeai, Z.; Almubarak, K.; Ashraf, A.; Alnuhait, D.; Alshahrani, S.; Abdulrahman, G. A. Q.; Ahmed, G.; Gawah, Q.; Saleh, Z.; Ghaleb, M.; Ali, Y.; and Al-Shaibani, M. S. 2024. CIDAR: Culturally Relevant Instruction Dataset For Arabic. arXiv:2402.03177. ↗
  2. Ankit Pal, M. S. 2024. OpenBioLLMs: Advancing Open-Source Large Language Models for Healthcare and Life Sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B. ↗
  3. Chen, Z.; Cano, A. H.; Romanou, A.; Bonnet, A.; Matoba, K.; Salvi, F.; Pagliardini, M.; Fan, S.; Köpf, A.; Mohtashami, A.; Sallinen, A.; Sakhaeirad, A.; Swamy, V.; Krawczuk, I.; Bayazit, D.; Marmet, A.; Montariol, S.; Hartley, M.-A.; Jaggi, M.; and Bosselut, A. 2023. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. arXiv preprint arXiv:2311.16079. ↗ our card
  4. Christophe, C.; Kanithi, P. K.; Raha, T.; Khan, S.; and Pimentel, M. A. 2024. Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142. ↗
  5. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  6. Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac’h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; and Zou, A. 2024. A framework for few-shot language model evaluation. find
  7. Helsinki-NLP. 2024. opus-mt-ar-en. https://huggingface.co/ Helsinki-NLP/opus-mt-ar-en. ↗
  8. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR). Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. ↗ our card
  9. Jin, D.; Pan, E.; Oufattole, N.; Weng, W.-H.; Fang, H.; and Szolovits, P. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14): 6421. find our card
  10. Jin, Q.; Dhingra, B.; Liu, Z.; Cohen, W.; and Lu, X. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2567–2577. find our card
  11. Jin, Y.; Chandra, M.; Verma, G.; Hu, Y.; De Choudhury, M.; and Kumar, S. 2024. Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. In Proceedings of the ACM Web Conference 2024, WWW ’24, 2627–2638. New York, NY, USA: Association for Computing Machinery. ISBN 9798400701719. find
  12. Kanithi, P. K.; Christophe, C.; Pimentel, M. A. F.; Raha, T.; Saadi, N.; Javed, H.; Maslenkova, S.; Hayat, N.; Rajan, R.; and Khan, S. 2024. MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications. arXiv:arXiv:2409.07314. ↗
  13. Lopez, M.; Parikh, A.; Yacouby, R.; and et al. 2023. GatorTron-M: Multilingual Clinical Language Models for Medical Natural Language Processing. find
  14. Lu, Y.; Zhu, W.; Li, L.; Qiao, Y.; and Yuan, F. 2024. LLa-MAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages. arXiv preprint arXiv:2407.05975. ↗
  15. Nori, H.; King, N.; McKinney, S. M.; Carignan, D.; and Horvitz, E. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. ↗ our card
  16. Pal, A.; Umapathi, L. K.; and Sankarasubbu, M. 2022. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. In Flores, G.; Chen, G. H.; Pollard, T.; Ho, J. C.; and Naumann, T., eds., Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, 248–260. PMLR. find our card
  17. Pieri, S.; Mullappilly, S. S.; Khan, F. S.; Anwer, R. M.; Khan, S.; Baldwin, T.; and Cholakkal, H. 2024a. Bimedix: Bilingual medical mixture of experts llm. arXiv preprint arXiv:2402.13253. ↗
  18. Pieri, S.; Mullappilly, S. S.; Khan, F. S.; Anwer, R. M.; Khan, S.; Baldwin, T.; and Cholakkal, H. 2024b. BiMediX: Bilingual Medical Mixture of Experts LLM. arXiv:2402.13253. ↗
  19. Sengupta, N.; Sahu, S. K.; Jia, B.; Katipomu, S.; Li, H.; Koto, F.; Marshall, W.; Gosal, G.; Liu, C.; Chen, Z.; et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. seyoungsong. 2024. flores101mm100175M. https://huggingface.co/seyoungsong/flores101-\ mm100\ 175M. ↗ our card
  20. Silma-AI. 2024. SILMA 1.0. https://huggingface.co/silma-ai/SILMA-9B-Instruct-v1.0. ↗
  21. Wang, X.; Chen, N.; Chen, J.; Hu, Y.; Wang, Y.; Wu, X.; Gao, A.; Wan, X.; Li, H.; and Wang, B. 2024. Apollo: Lightweight multilingual medical llms towards democratizing medical ai to 6b people. arXiv preprint arXiv:2403.03640. ↗
  22. Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; Dong, G.; Wei, H.; Lin, H.; Tang, J.; Wang, J.; Yang, J.; Tu, J.; Zhang, J.; Ma, J.; Xu, J.; Zhou, J.; Bai, J.; He, J.; Lin, J.; Dang, K.; Lu, K.; Chen, K.; Yang, K.; Li, M.; Xue, M.; Ni, N.; Zhang, P.; Wang, P.; Peng, R.; Men, R.; Gao, R.; Lin, R.; Wang, S.; Bai, S.; Tan, S.; Zhu, T.; Li, T.; Liu, T.; Ge, W.; Deng, X.; Zhou, X.; Ren, X.; Zhang, X.; Wei, X.; Ren, X.; Fan, Y.; Yao, Y.; Zhang, Y.; Wan, Y.; Chu, Y.; Liu, Y.; Cui, Z.; Zhang, Z.; and Fan, Z. 2024. Qwen2 Technical Report. arXiv preprint arXiv:2407.10671. ↗
  23. Yue, X.; Song, Y.; Asai, A.; Kim, S.; de Dieu Nyandwi, J.; Khanuja, S.; Kantharuban, A.; Sutawika, L.; Ramamoorthy, S.; and Neubig, G. 2024. Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages. arXiv preprint arXiv:2410.16153. ↗
  24. Zhang, X.; Wang, Y.; and Chen, H. 2023. Large Language Models in Medicine: The Potentials and Pitfalls. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saadi-2025-bridging-language-barriers-healthcare-study/