Elamur Research⌕
☰
home / papers / 2024 / Towards Building Multilingual Language Model for Medicine

Towards Building Multilingual Language Model for Medicine

Paper · 2024 · Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang et al.

Summary

General-purpose multilingual LLMs such as BLOOM and InternLM 2 handle medical questions poorly outside English, mostly because their training data contain little medical text. The authors address this with three open resources. MMedC is a medical corpus of roughly 25.5B tokens spanning six languages, namely Spanish, Russian, French, Japanese, Chinese and English. It was assembled from four sources: an automatic filter applied to large general multilingual corpora, OCR-processed medical textbooks, medical websites, and existing small medical datasets. MMedBench merges established medical multiple-choice exam datasets for the same languages and adds GPT-4-written explanations, so it tests both answer accuracy and rationale generation. The team evaluated many open and closed LLMs in zero-shot, PEFT (LoRA) and full fine-tuning setups, and checked automatic rationale metrics against human ratings. Continued auto-regressive training on MMedC consistently improved results. The resulting 8B model, MMed-Llama 3, leads all open-source models on MMedBench and on English medical benchmarks, and comes near GPT-4. The work shows that domain-specific multilingual pre-training data can close much of the gap. Ukrainian is not among the covered languages.

Key points

  • MMedC contains ~25.5B tokens. English is the largest share at about 42%, and Russian is the smallest at about 7% (~2B tokens). It is released on Hugging Face (Henrychur/MMedC) under CC BY-NC-SA; copyrighted textbooks are listed by title only.
  • MMedBench has 53,566 QA pairs across 21 medical fields, split into 45,048 train and 8,518 test pairs. A subset of 1,136 test items has human-checked rationales for reasoning evaluation. Sources include MedQA, IgakuQA, FrenchMedMCQA, RuMedDaNet and Head-QA.
  • Accuracy on MMedBench:
    • MMed-Llama 3: 67.75%, up from 62.79% for Llama 3 (+4.96).
    • MMedLM 2: 67.30%, up from 58.59% for InternLM 2 (+8.71).
    • GPT-4 reference: 74.27%.
  • Training setup:
    • Continued pre-training used 2048-token chunks with a 512-token overlap.
    • It ran with FSDP and BF16 on eight A100 GPUs; the 7B run took roughly 20 days.
    • LoRA fine-tuning used rank 16.
  • Released assets: code (CC BY-SA), plus the MMedLM, MMedLM 2 and MMed-Llama 3 weights. An English instruction-tuned variant, MMed-Llama-3-8B-EnIns, is also available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China63 works · 15,047 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China44 works · 10,162 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksMultilingual & cross-lingual NLP · 467 works

Cite

Cite as Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang and Weidi Xie. 2024. Towards Building Multilingual Language Model for Medicine. arXiv:2402.13963.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.13963 · arXiv 2402.13963

@misc{qiu2024building,
  title = {{Towards Building Multilingual Language Model for Medicine}},
  author = {Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang and Weidi Xie},
  year = {2024},
  doi = {10.48550/arxiv.2402.13963},
  eprint = {2402.13963},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.13963}
}

References

53 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
  2. Rohan Anil et al. Palm 2 technical report. ArXiv, abs/2305.10403, 2023.
  3. BIT-ENGD. baidu_baike. https://github.com/BIT-ENGD/baidu_baike, Accessed: Apr. 2024. ↗
  4. Pavel Blinov, Arina Reshetnikova, Aleksandr Nesterov, Galina Zubkova, and Vladimir Kokh. Rumedbench: A russian medical language understanding benchmark. ArXiv, abs/2201.06499, 2022. 2 https://www.apache.org/licenses/LICENSE-2.0 3 https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/LICENSE — [arXiv:2201.06499](https://arxiv.org/abs/2201.06499) ↗
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…
  6. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. ArXiv, abs/2311.16079, 2023.
  7. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024.
  8. Common Crawl. Common crawl maintains a free, open repository of web crawl data that can be used by anyone. https://commoncrawl.org/, Accessed: Apr. 2024. ↗
  9. Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. ArXiv, abs/2306.12420, 2023. find
  10. Wikimedia Foundation. Wikimedia downloads. https://dumps.wikimedia.org, Accessed: May. 2024. ↗
  11. Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. Medalpaca–an open-source collection of medical conversational ai models and training data. ArXiv, abs/2304.08247, 2023.
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020.
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
  14. Institute of Formal and Applied Linguistics. Ufal medical corpus. https://ufal.mff.cuni.cz/ufal_medical_ corpus, Accessed: Nov. 2024. ↗
  15. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  16. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081, 2020.
  17. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Conference on Empirical Methods in Natural Language Processing, 2019.
  18. Dan W Joyce, Andrey Kormilitzin, Katharine A Smith, and Andrea Cipriani. Explainable artificial intelligence for mental health through transparency and interpretability for understandability. npj Digital Medicine, 6(1):6, 2023. find
  19. Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. Evaluating GPT-4 and ChatGPT on Japanese medical licensing examinations. ArXiv, abs/2303.18027, 2023. find
  20. Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page…
  21. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digital health,…
  22. Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. Frenchmedmcqa: A french multiple-choice question answering dataset for medical domain. ArXiv, abs/2304.04280, 2023. find
  23. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains. ArXiv, abs/2402.10373, 2024.
  24. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv, abs/2005.11401, 2020. find
  25. Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6), 2023.
  26. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  27. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631–22648. PMLR, 2023.
  28. Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259–265, 2023. find
  29. Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-flamingo: A multimodal medical few-shot learner. ArXiv, abs/2307.15189, 2023. find
  30. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. ArXiv, abs/2309.09400, 2023.
  31. Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. ArXiv, abs/2311.16452, 2023. find
  32. Jesutofunmi A Omiye, Jenna C Lester, Simon Spichak, Veronica Rotemberg, and Roxana Daneshjou. Large language models propagate race-based medicine. npj Digital Medicine, 6(1):195, 2023. find
  33. OpenAI. Openai. introducing chatgpt. https://openai.com/blog/chatgpt/, Accessed: Dec. 2024. ↗
  34. Ankit Pal, Pasquale Minervini, Andreas Geert Motzfeldt, and Beatrice Alex. Open medical llm leaderboard. https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard, Accessed: Apr. 2024. ↗
  35. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann, editors, Proceedings of the Conference on Health, Inference…
  36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelp…
  37. BigScienceWorkshop Scao et al. Bloom: A 176b-parameter open-access multilingual language model. Nov 2022. find
  38. K. Singhal, Shekoofeh Azizi, Tao Tu, Said Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather J. Cole-Lewis, Stephen J. Pfohl, P A Payne, Martin G. Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, P. A. Mansfield, Blaise Agüera y Arcas, Dale…
  39. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. ArXiv, abs/2305.09617, 2023.
  40. Gemini Team. Gemini: A family of highly capable multimodal models. ArXiv, abs/2312.11805, 2023.
  41. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. ArXiv, abs/2403.08295, 2024. find
  42. InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM, Accessed: Feb. 2024. ↗
  43. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023.
  44. Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David J. Fleet, P. A. Mansfield, Sushant Prakash, Renee C Wong, Sunny Virmani, Christopher Semtur… find
  45. David Vilares and Carlos Gómez-Rodríguez. HEAD-QA: A healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, Florence, Italy, July 2019. Association for Computational Linguistics.
  46. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171, 2022.
  47. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. ArXiv, abs/2204.07705, 2022. find
  48. Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pmc-llama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, page ocae045, 2024. find
  49. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology. ArXiv, abs/2308.02463, 2023. find
  50. Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Robyn Fong, Curran Phillips, Kevin Alexander, Euan Ashley, Jack Boyd, Kathleen Boyd, Karen Hirsch, Curt Langlotz, Rita Lee, Joanna Melia, Joanna Nelson, Karim Sallam, Stacey Tullis, Melissa Ann Vogelsong, John Pa… find
  51. Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. Retrieve anything to augment large language models. ArXiv, abs/2310.07554, 2023. — [arXiv:2310.07554](https://arxiv.org/abs/2310.07554) find
  52. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019. find
  53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. \#\# 8 Acknowledgement This work is supported by… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qiu-2024-towards-building-multilingual-language-model/