Elamur Research⌕
☰
home / papers / 2026 / EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence

EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence

Paper · 2026 · Chaoyin She, Ruifang Lu, Lida Chen et al.

Summary

General-purpose vision-language models struggle with ultrasound. They lack domain knowledge and do not generalize well across organs, while human reading of sonograms is subjective and slow. The authors present EchoVLM, an open-source ultrasound VLM of roughly 10B parameters built on Qwen2-VL. They gathered data from 15 hospitals: 208,941 cases and 1.47 million key-frame images spanning seven organ systems and more than 100 diseases or findings. From this, a Self-Instruct-style pipeline produced 1.8 million instruction pairs. The pipeline was seeded with expert-written templates, deduplicated, and checked both by automated model judges and by clinician spot-review. Architecturally, a Dual-path MoE module with dynamic routing adds ultrasound knowledge without overwriting what the base model already knows. The model handles report writing, diagnosis, and VQA. In report generation it beats Qwen2-VL by 7.58 BLEU-1 and 3.45 ROUGE-1. This makes it a practical step toward a single ultrasound assistant for multiple organs, with code and weights released publicly.

Key points

  • Dataset scale: 208,941 cases, 1.47M images, and 1.8M instruction pairs covering liver, kidney, thyroid, vessels, gynecology, heart and breast. The pairs were generated from 21 expert templates and deduplicated with ROUGE-L and Simhash.
  • Architecture: a Dual-path MoE added to Qwen2-VL with Top-2 routing. Compared with Top-1, Top-2 raises report-generation BLEU-1 by 3.94 and diagnosis ROUGE-L by 5.00.
  • Expert count: moving from 0 to 4 experts improves report-generation BLEU-1 by 10.15. Going to 6 experts adds little (53.87 → 54.03) but raises training time from 55.36 to 59.10 h/epoch, so 4 experts was chosen.
  • Limitations: vascular cases are scarce in the long-tailed data, and performance there is weaker. On some open-ended VQA metrics, EchoVLM trails Lingshu (7B).
  • Availability: code and weights are released at github.com/Asunatan/EchoVLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 worksCritical Care and Intensive Care Medicine · 1 work

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Chaoyin She, Ruifang Lu, Lida Chen, Wei Wang and Qinghua Huang. 2026. EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.494

@inproceedings{she2026echovlm,
  title = {{EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence}},
  author = {Chaoyin She, Ruifang Lu, Lida Chen, Wei Wang and Qinghua Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.494},
  url = {https://doi.org/10.18653/v1/2026.acl-long.494}
}

References

45 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. 2020. Dataset of breast ultrasound images. Data in brief, 28:104863. find
  2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and 8 others… — Flamingo: a Visual Language Model for Few-Shot Learning
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13…
  4. Haoran Chen, Junyan Lin, Xinhao Chen, Yue Fan, Xin Jin, Hui Su, Jianfeng Dong, Jinlan Fu, and Xiaoyu Shen. 2025. Rethinking visual layer selection in multimodal llms. Preprint, arXiv:2504.21447. — [arXiv:2504.21447](https://arxiv.org/abs/2504.21447) find
  5. Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. 2024. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. Preprint, arXiv:2406.19280. find
  6. Matthew Christensen, Milos Vukadinovic, Neal Yuan, and David Ouyang. 2024. Vision–language foundation model for echocardiogram interpretation. Nature Medicine, 30(5):1481–1488. find
  7. Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, and 5 others. 2024. Mousi: Poly-visual-expert vision-language models. Prepr… — [arXiv:2401.17221](https://arxiv.org/abs/2401.17221) find
  8. Haifan Gong, Guanqi Chen, Ranran Wang, Xiang Xie, Mingzhi Mao, Yizhou Yu, Fei Chen, and Guanbin Li. 2021. Multi-task learning for thyroid nodule segmentation with thyroid region prior. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 257–261. IEEE. find
  9. Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. 2024. Regiongpt: Towards region understanding vision language model. Preprint, arXiv:2403.02330. — [arXiv:2403.02330](https://arxiv.org/abs/2403.02330) find
  10. Cogvlm2: Visual language models for image and video understanding. Preprint, arXiv:2408.16500.
  11. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  12. Oğuzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. 2024. Brave: Broadening the visual encoding of vision-language models. Preprint, arXiv:2404.07204. — [arXiv:2404.07204](https://arxiv.org/abs/2404.07204) find
  13. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408.03326.
  14. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Preprint, arXiv:2306.00890.
  15. Jun Li, Tongkun Su, Baoliang Zhao, Faqin Lv, Qiong Wang, Nassir Navab, Ying Hu, and Zhongliang Jiang. 2024b. Ultrasound report generation with cross-modality feature alignment via unsupervised guidance. IEEE Transactions on Medical Imaging, 44(1):19–30. find
  16. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  17. Xu Li, Yi Zheng, Haotian Chen, Xiaolei Chen, Yuxuan Liang, Chenghang Lai, Bin Li, and Xiangyang Xue. 2026. Instruction-guided fusion of multi-layer visual features in large vision-language models. Pattern Recognition, 170:111932. find
  18. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024a. Video-LLaVA: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971–5984, Miami, Florida, USA. Associat…
  19. Xiaoqing Guo, Mohammad Alsharid, He Zhao, Yipei Wang, Jayne Lander, Aris T Papageorghiou, and J Alison Noble. 2026. A visually grounded language model for fetal ultrasound understanding. Nature Biomedical Engineering, pages 1–17. find
  20. Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024b. VILA: On Pre-training for Visual Language Models . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26679–26689, Los Alamitos, CA, USA. IEEE Computer Society.
  21. Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, and 6 others. 2024. find
  22. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2024a. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations.
  23. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024b. Improved Baselines with Visual Instruction Tuning . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296, Los Alamitos, CA, USA. IEEE Computer Society.
  24. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc.
  25. Fadillah Maani, Numan Saeed, Tausifa Saleem, Zaid Farooq, Hussain Alasmawi, Werner Diehl, Ameera Mohammad, Gareth Waring, Saudabi Valappi, Leanne Bricker, and Mohammad Yaqub. 2025. Fetalclip: A visual-language foundation model for fetal ultrasound image analysis. Preprint, arXiv:2502.14807. — [arXiv:2502.14807](https://arxiv.org/abs/2502.14807) find
  26. Xingqing Nie, Xiaogen Zhou, Tong Tong, Xingtao Lin, Luoyan Wang, Haonan Zheng, Jing Li, Ensheng Xue, Shun Chen, Meijuan Zheng, Cong Chen, and Min Du. 2022. N-net: A novel dense fully convolutional neural network for thyroid nodule segmentation. Frontiers in Neuroscience, Volume 16 - 2022. find
  27. David Ouyang, Bryan He, Amirata Ghorbani, Matthew P. Lungren, Euan A. Ashley, David H. Liang, and James Y. Zou. 2019. Echonet-dynamic: a large new cardiac motion video data resource for medical machine learning. find
  28. Lina Pedraza, Carlos Vargas, Fabián Narváez, Oscar Durán, Emma Muñoz, and Eduardo Romero. 2015. An open access thyroid ultrasound image database. In 10th International symposium on medical information processing and analysis, volume 9287, pages 188–193. SPIE. find
  29. Hanna Piotrzkowska-Wróblewska, Katarzyna Dobruch-Sobczak, Michał Byra, and Andrzej Nowicki. 2017. Open access database of raw ultrasonic signals acquired from malignant and benign breast lesions. Medical Physics, 44(11):6105–6109. find
  30. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.
  31. Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025a. Gemma 3 technical report. Preprint, arXiv:2503.19786. find
  32. LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, and Yu Rong. 2025b. Lingshu: A generalist foundation model for unified mul… — [arXiv:2506.07044](https://arxiv.org/abs/2506.07044) find
  33. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568–9578, Los Alamitos, CA, USA. IEEE Computer Society.
  34. Noelia Vallez, Oscar Deniz, Roberto Espinosa, Enrique Santos, and Gloria Bueno. 2025. Bus-uclm: Breast ultrasound lesion segmentation dataset. Scientific Data, 12(1):1–10. find
  35. Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. 2024. MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15963–15974, Los Alamitos, CA, U… find
  36. Milos Vukadinovic, I-Min Chiu, Xiu Tang, Neal Yuan, Tien-Yu Chen, Paul Cheng, Debiao Li, Susan Cheng, Bryan He, and David Ouyang. 2026. Comprehensive echocardiogram evaluation with view primed vision language ai. Nature, 650(8103):970–977. find
  37. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the…
  38. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, a… find
  39. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Keqin Chen, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2024b. Cogvlm: Visual expert for pre-trained language models. In Advances in Neural Information Processi… find
  40. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Ali… find
  41. Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024a. Long-clip: Unlocking the long-text capability of clip. Preprint, arXiv:2403.15378. — [arXiv:2403.15378](https://arxiv.org/abs/2403.15378) find
  42. Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng YAN. 2024b. OMG-LLaVA: Bridging image-level, object-level, pixel-level reasoning and understanding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  43. Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. 2024c. From redundancy to relevance: Information flow in lvlms across reasoning tasks. Preprint, arXiv:2406.06579.
  44. Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2024d. Llavar: Enhanced visual instruction tuning for text-rich image understanding. Preprint, arXiv:2306.17107. find
  45. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2025. Llava-video: Video instruction tuning with synthetic data. Preprint, arXiv:2410.02713. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/she-2026-echovlm-dynamic-mixture-experts-vision/