Elamur Research⌕
☰
home / papers / 2026 / BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs

BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs

Paper · 2026 · Mainak Singha, Tanisha Gupta, Ankit Jha et al.

Summary

Biomedical vision-language models such as BioMedCLIP work well on average, but they lose accuracy on hard imaging modalities. This is most visible when classes look very similar, when acquisition conditions vary, and when only a few labelled examples exist. BioVLM addresses this through prompt learning on a frozen BioMedCLIP backbone, without fine-tuning the backbone itself.

The method works in three steps:

  • An LLM generates visual attributes for each class, and each attribute gets its own learnable prompt, forming a diverse prompt bank.
  • For every image, the model keeps only the prompts whose predictive distribution has low entropy, so prompts are selected per input rather than relying on one fixed prompt.
  • Confident LLM-derived attributes are distilled into the prompts, with consistency enforced between strongly and weakly augmented views.

At test time the model picks the prompts that suit the target modality, with no LLM needed during inference. Across 11 MedMNIST+ 2D datasets, BioVLM sets a new state of the art in few-shot, base-to-new and out-of-distribution evaluation, while keeping the trainable parameter count small. This matters for clinical imaging, where labelled data is scarce and modalities differ widely.

Key points

  • Base-to-new results: BioVLM reaches a harmonic mean of 51.77%, compared with 42.54% for the previous best method, BioMedCoOp.
  • Few-shot results: Averaged few-shot accuracy is 56.51% against 41.37% for BioMedCoOp. The model trains only about 0.031M parameters, versus 5.35M for MaPLe.
  • Out-of-distribution results: Average OOD accuracy is 32.34%, about 0.8 points above BioMedCoOp, though the gain is not uniform: BioVLM is worse on some datasets, such as RetinaMNIST (−3.31).
  • Prompt selection ablation: Entropy-based selection outperforms softmax-based selection by roughly 5.5–5.9 points and mean-based selection by roughly 2.3–3.4 points.
  • Ablations on prompts and losses: Accuracy rises as prompts per class increase from 1 to 10, then plateaus or slightly drops at 50–100 prompts. Each added loss term (L_LER, L_CMD, L_ASA) improves over cross-entropy alone. The authors list limitations: dependence on LLM attribute quality and evaluation on 2D data only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of TrentoTrento · Italy85 works · 15,524 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🎓Indian Institute of Technology BombayMumbai · India29 works · 2,104 citations🎓LNM Institute of Information TechnologyJaipur · India2 works · 351 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Mainak Singha, Tanisha Gupta, Ankit Jha, Muhammad Haris Khan, Sayantani Ghosh and Biplab Banerjee. 2026. BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2036

@inproceedings{singha2026biovlm,
  title = {{BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs}},
  author = {Mainak Singha, Tanisha Gupta, Ankit Jha, Muhammad Haris Khan, Sayantani Ghosh and Biplab Banerjee},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2036},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2036}
}

References

59 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.
  2. Michael Marchetti, and 1 others. 2019. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368. — [arXiv:1902.03368](https://arxiv.org/abs/1902.03368) find
  3. Sedigheh Eslami, Gerard De Melo, and Christoph Meinel. 2021. Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906. — [arXiv:2112.13906](https://arxiv.org/abs/2112.13906) find
  4. Zhe Gan, Yen-Chun Chen, Linjie Li, Yu Cheng, Jingjing Liu, and Lijuan Wang. 2022. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends in Computer Graphics and Vision, 14(3):163–352. find
  5. Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2021. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544. find
  6. Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Tsung-Yi Lin, Quoc V Le, Zhengyuan Deng, and Xiaohua Zhai. 2022. Scaling open-vocabulary image segmentation with image-level labels. In ECCV. find
  7. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. 2019. Dataset of breast ultrasound images. data brief 28, 104863 (2020). Crossref, Web of Science. find
  10. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. find
  11. Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, and 1 others. 2022. Making the most of text semantics to improve biomedical vision–language processing. In European conferenc… find
  12. Asif Hanif, Fahad Shamshad, Muhammad Awais, Muzammal Naseer, Fahad Shahbaz Khan, Karthik Nandakumar, Salman Khan, and Rao Muhammad Anwer. 2024. Baple: Backdoor attacks on medical foundational models using prompt learning. In International Conference on Medical Image Computing and Computer-Assisted I… find
  13. Andrew A Borkowski, Marilyn M Bui, L Brannon Thomas, Catherine P Wilson, Lauren A DeLand, and Stephen M Mastorides. 2019. Lung and colon cancer histopathological image dataset (lc25000). arXiv preprint arXiv:1912.12142. — [arXiv:1912.12142](https://arxiv.org/abs/1912.12142) find
  14. Shirsha Bose, Ankit Jha, Enrico Fini, Mainak Singha, Elisa Ricci, and Biplab Banerjee. 2024. Stylip: Multi-scale style-conditioned prompt learning for clip-based domain generalization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5542–5552. find
  15. Pingjun Chen. 2018. Knee osteoarthritis severity grading dataset. Mendeley Data, 1(10.17632):30784984. find
  16. Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of…
  17. Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. 2024. Lp++: A surprisingly strong linear probe for few-shot clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23773–23782. find
  18. Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. 2023. A visual– language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307–2316. find
  19. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  20. Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. 2023. Quilt-1m: One million image-text pairs for histopathology. Advances in neural information processing systems, 36:37995–38017. find
  21. Md Nazmul Islam, Mehedi Hasan, Md Kabir Hossain, Md Golam Rabiul Alam, Md Zia Uddin, and Ahmet Soylu. 2022. Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from ct-radiography. Scientific Reports, 12(1):11440. find
  22. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR.
  23. Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In European Conference on Computer Vision, pages 709– 727. Springer. find
  24. Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner. 2016. Multi-class texture analysis in colorectal cancer histology. Scientific reports, 6(1):27988. find
  25. Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, and 1 others. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning. cell, 172(5):1122–1131. find
  26. Momin Ahmad Khan, Yasra Chandio, Eugene Bagdasarian, and Fatima Anwar. 2025. Compromising federated medical ai-backdoor risks in prompt learning. In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems, pages 630–631. find
  27. Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023a. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19113–19122. find
  28. Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2023b. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15190– 15200. find
  29. Konwoo Kim, Michael Laskin, Igor Mordatch, and Deepak Pathak. 2021. How to adapt your large-scale vision-and-language model. find
  30. Thomas Köhler, Attila Budai, Martin F Kraus, Jan Odstrčilik, Georg Michelson, and Joachim Hornegger. 2013. Automatic no-reference quality assessment for retinal fundus images using vessel segmentation. In Proceedings of the 26th IEEE international symposium on computer-based medical systems, pages… find
  31. Taha Koleilat, Hojat Asgariandehkordi, Hassan Rivaz, and Yiming Xiao. 2025. Biomedcoop: Learning to prompt for biomedical vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14766–14776. find
  32. Wang Lu, HU Xixu, Jindong Wang, and Xing Xie. Fedclip: Fast generalization and personalization for clip in federated learning. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models. find
  33. Sachit Menon and Carl Vondrick. 2022. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183. — [arXiv:2210.07183](https://arxiv.org/abs/2210.07183) find
  34. Munish Monga, Sachin Kumar Giroh, Ankit Jha, Mainak Singha, Biplab Banerjee, and Jocelyn Chanussot. 2024. Cosmo: Clip talks on open-set multi-target domain adaptation. arXiv preprint arXiv:2409.00397. — [arXiv:2409.00397](https://arxiv.org/abs/2409.00397) find
  35. Msoud Nickparvar and 1 others. 2021. Brain tumor mri dataset. find
  36. Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, and 1 others. 2017. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease dete… find
  37. Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sa-hasrabuddhe, and Fabrice Meriaudeau. 2018. Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data, 3(3):25. find
  38. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8…
  39. Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. 2022. Test-time prompt tuning for zero-shot generalization in vision-language models. arXiv preprint arXiv:2209.07511. find
  40. Julio Silva-Rodriguez, Hadi Chakor, Riadh Kobbi, Jose Dolz, and Ismail Ben Ayed. 2025. A foundation language-image model of the retina (flair): Encoding expert knowledge in text supervision. Medical Image Analysis, 99:103357. find
  41. Mainak Singha, Sarthak Mehrotra, Paolo Casari, Subhasis Chaudhuri, Elisa Ricci, and Biplab Banerjee. 2026. Clipoint3d: Language-grounded few-shot unsupervised 3d point cloud domain adaptation. arXiv preprint arXiv:2602.20409. — [arXiv:2602.20409](https://arxiv.org/abs/2602.20409) find
  42. Mainak Singha, Subhankar Roy, Sarthak Mehrotra, Ankit Jha, Moloud Abdar, Biplab Banerjee, and Elisa Ricci. 2025. Fedmvp: Federated multimodal visual prompt tuning for vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17869–17878. find
  43. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
  44. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.
  45. Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pages 493–510. Springer. find
  46. Anas M Tahir, Muhammad EH Chowdhury, Amith Khandakar, Tawsifur Rahman, Yazan Qiblawey, Uzair Khurshid, Serkan Kiranyaz, Nabil Ibtehaz, M Sohel Rahman, Somaya Al-Maadeed, and 1 others. 2021. Covid-19 infection localization and severity grading from chest x-ray images. Computers in biology and medicin… find
  47. Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, and 1 others. 2025. A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI, 2(1):AIoa2400640. find
  48. Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. 2018. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):180161. find
  49. Minfan Zhao, Ziqi Zhu, Jun Shi, Zhaohui Wang, Junshi Chen, Hong An, and Bing Yan. 2025. Prompt-seg: Learning to segment medical image via visual prompts. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  50. Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. 2022. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2022, page… find
  51. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308.02463. find
  52. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  53. Junhao Yang, Rui Shi, Dapeng Wei, Zai Wang, Ling Wang, and Jiayu Zhou. 2023. Medmnist v2: A large-scale lightweight benchmark for biomedical image classification. IEEE Transactions on Medical Imaging. find
  54. Hong-Yu Zhou, Cheng Li, Zhihong Jiang, Felix X Yu, Xiang Bai, Sanjiv Kumar, and Thomas S Huang. 2022a. Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports. In International Conference on Learning Representations (ICLR). find
  55. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825. find
  56. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022c. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348. find
  57. Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. 2022. Prompt-aligned gradient for prompt tuning. arXiv preprint arXiv:2205.14865. find
  58. Hantao Yao, Rui Zhang, and Changsheng Xu. 2023. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6757–6767. find
  59. Hantao Yao, Rui Zhang, and Changsheng Xu. 2024. Tcp: Textual-based class-aware prompt tuning for visual-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23438–23448. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singha-2026-biovlm-routing-prompts-not-parameters/