LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2306.00890

Summary

General-purpose multimodal assistants perform poorly on biomedical images. They often decline to answer or produce hallucinated replies. Earlier biomedical VQA systems also mostly pick answers from a fixed set, so they cannot hold free-form conversations. The authors present LLaVA-Med, which adapts the general-domain LLaVA model to biomedicine at low cost. They build training data from PMC-15M, a large collection of figure–caption pairs from PubMed Central. Text-only GPT-4 turns captions, plus sentences in the source papers that reference each figure, into multi-turn dialogues about the image, with no human labeling required. Training follows a two-stage curriculum. First, the model learns biomedical concepts by describing images from their raw captions. Second, it is tuned on the synthetic conversations to handle open-ended instructions. The whole process takes under 15 hours on eight A100 GPUs. After further fine-tuning on VQA-RAD, SLAKE and PathVQA, LLaVA-Med beats earlier supervised state-of-the-art systems on some metrics. The work shows a cheap, annotation-free route to domain-specialized visual chat assistants.

Key points

• Stage 1 (concept alignment) uses 600K image–caption pairs sampled from PMC-15M. Stage 2 uses 60K pairs covering five modalities: CXR, CT, MRI, histopathology and gross pathology.
• The architecture links a CLIP vision encoder to a 7B LLaMA/Vicuna language model through a linear projection. Variants with a 13B LM and a BiomedCLIP encoder were also tested.
• Instruction data enriched with in-paper figure mentions (60K-IM) gave the best results. For example, VQA-RAD closed-question accuracy reached 84.19, compared with 83.50 for M2I2. The BiomedCLIP variant scored 87.11 on SLAKE open questions, compared with 82.05 for BiomedCLIP.
• Longer downstream fine-tuning (around 9 epochs) and the larger 13B model both improved performance.
• The model could answer Chinese questions on SLAKE without any Chinese training data. Remaining weaknesses include hallucination and shallow reasoning.