📄 BiMediX2 : Bio-Medical EXpert LMM for Diverse Medical Modalities

Summary

BiMediX2 is a medical assistant model that works with both text and images and converses in English and Arabic. It targets two gaps in earlier medical vision-language systems. Adding image support often weakened their text-only medical skills, and most covered only English or a narrow set of imaging types. The authors built BiMed-V, a set of roughly 1.6M bilingual instructions that mixes text-only and image-text samples. The Arabic part was produced by translating English data with GPT-4o, and bilingual clinicians checked a random sample. They then trained a LLaVA-style model on Llama 3.1 in two stages. First a projector learns to map medical images into the language model's input space, and then LoRA instruction tuning is applied. They also released BiMed-MBench, an expert-verified bilingual benchmark for medical multimodal models. The model leads open-source competitors on that benchmark, by about 9% in English and over 20% in Arabic. It also does well on medical VQA, report generation and summarization, and text QA. This shows that one checkpoint can cover many tasks and modalities without losing its text competence.

Key points

• Data:
  • BiMed-V contains about 1.6M instructions. Sources include PMC-OA, Rad-VQA, Path-VQA and SLAKE.
  • It adds 163k VQA items rebuilt from the LLaVA-Med 60K-IM set.
  • Over 10k LLaVA-Med captions were turned into dialogues with Llama 3.1 70B.
• Training: Stage 1 trains only the projector on 467K medical image-caption pairs. Stage 2 applies LoRA tuning on BiMed-V. Released sizes are 4B, 8B and 70B.
• Benchmark: BiMed-MBench has 386 expert-verified queries in English and Arabic. It covers CT, MRI, CXR, histology and gross pathology images.
• Text benchmarks: The 70B model averages 84.6 on the clinical LLM benchmarks (MMLU medical subsets, MedMCQA, MedQA, USMLE, PubMedQA). Compared with GPT-4, it scores more than 8% higher on USMLE and about 9% higher on UPHILL factual accuracy.
• Translation baseline: An English-only 8B model with Google, Bing or Alibaba translation scores 44.9–45.3 on Arabic BiMed-MBench. The natively bilingual model scores 50.47.

Interesting ideas

• I1. The authors tested their English model behind commercial translation systems, which shows whether native bilingual training adds anything beyond a cheap cascade.
• I2. In the staged ablation, text-only scores stayed exactly the same after the projector-only stage, which confirms that this stage left the language model untouched.
• I3. Training data deliberately mixed text-only and image-text instructions so that the original text skills would not erode under the new modality.
• I4. The authors used a strong LLM to turn flat image captions into multi-turn dialogues, which converts cheap static annotations into richer interactive training data.
• I5. The Arabic data was machine-translated in bulk, and experts checked only a random subset, which gives quality assurance at a fraction of full review cost.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Every new learned component could be benchmarked against a trivial cascade or pipeline baseline as a negative control, which proves the added machinery earns its cost.
• I2 (tree): When gparse gains a new input feature or adapter, metrics on old treebanks could be checked for bit-identical results while the old weights stay frozen, which catches leakage between stages.
• I3 (tree): When adapting the parser to a new UD dialect or domain, a share of the original treebank data could be kept in each batch to prevent forgetting.
• I4 (learn): An LLM teacher tier could expand fable or school-text sentences into question-answer exchanges, giving the answer readers silver training material.
• I5 (learn): Silver data agreed by the witness parsers could be audited the same way, by hand-checking a random sample to estimate error rates before training.