📄 MedLingua at MedArabiQ2025: Zero- and Few-Shot Prompting of Large Language Models for Arabic Medical QA

Summary

The MedLingua team took part in the MedArabiQ2025 shared task, Track 2, Sub-Task 1: answering Arabic medical multiple-choice questions by returning a single answer letter. The rules banned fine-tuning, so the team used in-context learning only. They wrote structured prompts that combine a system instruction, optional worked examples and the question, and they asked for step-by-step reasoning inside a <thinking> block. A multi-stage parser then pulled the final letter out of each reply. They tested 11 models. Some were large general-purpose LLMs, including Gemini 2.5 Pro, Qwen, Llama 3 and Mixtral. Others were medical specialists, including BiMediX2, MedGemma, BioMistral, OpenBioLLM and Palmyra-Med. The generalists won clearly. Gemini 2.5 Pro scored 78% on development data and 74% on the blind test, which placed third on the leaderboard. Most medical models were held back by weak Arabic or by failing to follow the output format. The authors conclude that solid command of the target language must come before domain expertise can help. This matters for any specialised task in a lower-resource language.

Key points

• The validation data had three 100-question sets: standard MCQ, MCQ with deliberately misleading phrasing ("w/ Bias"), and fill-in-the-blank with choices. The blind test was 100 mixed questions covering 12 medical specialties.
• Blind-test accuracy by model:
  • Gemini 2.5 Pro: 74%
  • Qwen: 67%
  • MedGemma: 53%
  • Colosseum: 51%
  • Palmyra-Med: 49%
  • Llama3 70B: 45%
  • BiMediX2: 37%
  • Mixtral and OpenBioLLM: 21% each
  • BioMistral: 19%
  • DeepSeek: 17%
• Physiology was the hardest category. Gemini made 6 errors there, compared with 11–14 for the other top-5 models. Gemini made only 1 error in Neurosurgery and tied for fewest in Oncology and Biochemistry.
• BioMistral produced repetitive gibberish when given Arabic few-shot examples. In zero-shot mode it reasoned correctly in English and gave a properly formatted final answer.
• A physician co-author re-annotated the test questions by specialty. She found that some items labelled "Physiology" were really about research methodology. English-centric models such as Palmyra-Med needed translation, which risks losing information.

Interesting ideas

• I1. The prompting strategy was chosen per model rather than globally, because few-shot examples helped some models and broke others.
• I2. Answers were extracted by a hierarchical, fallback-based parser, and formatting failures were tracked as an error class separate from wrong reasoning.
• I3. One validation split rephrased questions in a misleading way to probe robustness, beyond plain accuracy.
• I4. An expert re-labelled the error analysis by subject area, which exposed both domain weak spots and mislabelled categories in the original data.
• I5. The paper argues that strong base-language competence is a precondition for domain reasoning, since specialised but language-weak models lost to generalists.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Each LLM teacher tier could be given its own prompt format, chosen by a small ablation on held-out items, instead of assuming one template suits every teacher.
• I2 (learn): When harvesting silver labels from teachers, malformed outputs should be logged apart from substantively wrong ones, so that format noise is not mistaken for disagreement among witnesses.
• I3 (answer): School-test and fable-QA readers could be checked on adversarially worded variants of the same question, acting as a negative control that a truly understanding reader should pass.
• I4 (answer): Breaking reader errors down by course topic, and auditing the topic labels themselves, would show whether failures come from missing world knowledge or from bad benchmark metadata.
• I5 (tree): Parsing quality on in-domain text should come before adding domain ontologies, because frames and the world model built on a faulty UD tree cannot be rescued by extra knowledge.