📄 baellouf at AbjadMed: Efficient Fine-tuning with All-Linear LoRA for Arabic Medical QA Classification

Summary

The paper describes an entry to the AbjadMed shared task at AbjadNLP 2026. The task is to assign Arabic medical question–answer pairs to one of 82 fine-grained, heavily imbalanced categories, several of which overlap closely. The author fine-tunes Qwen3-8B with QLoRA. Following the "LoRA Without Regret" recommendations, adapters are attached to every linear layer, including the MLP blocks, rather than only to the attention projections. Training uses a loss that mixes Dice and cross-entropy in equal parts, so that the objective tracks the Macro F1 metric more closely. A controlled comparison shows that all-linear adapters beat attention-only ones by 4 points. The Dice+CE mix also outperforms Focal Loss. The author also tried adding machine-translated external medical QA data, and it slightly lowered the score. The best run reaches 0.4441 Macro F1 on the test set. The work shows that a modest single-GPU setup with sensible adapter placement can match a much larger domain-specialised model on a low-resource, imbalanced classification problem.

Key points

• Data: 27,951 training samples across 82 categories. Class sizes range from 7 to over 600 (roughly 85:1). Some areas are split into many neighbouring labels, for example 5 dental classes and 3 mental-health classes.
• Setup:
  • Qwen3-8B with QLoRA, rank 256, α = 16, no dropout.
  • Learning rate 2×10⁻⁴, 3 epochs, effective batch size 32, maximum length 1024 tokens.
  • 10% of the data held out for validation, on one H100.
  • The adapter is released on Hugging Face.
• Adapter placement: with all other settings fixed, all-linear LoRA scores 0.444 Macro F1 and attention-only (q, k, v, o) scores 0.404.
• Loss function: on Qwen3-4B, Dice+CE scores 0.419 and Focal Loss (γ=2) scores 0.403.
• Model comparison: MedGemma-27B reaches only 0.423, barely ahead of Qwen3-4B.
• Augmentation: adding machine-translated iCliniq data gave 0.4398, slightly worse than training on the original data alone.

Interesting ideas

• I1. When adapting a large pretrained network cheaply, put low-rank adapters on the feed-forward/MLP layers as well as attention, and verify the choice with a single-variable ablation; the cited work indicates the gain is not just from adding parameters.
• I2. Blend a metric-shaped loss (Dice, a soft F1 surrogate) with cross-entropy for stability, so that rare classes count as much in training as they do in macro-averaged evaluation.
• I3. Before modelling, map out clusters of near-duplicate labels in the taxonomy, since these overlaps, not raw class count, drive much of the difficulty.
• I4. Extra data from machine translation is not automatically useful, so keep an untouched original-data baseline and judge each augmented run against it.
• I5. A smaller general multilingual model nearly matched a model three times larger that was trained specifically on medical text, which suggests trying cheap baselines before scaling up.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): when fine-tuning the gparse transformer encoder on new treebanks or UD dialects, compare attention-only adapters against all-linear ones before investing in full fine-tuning.
• I2 (tokens): the tagger and morphology heads could use a Dice+CE mix so that rare POS tags and morphological features are not drowned out by frequent ones.
• I3 (world): sense and frame inventories from WordNet/FrameNet have similar near-synonym clusters, and grouping them could guide both the error analysis and where finer distinctions really matter.
• I4 (learn): silver data from witness parsers or LLM teacher tiers should always be gated by a held-out comparison against the gold-only model, which serves as a built-in negative control.
• I5 (answer): before using a larger LLM teacher for QA readers, check whether a smaller tuned model already reaches the same score on school tests or fable QA.