📄 Severity-Aware Weighted Loss for Arabic Medical Text Generation

Summary

Fine-tuning language models for medical question answering usually treats every training example the same, even though a bad reply to a dangerous complaint costs far more than a bad reply to a trivial one. The authors address this for Arabic by attaching a risk signal to each training pair and building it into the loss. They take 32,000 patient complaint–professional answer pairs from the MAQA corpus and run a fine-tuned AraBERT classifier over the complaints. They keep the classifier's full probability vector over three severity classes: non-critical, neutral and critical. Each example gets a scalar weight that is a linear mix of those probabilities, and this weight scales its token-level cross-entropy. The model architecture is unchanged. Across ten small Arabic-capable LLMs, plain cross-entropy fine-tuning helps only slightly. The weighted variant gives larger and more uniform gains, especially with a balanced coefficient setting, reaching up to about 12 points over the untuned models. The method is a cheap, drop-in way to make training focus on high-stakes cases, although the severity signal itself comes from a weak classifier.

Key points

• Data: a 32,000-pair subset of MAQA, which covers both Modern Standard Arabic and colloquial Arabic. Each pair is augmented with a predicted severity label and three class probabilities. The resulting split is 46% non-critical, 30% neutral and 24% critical.
• The AraBERT-based severity classifier reaches only about 52% accuracy. The authors therefore use its soft probabilities rather than only the argmax label, to soften the effect of its errors.
• Each instance has a weight w = α·p_nc + β·p_n + γ·p_c, which multiplies the length-averaged token negative log-likelihood. Three settings for (α, β, γ) were tested: Mild (0.75/1/1.25), Strong (0.25/1/1.75) and Balanced (0.5/1/1.5).
• Ten models were evaluated, including AraGPT2-Base/Medium, Bloomz-560M, Al-Atlas-0.5B, GPT2-Small-Arabic, LFM2-350M/700M, Dialect-ar-gpt-2021, Qwen2.5-0.5B and Qwen3-0.6B. The Balanced setting gave the best score for every model. Examples include AraGPT2-Base rising from 54.04% to 66.14% and Qwen2.5-0.5B from 57.83% to 66.86%. The peak was 67.18%, reached by both AraGPT2-Medium and LFM2-700M.
• The Strong setting sometimes produced big jumps, as with AraGPT2-Base and Qwen2.5-0.5B, but it was unstable across architectures. Smaller models benefited the most. The authors list fixed coefficients and noisy, automatically inferred severity as limitations.

Interesting ideas

• I1. Use a weak auxiliary classifier's probability vector as a continuous training weight instead of trusting its hard label, so an unreliable signal still nudges training without dominating it. — For Mova (learn): The fraction of witness parsers that agree on a silver sentence or arc could serve as a soft per-example loss weight, rather than a keep-or-drop threshold.
• I2. Put all the domain knowledge into the loss only, with no architectural change, so the same trick applies to any model with a token-level objective. — For Mova (tree): gparse could up-weight arcs that frames depend on, such as core arguments like nsubj, obj or obl of the main predicate, relative to punct or det arcs, simply by scaling their loss terms.
• I3. Anchor the middle class at weight 1 and shift weight symmetrically between the extremes, then sweep mild, strong and balanced settings; the paper shows that heavy down-weighting of easy cases destabilises training. — For Mova (learn): Any importance-weighted retraining should include such a sweep and report its effect on low-weight data, as a guard against silently degrading common cases.
• I4. Make training cost-sensitive according to how harmful an error is downstream, not only how frequent it is. — For Mova (answer): Training examples for the readers could be weighted more heavily when a mistake flips the final answer, as in quantities in maths word problems, than when it only changes wording.