📄 ArabicMedicalBERT-QA-82 at AbjadMed: Fighting Class Imbalance in Arabic Medical Text Classification

Summary

This system paper, written for the AbjadMed shared task, addresses classification of Arabic medical question-answer pairs into 82 categories. The label distribution is very skewed, and several categories overlap in meaning. The author compares three Arabic BERT-style backbones under the same fine-tuning recipe. The recipe uses class-weighted cross-entropy, separate learning rates for the head and the backbone, and 10 epochs. The strongest starting point is an AraBERTv2 checkpoint already trained on a related 89-class Arabic medical classification task, with its labels partially remapped to the shared-task categories. Even without further training, this checkpoint beats the general-purpose models. It improves most when the backbone is updated only gently, which suggests that preserving its prior knowledge matters more than heavy adaptation. The submitted 10-fold majority-vote ensemble reaches a macro F1 of about 0.51 on the private test split, well above the official baseline. Cheap LLMs prompted in zero-shot and few-shot modes perform far worse and often invent category names outside the allowed list. The work shows that transferring supervision from a related task is a practical way to handle low-resource, imbalanced medical domains.

Key points

• Data: 27,951 labelled training pairs and 18,634 test pairs, split evenly between the public and private leaderboards. Class sizes range from about 600 examples down to 7 (In vitro fertilization).
• Backbones compared: CAMeL-Lab/bert-base-arabic-camelbert-da, aubmindlab/bert-base-arabertv2 and AymanElbery/arabic-medical-classifier-arabertv2. The last one scores 0.4848 private macro F1 with remapping alone, against 0.2896 for the baseline.
• Best configuration: a backbone learning rate of 1e-6 with a head learning rate of 5e-4.
  • The 10-fold ensemble scores 0.5071 public and 0.5139 private.
  • A single model trained on all the data does slightly better, at 0.5165 public and 0.5153 private.
  • Over 88% of test instances receive the same prediction from at least 9 of the 10 folds.
• LLM comparison: gpt-4.1-nano and deepseek-v3.2 were tested with constrained re-prompting of up to 100 attempts, falling back to "General Medicine" if no valid label appeared. The best result was deepseek-v3.2 few-shot at 0.3157 private.
• Released model: the final model is available on HuggingFace as gleb-shnshn/arabic-medical-bert-qa-82.