Elamur Research⌕
☰
home / papers / 2026 / Sujith Kanakkassery at AbjadMed: Imbalance-Aware Transformer Fine-tuning for Arabic Medical Text Classification

Sujith Kanakkassery at AbjadMed: Imbalance-Aware Transformer Fine-tuning for Arabic Medical Text Classification

Paper · 2026 · Sujith Kanakkassery

Summary

This system paper describes an entry to the AbjadMed 2026 shared task. The task requires assigning one of 82 fine-grained medical categories to Arabic question–answer texts. The label distribution has a long tail, so systems are ranked by macro-averaged F1, which rewards good performance on rare categories. The author fine-tunes MARBERT with a small feedforward head on the [CLS] representation. Several measures target the imbalance:

  • rule-based removal of greetings and boilerplate,
  • a longer input window of 256 tokens,
  • cross-entropy weighted by inverse class frequency,
  • mild label smoothing,
  • a shorter training schedule.

An incremental ablation on a held-out validation split raises macro-F1 from 0.258 to 0.340. Error analysis traces the remaining failures to two causes: categories with almost no training examples, and pairs of categories whose meanings overlap. The work shows that cheap cost-sensitive tweaks help on a skewed label space. It also shows that sparsity in the labels themselves caps what fine-tuning alone can achieve.

Key points

  • Data: 27,951 training instances, split 90/10 into 25,156 for training and 2,795 for validation, stratified. The hidden Kaggle test set has 18,634 items.
  • Backbone: MARBERT outperformed CAMeLBERT. The author credits MARBERT's larger and noisier pretraining corpus. The final configuration is called CustomMedicalBert.
  • Training setup:
    • learning rate 2×10⁻⁵ with a cosine schedule and 0.1 warmup,
    • label smoothing 0.05,
    • 3 epochs at batch size 16,
    • about 22 minutes on one P100 GPU.
  • Ablation (validation macro-F1): baseline 0.258, data cleaning 0.265, length 256 tokens 0.293, class weights 0.320, fewer epochs 0.340.
  • Final validation scores: macro-F1 0.3408 versus weighted F1 0.4716. Twenty-one classes scored zero F1, and all of them had support of 15 or fewer instances.

Interesting ideas

  • I1. Weighting the loss by N/(C·n_c) is a cheap way to stop frequent labels from dominating training when the label inventory is long-tailed.
  • I2. Errors can be ranked by counting off-diagonal confusion-matrix pairs, which shows whether mistakes cluster between semantically neighbouring labels or are scattered at random.
  • I3. Listing each zero-score class next to its support count separates "too few examples to learn" from "the model systematically fails".
  • I4. Stripping formulaic conversational boilerplate by rule, while leaving the remaining surface forms unnormalized, removes noise without erasing informative wording.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Inverse-frequency weights could be applied to rare dependency relations in the perceptron parser or gparse, so that infrequent deprels are not swamped by nsubj or det.
  • I2 (learn): Ranking the most confused deprel or semantic-role pairs would point to where self-induced repair rules or label merges would pay off most.
  • I3 (learn): Evaluation reports could show per-label support beside F1, so effort goes to genuine failures rather than labels with a handful of instances.
  • I4 (tokens): A rule-based pre-filter for greetings and sign-offs in user questions could run before tagging, keeping original word forms intact for morphology and lemmas.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Sujith Kanakkassery. 2026. Sujith Kanakkassery at AbjadMed: Imbalance-Aware Transformer Fine-tuning for Arabic Medical Text Classification. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.abjadnlp-1.48

@inproceedings{kanakkassery2026sujith,
  title = {{Sujith Kanakkassery at AbjadMed: Imbalance-Aware Transformer Fine-tuning for Arabic Medical Text Classification}},
  author = {Sujith Kanakkassery},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.48},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.48}
}

References

8 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Nouar AlDahoul and Yasir Zaki. 2025. NYUAD at AraHealthQA shared task: Benchmarking the medical understanding and reasoning of large language models in Arabic healthcare tasks. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 119–125, Suzhou, China. Association for Computational Linguistics.
  3. Hassan Alhuzali, Walid Al-Eisawi, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Renad Al-Monef, Ali Alqahtani, Lama Ayash, Leen Kharouf, Farah E. Shamout, and Nizar Habash. 2025. AraHealthQA 2025: The first shared task on Arabic health question answering. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 107– 118, Suzhou, China. Association for Computational Linguistics. ↗
  4. Pranav Gupta, Niranjan Kumar M, Balaji Nagarajan, Imed Zitouni, and Mo El-Haj. 2026. Abjadmed: Arabic medical text classification at abjadnlp 2026. In Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script (AbjadNLP 2026), co-located with the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), Rabat, Morocco.
  5. Ahmed Ibrahim, Abdullah Hosseini, Hoda Helmy, Wafa Lakhdhar, and Ahmed Serag. 2025. Bridging dialectal gaps in Arabic medical LLMs through model merging. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 338– 346, Suzhou, China. Association for Computational Linguistics. find
  6. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 101, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  7. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, Louisiana, USA. OpenReview.net. find
  8. Mohamed Zaytoon, Ahmed Mahmoud Salem, Ahmed Sakr, and Hossam Elkordi. 2025. AraMinds at Ara-HealthQA 2025: A retrieval-augmented generation system for fine-grained classification and answer generation of Arabic mental health Q&A. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 198–203, Suzhou, China. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kanakkassery-2026-sujith-kanakkassery-abjadmed-imbalance-aware/