Elamur Research⌕
☰
home / papers / 2026 / GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification

GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification

Paper · 2026 · Ahmed Khamis

Summary

The paper describes a system for the AbjadMed shared task. The task is to sort Arabic medical questions into 82 categories. The class distribution is heavily skewed, and some gold labels are inconsistent. The author fine-tunes AraBERTv2 with a custom classification head that joins mean-pooled and attention-pooled token vectors. Multi-sample dropout and label smoothing are added to cope with rare classes and noisy annotation. This setup is compared with other Arabic and multilingual encoders and with two decoder-based alternatives:

  • a frozen Qwen 3 3B used only as a feature extractor;
  • a training-free pipeline in which Llama 3.3 70B chooses the final label from AraBERT's top candidates.

The fine-tuned AraBERTv2 achieves the best Macro-F1 on the official test set. Both decoder approaches score lower, and the frozen decoder features perform far worse. The study shows that for fine-grained labels defined by one particular annotation scheme, a modest encoder trained discriminatively can beat much larger general-purpose LLMs. The author explains this by the bidirectional context of encoders and their alignment with annotator conventions.

Key points

  • Data: 27,951 training queries and 18,634 unlabeled test queries across 82 classes. Class sizes range from 600 examples (e.g., Hematological diseases, Addiction) down to 7 (e.g., Biochemistry, IVF). Manual inspection revealed label noise, such as dermatology cases tagged as General Medicine.

  • Main model: bert-base-arabertv02 with these additions:

    • concatenated mean and attention pooling, which doubles the hidden size;
    • a dense layer with LayerNorm and GELU;
    • five dropout branches (rates 0.1–0.3) whose logits are averaged.

  • Training: label smoothing 0.1, layer-wise learning-rate decay 0.95, AdamW with cosine schedule at 2e-5, and per-batch dynamic padding.

  • Macro-F1 results:

    SystemMacro-F1
    AraBERTv20.3934
    multilingual-E5-large0.3804
    CamelBERT0.3603
    AraBERTv2 + Llama 3.3 70B re-ranking of top-150.3035
    Qwen 3 3B features0.1278

    EuroBERT and mmBERT are reported as clearly weaker.

  • Limitation: there are no ablations, so the separate contributions of pooling, dropout and smoothing remain unknown.

Interesting ideas

  • I1. Joining an averaged representation with a learned attention-weighted one keeps both the overall topic and the few decisive words, which helps when the key evidence can appear anywhere in the input.
  • I2. Running several dropout masks in parallel and averaging their logits works as an almost free internal ensemble that steadies predictions for rare classes.
  • I3. Label smoothing is used as a deliberate response to observed annotation noise, softening targets where category boundaries are fuzzy.
  • I4. A strong LLM asked to re-rank candidates picked labels that were sensible but outside the task's own conventions, lowering the score ("schema mismatch").

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A speech-act or irony classifier over a sentence could pool tokens this way, and the attention weights would also show which words drove the decision, supporting explainability.
  • I2 (learn): Heads trained on small or long-tailed silver sets, such as rare dependency relations in gparse, could use this to reduce variance without training multiple models.
  • I3 (learn): Silver data agreed by witness parsers still contains errors, so smoothing targets, possibly more strongly where witnesses disagreed, could keep the student from memorising them.
  • I4 (learn): LLM teacher tiers should be checked against the exact UD/VerbNet label conventions before their outputs become training data, because plausible-but-off-schema answers are a specific failure mode to filter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmed Khamis. 2026. GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.abjadnlp-1.13

@inproceedings{khamis2026gatech,
  title = {{GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification}},
  author = {Ahmed Khamis},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.13},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.13}
}

References

20 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Arabert: Transformer-based model for arabic language understanding. Preprint, arXiv:2003.00104. of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics. ↗
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. Preprint, arXiv:1607.06450. ↗
  3. Hiroshi Inoue. 2020. Multi-sample dropout for accelerated training and better generalization. Preprint, arXiv:1905.09788. ↗
  4. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. 2025. Eurobert: Scaling multilingual encoders for european languages. Preprint, arXiv:2503.05500. ↗
  5. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. ↗
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165. ↗
  7. Rafael Müller, Simon Kornblith, and Geoffrey Hinton. 2020. When does label smoothing help? Preprint, arXiv:1906.02629. ↗
  8. Fang Chen, Gourav Datta, Souvik Kundu, and Peter Beerel. 2022. Self-attentive pooling for efficient deep learning. Preprint, arXiv:2209.07659. ↗
  9. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971. ↗
  10. Qian Chen, Zhen-Hua Ling, and Xiaodan Zhu. 2018. Enhancing sentence embedding with generalized pooling. Preprint, arXiv:1806.09828. ↗
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  12. Pranav Gupta, Niranjan Kumar M, Balaji Nagarajan, Imed Zitouni, and Mo El-Haj. 2026. Abjadmed: Arabic medical text classification at abjadnlp 2026. In Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script (AbjadNLP 2026), co-located with the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), Rabat, Morocco.
  13. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. Preprint, arXiv:2509.06888. ↗
  14. Michael Toker, Ido Galil, Hadas Orgad, Rinon Gal, Yoad Tewel, Gal Chechik, and Yonatan Belinkov. 2025. Padding tone: A mechanistic analysis of padding tokens in t2i models. Preprint, arXiv:2501.06751. ↗
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762. ↗
  16. Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. 2024. [cls] token tells everything needed for training-free efficient mllms. Preprint, arXiv:2412.05819. ↗
  17. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  18. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. Preprint, arXiv:2006.03654. ↗
  19. Dan Hendrycks and Kevin Gimpel. 2023. Gaussian error linear units (gelus). Preprint, arXiv:1606.08415. ↗
  20. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khamis-2026-gatech-abjadmed-bidirectional-encoders-vs/