Elamur Research⌕
☰
home / papers / 2025 / An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction

An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction

Summary

The paper addresses disease-type classification of noisy, user-written Arabic medical posts from an online health platform. The dataset was annotated under a medical advisor's supervision and spans seven specialty classes, with Internal Medicine the largest. LLAMA3 produces three alternative views of each post:

  • a cleaned "refined" rewrite,
  • a short summary,
  • a list of extracted medical entities (NER: symptoms, conditions, drugs).

Each view is attached to the original post rather than replacing it, giving three paired inputs. Three Arabic BERT models (CAMeLBERT, AraBERT, AsafayaBERT) are fine-tuned on these pairs with identical hyperparameters. A majority-voting ensemble then combines their predictions. Individual configurations peak at roughly 75% accuracy, while the ensemble reaches 80.56%.

The authors present this as the first combination of LLM preprocessing, Arabic transformer fine-tuning and ensembling for Arabic social-telehealth data. The work matters because it shows that several cheap, differently preprocessed views can be pooled to outperform any single noisy-text classifier.

Key points

  • Backbones are CAMeL-Lab/bert-base-arabic-camelbert-mix, aubmindlab/bert-base-arabert and asafaya/bert-base-arabic. Each gets a dropout layer and a linear head over the 7 disease classes.
  • All models share one configuration:
    • dropout 0.05, learning rate 1e-4, weight decay 0.01
    • batch size 4, 25 epochs
    • cross-entropy loss
  • LLAMA3 performs all three preprocessing steps (refinement, summarization, NER). The inputs are original+refined, original+summarized and original+NER.
  • Single-model accuracies in Table II range from about 64.9% to 75.6%. The summarized variants are usually weakest (64.89% CAMeLBERT, 65.94% AraBERT), and majority voting reaches 80.56%.
  • This work extends the authors' earlier study (arXiv:2502.00063), which fine-tuned the same three models on the same data without ensembling.

Interesting ideas

  • I1. Instead of replacing a messy input with an LLM's cleaned version, append the rewrite to the original so the model keeps both, which guards against the rewriter dropping or distorting facts.
  • I2. Ensemble diversity here comes partly from different input views of the same text, not only from different models, and voting over these views beats every single member.
  • I3. Summaries were usually the weakest view, a reminder that compression can strip exactly the details that discriminate between classes.
  • I4. Training every backbone with one shared hyperparameter setup makes differences between models and input views attributable to the representation, not to tuning luck.
  • I5. A compact entity-only view (symptoms, conditions, drugs) still carried enough signal to classify fairly well, suggesting extracted entities form a useful lean representation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When an LLM teacher normalizes noisy sentences before silver parsing, keep the raw-sentence parse as a separate witness so normalization errors can be detected rather than silently trusted.
  • I2 (learn): Witness parsers could be run on several normalized variants of a sentence (raw, spelling-fixed, simplified), and only arcs that agree across variants promoted into silver data.
  • I3 (back): Test the text → graph → text compressor by checking that a downstream classifier or QA reader gives the same answer on the reconstructed text as on the original.
  • I4 (learn): Fix training settings when comparing tagger or parser variants (perceptron vs gparse, different UD conversions), so evaluation deltas reflect the design change.
  • I5 (world): Classify or retrieve texts from the extracted entity-and-state inventory of the text's world, as an explainable alternative to whole-text features.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Language models & pretraining Semitic languages

 Institutions

🎓Qatar UniversityDoha · Qatar7 works · 78 citations🎓October University of Modern Sciences and ArtsCairo · Egypt7 works · 25 citations

Cite

Cite as Ali Hamdi, Malak Mohamed, Rokaia Emad and Khaled Shaban. 2025. An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction. arXiv:2509.02446.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.02446 · arXiv 2509.02446

@misc{hamdi2025ensemble,
  title = {{An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction}},
  author = {Ali Hamdi, Malak Mohamed, Rokaia Emad and Khaled Shaban},
  year = {2025},
  doi = {10.48550/arxiv.2509.02446},
  eprint = {2509.02446},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.02446}
}

References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Chen and Y. Wang, “Social media use for health purposes: Systematic review,” Journal of Medical Internet Research, vol. 23, no. 5, 2020. find
  2. R. B. Correia, I. B. Wood, J. Bollen, and L. M. Rocha, “Mining social media data for biomedical signals and health-related behavior,” Annual Review of Biomedical Data Science, vol. 3, no. 1, pp. 433–458, 2020. find
  3. A. Magge et al., “Overview of the Sixth Social Media Mining for Health Applications (SMM4H) Shared Tasks at NAACL 2021,” in Proc. of the Sixth Social Media Mining for Health Applications (SMM4H) Shared Task at NAACL, Jan. 2021. find
  4. Y. Guo, A. Ovadje, M. A. Al-Garadi, and A. Sarker, “Evaluating large language models for health-related text classification tasks with public social media data,” Journal of the American Medical Informatics Association, 2024. find
  5. K. Kowsari, K. J. Meimandi, M. Heidarysafa, S. Mendu, L. Barnes, and D. Brown, “Text classification algorithms: A survey,” Information, vol. 10, no. 4, p. 150, Apr. 2019. find
  6. S. S. Alahmari, L. O. Hall, P. R. Mouton, and D. B. Goldgof, “Repeatability of fine-tuning large language models illustrated using QLORA,” IEEE Access, 2024. find
  7. N. Niraula, S. Ayhan, B. Chidambaram, and D. Whyatt, “Multi-label classification with generative large language models,” in Proc. of the 43rd Digital Avionics Systems Conference (DASC), 2024. find
  8. R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. A. Almazroi, “A fine-tuned BERT-based transfer learning approach for text classification,” Journal of Healthcare Engineering, 2022. find
  9. V. Radivchev and A. Nikolov, “Nikolov-Radivchev at SemEval-2019 Task 6: Offensive tweet classification with BERT and ensembles,” in Proc. of the 13th Int. Workshop on Semantic Evaluation (SemEval-2019), pp. 691–695, 2019. find
  10. S. Casola and A. Lavelli, “FBK @ SMM4H 2020: RoBERTa for detecting medications on Twitter,” in Proc. of the Fifth Social Media Mining for Health Applications Workshop & Shared Task, pp. 101–103, 2020. find
  11. B. Büyüköz, A. Hürriyetoğlu, and A. Özgür, “Analyzing ELMo and DistilBERT on socio-political news classification,” in Proc. of the Workshop on Automatic Extraction of Socio-political Events from News, pp. 9–18, USA, May 2020. find
  12. B. Li and F. Rudzicz, “TorontoCL at CMCL 2021 shared task: RoBERTa with multi-stage fine-tuning for eye-tracking prediction,” in Proc. of the Workshop on Cognitive Modeling and Computational Linguistics (CMCL), pp. 4–9, 2021. find
  13. Y. Zhao and X. Tao, “ZYJ123@DravidianLangTech-EACL2021: Offensive language identification based on XLM-RoBERTa with DPCNN,” in Proc. of the First Workshop on Speech and Language Technologies for Dravidian Languages, pp. 216–221, Parkville, Victoria: EACL, 2021. find
  14. X. Ou and H. Li, “YNU @ Dravidian-CodeMix-FIRE2020: XLM-RoBERTa for multi-language sentiment analysis,” in Proc. of the Forum for Information Retrieval Evaluation (FIRE), pp. 4–9, 2020. find
  15. O. Hamad, K. Shaban, and A. Hamdi, “ASEM: Enhancing empathy in chatbot through attention-based sentiment and emotion modeling,” in Proc. of the Int. Conf. on Computational Linguistics (COLING), pp. 1588–1601, 2024. find
  16. K. A. Das, A. Baruah, F. A. Barbhuiya, and K. Dey, “Ensemble of ELECTRA for profiling fake news spreaders,” in Proc. of the Forum for Information Retrieval Evaluation (FIRE), pp. 22–25, 2020. find
  17. C. Popa and T. Rebedea, “BART-TL: Weakly-supervised topic label generation,” in Proc. of the 16th Conf. of the European Chapter of the ACL (EACL), pp. 1418–1425, 2021. find
  18. A. Mustar, S. Lamprier, and B. Piwowarski, “Using BERT and BART for query suggestion,” in Proc. of the CEUR Workshop on NLP, vol. 2621, 2020. find
  19. M. de Bruyn, E. Lotfi, J. Buhmann, and W. Daelemans, “BART for knowledge grounded conversations,” in Proc. of the CEUR Workshop on NLP, vol. 2666, 2020. find
  20. O. H. Abdellatif, A. N. Hassan, and A. Hamdi, “LMRPA: Large language model-driven efficient robotic process automation for OCR,” in The Int. Conf. of Advanced Computing and Informatics, pp. 35–44, Springer, 2024. find
  21. A. Hamdi, A. A. Mazrou, and M. Shaltout, “LLM-SEM: A sentiment-based student engagement metric using LLMs for e-learning platforms,” in The Int. Conf. of Advanced Computing and Informatics, pp. 145–154, Springer, 2024. find
  22. M. Mohamed, R. Emad, and A. Hamdi, “A multi-layered large language model framework for disease prediction,” arXiv preprint arXiv:2502.00063, 2025. ↗
  23. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of the 2019 Conf. of the NAACL, pp. 4171–4186, 2019. our card
  24. I. Ait Talghalit, H. Alami, and S. Ouatik El Alaoui, “Contextual semantic embeddings based on transformer models for Arabic biomedical questions classification,” HighTech and Innovation Journal, vol. 5, no. 4, pp. 1024, 2024. find
  25. O. Hamad, A. Hamdi, S. Hamdi, and K. Shaban, “StEduCov: An explored and benchmarked dataset on stance detection in tweets towards online education during COVID-19 pandemic,” Big Data and Cognitive Computing, vol. 6, no. 3, pp. 88, 2022. find
  26. A. Al-Ali, A. Hamdi, M. Elshrif, et al., “CLASEG: Advanced multi-classification and segmentation for differential diagnosis of oral lesions using deep learning,” Scientific Reports, vol. 15, no. 1, pp. 23016, 2025. find
  27. A. Allam, S. Ahmed, A. Hamdi, and A. Mohammed, “Arabic large language models for medical text generation,” in 2025 4th Int. Conf. on Computer Technologies (ICCTech), pp. 1–6, 2025. our card
  28. J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in Proc. of the 2014 Conf. on Empirical Methods in NLP (EMNLP), pp. 1532–1543, 2014. our card
  29. W. Antoun, F. Baly, and H. Hajj, “AraBERT: Transformer-based model for Arabic language understanding,” in Proc. of the LREC 2020 Workshop on Resources for African Indigenous Languages, pp. 9–15, 2020. our card
  30. J. Wei and K. Zou, “EDA: Easy data augmentation techniques for boosting performance on text classification tasks,” Proc. of the 2019 Conf. on Empirical Methods in NLP (EMNLP), pp. 6382–6388, 2019. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hamdi-2025-ensemble-classification-approach-multi-layered/