Elamur Research⌕
☰
home / papers / 2026 / MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection

MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection

Summary

The paper tackles multi-class detection of mental health disorders in Arabic tweets. This task is hard because of dialects, informal spelling, scarce labelled data and skewed class sizes. The authors built a new labelled corpus of 50,670 tweets spanning six classes: five disorders (depression, anxiety, OCD, PTSD and bipolar disorder) plus a "None" class for tweets showing no disorder. They also gathered a large unlabelled set of mental-health tweets.

The work has two phases. In Phase 1, AraBERT, CAMeLBERT and MARBERT each received further domain-adaptive and task-adaptive pretraining (DAPT and TAPT) and were compared under one protocol. In Phase 2, the winning backbone, MentalMARBERT, was tested in four set-ups that cross single-stage versus hierarchical two-stage classification with full fine-tuning versus LoRA.

The adapted MARBERT was the only backbone with significant gains in both accuracy and macro-F1. The best configuration was two-stage with full fine-tuning, at macro-F1 0.8617 and accuracy 0.8778. The work matters because Arabic mental-health NLP has mostly been binary and has relied on unadapted general models.

Key points

  • New dataset: 50,670 tweets in six categories, with inter-annotator agreement of Krippendorff's Alpha 0.733 and average pairwise agreement of 0.797.
  • Adaptation helped unevenly across backbones:
    • AraBERT showed no significant change after DAPT+TAPT.
    • CAMeLBERT improved only in recall.
    • MARBERT improved significantly in accuracy and macro-F1, which the authors attribute to its Twitter-based original pretraining.
  • Full fine-tuning beat LoRA in both architectures. In single-stage, macro-F1 was 0.8564 vs 0.8521; in two-stage, the gap was larger, 0.8617 vs 0.8506. Recall did not differ significantly in either case.
  • The two-stage design beat single-stage only under full fine-tuning (accuracy +0.0075, macro-F1 +0.0053). Under LoRA, no metric differed significantly, with p between 0.08 and 0.31.
  • Results are reported as mean ± SD with per-metric p-values. The authors credit the hierarchy with cutting confusion between None and the disorder classes.

Interesting ideas

  • I1. Splitting the decision into "is anything there at all?" and then "which kind?" reduced confusion with the dominant empty class; this is useful wherever a "none" label swamps the fine-grained ones. — For Mova (frames): Irony and indirect speech-act detection in prag could first gate whether a clause is non-literal at all, then pick the type, so literal clauses stop leaking into rare categories.
  • I2. Continued pretraining paid off only for the backbone whose original data already resembled the target genre, so source–target alignment can matter more than the adaptation step itself. — For Mova (tree): Before continued pretraining of gparse on in-genre unlabeled text (fables, school tests), start from the encoder whose pretraining genre is closest, and verify the gain per backbone instead of assuming it.
  • I3. Each comparison was repeated across runs and judged by per-metric significance tests, which exposed effects that hold for one metric and not another. — For Mova (learn): Parser and reader ablations could report mean ± SD over seeds with a p-value per metric, so that small LAS or answer-accuracy deltas are not mistaken for real wins.
  • I4. With mostly frozen weights (LoRA), the architectural choice stopped making a measurable difference, so cheap adapter training can hide design effects. — For Mova (learn): Architecture ablations for gparse should be run under full training, or at least confirmed there, before concluding that a design change is useless.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.12649 · arXiv 2606.12649

@misc{alharigy2026mentalmarbert,
  title = {{MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection}},
  author = {Alharigy;, Fatimah Almalki; Areej Alhothali; Lulwah and Aladeem, Abdulrahman},
  year = {2026},
  doi = {10.48550/arxiv.2606.12649},
  eprint = {2606.12649},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.12649}
}

References

33 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. World Health Organization. Mental health. https://www.who.int, 2025. Accessed September 2025. ↗
  2. M. Effatpanah et al. Burden of mental disorders in the eastern mediterranean region, 1990–2019: Findings from the global burden of disease study 2019. Balkan Medical Journal, 40(2):112–122, 2023. find
  3. Abdulaziz M. Al-Garni, Ayed A. Shati, Nora A. Almonawar, Ghadi M. Alamri, Lujain A. Alasmre, Thamraa N. Saad, Faisal M. Alshehri, Esraa Abdellatif Hammouda, and Ramy Mohammed Ghazy. Prevalence of depression, anxiety, and stress among students enrolled at king khalid university: a cross-sectional study. BMC Public Health, 25(354):1–11, 2025. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, 2020. find our card
  5. Malek Tounsi, Hanen Ameur, and Salma Jamoussi. Deep Learning for Detecting Mental Disorder from User Content on Social Media. SN Computer Science, 2025. find
  6. Tianlin Zhang, Annika M. Schoene, Shaoxiong Ji, and Sophia Ananiadou. Natural language processing applied to mental illness detection: a narrative review. npj Digital Medicine, 5(1):1–14, 2022. find
  7. Abhishek Pandey and Sanjay Kumar. Mental health and stress prediction using nlp and transformer-based techniques. 2024 IEEE Symposium on Wireless Technology & Applications (ISWTA), pages 61–66, 2024. p-value Two-Stage Mean Diff Sig 0.8654 ± 0.0023 0.8421 ± 0.0019 0.8625 ± 0.0042 0.8506 ± 0.0013 × × × × +0.0016 +0.0018 +0.0008 +0.0015 0.09 0.12 0.31 0.08 find
  8. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. Mentalbert: Publicly available pretrained language models for mental healthcare. In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7184–7190, Marseille, France, 2022. European Language Resources Association (ELRA). find
  9. Muhammad Abdul-mageed. ARBERT & MARBERT : Deep Bidirectional Transformers for Arabic. (ii):7088– 7105, 2021. find our card
  10. Anoushka Thakkar, Ankita Gupta, and Avinash De Sousa. Artificial intelligence in positive mental health: a narrative review. Frontiers in Digital Health, Volume 6 - 2024, 2024. find
  11. Giuliano Lorenzoni, Cristina Tavares, Nathalia Nascimento, Paulo Alencar, and Donald Cowan. Assessing ML Classification Algorithms and NLP Techniques for Depression Detection: An Experimental Case Study. 2024. find
  12. AI Jaman, M Islam, S Sakib, and M Khan. Utilization of machine learning classifiers to predict different forms of mental illness: schizophrenia, PTSD, bipolar disorder and depression. (June), 2021. find
  13. Areej Alzoubi, Ahmad Alaiad, Khaled Alkhattib, Ahed J Alkhatib, Aseel Abu Aqoulah, Almo’men bellah Alawnah, and Ola Hayajnah. Detection of depression from arabic tweets using machine learning. Sustainable Machine Intelligence Journal, 6:1–12, 2024. find
  14. Rabia Qayyum, Hammad Afzal, Khawir Mahmood, and Naima Iltaf. Detection and Analysis of Mental Health Illness using Social Media. 4th International Conference on Communication Technologies, ComTech 2023, pages 34–41, 2023. find
  15. Muhammad Arif, Iqra Ameer, Necva Bölücü, Grigori Sidorov, Alexander Gelbukh, and Vinnayak Elangovan. Mental Illness Classification on Social Media Texts Using Deep Learning and Transfer Learning. Computacion y Sistemas, 28(2):451–464, 2024. find
  16. Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K. Dey, and Dakuo Wang. Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(1):1–32, 2024. find
  17. Abdelrahaman A. Hassan, Radwa J. Hanafy, and Mohammed E. Fouda. Automated Multi-Label Annotation for Mental Health Illnesses Using Large Language Models. 2024. find
  18. Suzan Elmajali and Irfan Ahmad. Toward Early Detection of Depression: Detecting Depression Symptoms in Arabic Tweets Using Pretrained Transformers. IEEE Access, 12(June):88134–88145, 2024. find
  19. Loukas Ilias, Spiros Mouzakitis, and Dimitris Askounis. Calibration of Transformer-Based Models for Identifying Stress and Depression in Social Media. IEEE Transactions on Computational Social Systems, 11(2):1979–1990, 2024. find
  20. Alireza Pourkeyvan, Ramin Safa, and Ali Sorourkhah. Harnessing the power of hugging face transformers for predicting mental health disorders in social networks. IEEE Access, 12:28025–28035, 2024. find
  21. Yining Hua, Fenglin Liu, Kailai Yang, Zehan Li, Yi-han Sheu, Peilin Zhou, Lauren V. Moran, Sophia Ananiadou, and Andrew Beam. Large Language Models in Mental Health Care: a Scoping Review. pages 1–22, 2024. find
  22. Wei Zhai, Hongzhi Qi, Qing Zhao, Jianqiang Li, Ziqi Wang, Han Wang, Bing Xiang Yang, and Guanghui Fu. Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis. 2023. find
  23. Apify. Twitter search scraper, 2025. Accessed: 2025-09-01. find
  24. Klaus Krippendorff. Content Analysis: An Introduction to Its Methodology. Sage Publications, 4th edition, 2018. find
  25. Joseph L. Fleiss, Bruce Levin, and Myunghee Cho Paik. Statistical Methods for Rates and Proportions. John Wiley & Sons, 3rd edition, 2003. find
  26. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. find
  27. Mary L. McHugh. Interrater reliability: The kappa statistic. Biochemia Medica, 22(3):276–282, 2012. find
  28. J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174, 1977. find
  29. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. The interplay of variant, size, and task type in arabic pre-trained language models. 9 2021. find our card
  30. Suchin Gururangan, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don ’ t Stop Pretraining : Adapt Language Models to Domains and Tasks. 2019. find
  31. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. find
  32. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019. find
  33. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. 7 2019. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/almalki-2026-mentalmarbert-domain-adaptive-pre-training/