Elamur Research⌕
☰
home / papers / 2025 / CARMA: Comprehensive Automatically-annotated Reddit Mental Health Dataset for Arabic

CARMA: Comprehensive Automatically-annotated Reddit Mental Health Dataset for Arabic

Summary

Arabic-language mental health detection has little annotated data. The existing corpora are mostly older Twitter collections that cover only depression and suicide. The authors build CARMA, a Reddit-based Arabic dataset of over 340K posts. It covers six conditions (ADHD, Anxiety, Autism, Depression, OCD, Suicide) plus a control group, and is labeled automatically.

Diagnosed users are found with self-reported diagnosis patterns: a diagnosis keyword must appear near a condition phrase. The pattern lists include dialect and misspelling variants. Control users come from the same subreddits, and anyone who ever triggers the pattern is excluded. The data is then filtered by language and post length, and dialects are profiled.

The authors also compare lexical markers per condition using TF-IDF differences. They test two setups:

  • User-level: shallow classifiers on Arabic transformer embeddings reach F1 between 0.67 and 0.83.
  • Post-level: directly fine-tuned models perform far worse.

The resource broadens Arabic mental health NLP beyond depression and suicide. It is available under a data usage agreement.

Key points

  • Collection: Data was gathered through the Arctic Shift API from eight general Arabic-active subreddits, including r/Egypt, r/jordan, r/CAIRO and r/SaudiForSaudis. Arabic Reddit has no dedicated mental health communities.
  • Matching and audit: The keyword–phrase distance threshold is 40 characters. Matching yielded 959 candidate diagnosed users. In a manual audit of 100 matched posts, 83 were valid self-diagnoses. The false positives mostly described someone else's diagnosis or were negated statements.
  • Final dataset size:
    • Control: 4,086 users and 270,622 posts.
    • Anxiety: 187 users and 22,306 posts.
    • Depression: 171 users and 21,085 posts.
    • Suicide: the smallest class, with 36 users.
  • Cleaning:
    • langdetect removed about 30% of diagnosed-user posts and 13% of control posts.
    • Posts under 10 words were dropped.
    • A BERT dialect identifier showed Egyptian Arabic dominates, followed by Jordanian, Saudi and Palestinian.
  • User-level results (1-to-1 against control):
    • Anxiety: best F1 0.83, using SVM on Arabic-labse embeddings.
    • OCD and Autism: F1 0.82.
    • Suicide: lowest F1, 0.67.
  • Post-level results: Fine-tuning CAMeLBERT, AraBERT, MARBERTv2 and similar models gave F1 of only 0.21–0.37.

Interesting ideas

  • I1. Label by checking that a diagnosis keyword and a condition phrase occur within a short character window, then hand-audit a small sample to name the error types (third-party mentions, negation); this gives cheap, high-recall silver labels plus a map of where they fail. — For Mova (frames): The two audited failure types map directly onto tree features, namely whether the experiencer of "diagnosed" is first person and whether a negation dependent is present. A frame-level check could therefore filter such silver labels far more precisely than a character window.
  • I2. Build controls from the same communities and drop any user who ever matches the positive pattern, so the classifier cannot exploit venue differences or hidden positives in the negative class. — For Mova (learn): Negative-control sets could be drawn from the same source texts as the positives. Any item that any witness parser or rule flags would then be excluded, keeping evaluation free of leaked positives.
  • I3. The large gap between user-level and post-level scores shows that weak signals become reliable only when pooled over many texts from one source. — For Mova (world): Judgments about an entity's state, such as an emotion or a holding, could be accumulated across all mentions in the text world rather than decided from one sentence. Each accumulated judgment would keep explainable links back to the sentences that support it.
  • I4. The matching lexicons were enriched with dialect variants and common misspellings to recover recall in noisy user text. — For Mova (tokens): A curated variant table mapping misspellings and informal forms to canonical lemmas could sit in front of the lemmatizer. This would help on noisy English inputs without turning into an opaque normalizer.
  • I5. Comparing per-condition TF-IDF differences against the control group gives an interpretable, lexical view of what separates the classes before any model is trained. — For Mova (learn): Contrasting feature frequencies between a target set and a matched control set could be used to find which lemmas or frame types drive parser or reader errors. The findings could then seed self-induced repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.03102 · arXiv 2511.03102

@misc{mankarious2025carma,
  title = {{CARMA: Comprehensive Automatically-annotated Reddit Mental Health Dataset for Arabic}},
  author = {Mankarious, Saad and Zirikly, Ayah},
  year = {2025},
  doi = {10.48550/arxiv.2511.03102},
  eprint = {2511.03102},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.03102}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hassan Alhuzali, Farah Shamout, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Walid Al-Eisawi, Renad Al-Monef, Ali Alqahtani, Lama Ayash, et al. Arahealthqa 2025: The first shared task on arabic health question answering. arXiv e-prints, pages arXiv–2508, 2025. find
  2. Mariam Hassib, Nancy Hossam, Jolie Sameh, and Marwan Torki. AraDepSu: Detecting depression and suicidal ideation in Arabic tweets using transformers. In Houda Bouamor, Hend Al-Khalifa, Kareem Darwish, Owen Rambow, Fethi Bougares, Ahmed Abdelali, Nadi Tomeh, Salam Khalifa, and Wajdi Zaghouani, editors, Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 302–311, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. find
  3. Mohammad El-Ramly, Hager Abu-Elyazid, Youseef Mo’men, Gameel Alshaer, Nardine Adib, Kareem Alaa Eldeen, and Mariam El-Shazly. Cairodep: Detecting depression in arabic posts using bert transformers. In 2021 Tenth International Conference on Intelligent Computing and Information Systems (ICICIS), pages 207–212, 2021. find
  4. Ashwag Maghraby and Hosnia Ali. Modern standard arabic mood changing and depression dataset. Data in Brief, 41:107999, 2022. find
  5. Damian F. Santomauro, Ana M. Mantilla Herrera, Junfang Shadid, Peng Zheng, Charlie Ashbaugh, Giorgia Pigott, Harish G. Sheena, and et al. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the covid-19 pandemic. The Lancet, 398(10312):1700–1712, 2021. find
  6. La Vonne A Downey, Leslie S Zun, and Trena Burke. Undiagnosed mental illness in the emergency department. The Journal of Emergency Medicine, 43(5):876–882, 2012. find
  7. Hania El Khatib, Aisha Alyafei, and Madiha Shaikh. Understanding experiences of mental health help-seeking in arab populations around the world: A systematic review and narrative synthesis. BMC Psychiatry, 23(1):324, 2023. find
  8. Andrew G. Reece and Christopher M. Danforth. Instagram photos reveal predictive markers of bipolar disorder. EPJ Data Science, 6(1):1–12, 2017. find
  9. Minsu Park, David W. McDonald, and Meeyoung Cha. Examining user behaviors on social media for mental health status. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pages 3207–3216. ACM, 2013. find
  10. Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney, and Nazli Goharian. Smhd: A large-scale resource for exploring online language usage for multiple mental health conditions. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 1485–1497, Santa Fe, New Mexico, USA, 2018. Association for Computational Linguistics. find
  11. Han-Chin Shing, Suraj Nair, Ayah Zirikly, Meir Friedenberg, Hal Daumé III, and Philip Resnik. Expert, crowdsourced, and machine assessment of suicide risk via online postings. In Kate Loveys, Kate Niederhoffer, Emily Prud’hommeaux, Rebecca Resnik, and Philip Resnik, editors, Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pages 25–36, New Orleans, LA, June 2018. Association for Computational Linguistics. Reddit permits third-party access to public content via APIs (see https://www.reddit.com/policies/privacy-policy for details). ↗ our card
  12. Ayah Zirikly, Philip Resnik, Özlem Uzuner, and Kristy Hollingshead. CLPsych 2019 shared task: Predicting the degree of suicide risk in reddit posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology (CLPsych), pages 24–33. Association for Computational Linguistics, 2019. find our card
  13. Hassan Alhuzali, Ashwag Alasmari, and Hamad Alsaleh. MentalQA: An Annotated Arabic Corpus for Questions and Answers of Mental Healthcare. IEEE Access, 2024. find
  14. Alina Trifan, Rui Antunes, Sérgio Matos, and Jose Luís Oliveira. Understanding depression from psycholinguistic patterns in social media texts. In European conference on information retrieval, pages 402–409. Springer, 2020. find
  15. Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. From ADHD to SAD: Analyzing the language of mental health on twitter through self-reported diagnoses. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 1–10, Denver, Colorado, 2015. Association for Computational Linguistics. find
  16. Tamás Kovács, Florian Atzenhofer-Baumgartner, Sandy Aoun, Anguelos Nicolaou, Daniel Luger, Franziska Decker, Florian Lamminger, and Georg Vogeler. langdetect (revision 0215f72), 2022. find
  17. Jenny Chim, Adam Tsakalidis, Dimitris Gkoumas, Dana Atzil-Slonim, Yaakov Ophir, Ayah Zirikly, Philip Resnik, and Maria Liakata. Overview of the CLPsych 2024 shared task: Leveraging large language models to identify evidence of suicidality risk in online posts. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 177–190, St. Julians, Malta, March 2024. Association for Computational Linguistics. find
  18. Talia Tseriotou, Jenny Chim, Ayal Klein, Aya Shamir, Guy Dvir, Iqra Ali, Cian Kennedy, Guneet Singh Kohli, Anthony Hills, Ayah Zirikly, Dana Atzil-Slonim, and Maria Liakata. Overview of the CLPsych 2025 shared task: Capturing mental health dynamics from social media timelines. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025), pages 193–217, Albuquerque, New Mexico, May 2025. Association for Computational Linguistics. find
  19. AbdulRahman A. Morsy, Saad Mankarious, and Ayah Zirikly. Sindbad at AraHealthQA track 1: Leveraging large language models for mental health Q&A. In Kareem Darwish, Ahmed Ali, Ibrahim Abu Farha, Samia Touileb, Imed Zitouni, Ahmed Abdelali, Sharefah Al-Ghamdi, Sakhar Alkhereyf, Wajdi Zaghouani, Salam Khalifa, Badr AlKhamissi, Rawan Almatham, Injy Hamed, Zaid Alyafeai, Areeb Alowisheq, Go Inoue, Khalil Mrini, and Waad Alshammari, editors, Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 184–191, Suzhou, China, November 2025. Association for Computational Linguistics. find
  20. Andrew Yates, Arman Cohan, and Nazli Goharian. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2968–2978. ACL, 2017. find our card
  21. Bhargavi Zanwar, Azza Abouzied, João Sedoc, and Walid Magdy. Smhd-ger: A large-scale benchmark dataset for automatic mental health detection from social media in german. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1733–1745. Association for Computational Linguistics, 2023. find
  22. Omer Nacar and Anis Koubaa. Enhancing semantic similarity understanding in arabic nlp with nested embedding learning, 2024. find our card
  23. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online, August 2021. Association for Computational Linguistics. find our card
  24. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online), April 2021. Association for Computational Linguistics. find our card
  25. Vladimir Araujo, Andres Carvallo, Denis Parra, Daniel Trister, and Felipe Bravo-Marquez. Multilingual BERT for CLIR in Low-Resource languages. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, pages 2283–2287, New York, NY, USA, 2021. Association for Computing Machinery. find
  26. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. find our card
  27. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT 2020), pages 9–15, Marseille, France, 2020. European Language Resource Association (ELRA). Artifact Emojis Numbers English Words URLs Hashtags Mentions Table 6: Unwanted artifacts detected in the dataset and percentage of affected posts. All such artifacts were removed from the dataset. Figure 3: Dialects represented in the dataset. find our card
  28. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. Arbert & marbert: Deep bidirectional transformers for arabic. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7088–7105, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. find our card
  29. Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish, Mohamed Eldesouki, Mohamed Elaraby, Salam Khalifa, Rawan Almatham, and Saad Al-Kahtani. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 14005–14022, Singapore, 2023. Association for Computational Linguistics. find our card
  30. Wenhui Wang, Furu Wei, Li Dong, Hang Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020. find
  31. Armen Aghajanyan, Akshat Gupta, Akshay Shrivastava, Luke Zettlemoyer, and Sonal Gupta. MUPPET: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5799–5811. Association for Computational Linguistics, 2021. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mankarious-2025-carma-comprehensive-automatically-annotated-reddit/