Elamur Research⌕
☰
home / papers / 2024 / Evaluating the Effectiveness of the Foundational Models for Q&A Classification in Mental Health care

Evaluating the Effectiveness of the Foundational Models for Q&A Classification in Mental Health care

Paper · 2024 · Hassan Alhuzali and Ashwag Alasmari

Summary

The paper asks how well current language models can sort Arabic mental-health questions and answers into types, a task that could help route help-seekers to suitable resources. Arabic is poorly covered by mental-health NLP resources, so the authors run the first experiments on MentalQA, a recently released Arabic corpus of mental-health Q&A exchanges. They compare four regimes on the same data: hand-crafted features with SVM, frozen PLMs used only as feature extractors, fully fine-tuned Arabic PLMs (AraBERT, CAMeLBERT, MARBERT), and prompting GPT-3.5 and GPT-4 with zero or a few examples. Fine-tuned MARBERT performs best, with a Jaccard score of 0.80 on questions and 0.86 on answers. The classical SVM baseline is already competitive. Follow-up analyses look at fine-tuning versus freezing, training-set size, and the kinds of errors the models make. Few-shot prompting clearly beats zero-shot. The work matters because it sets baselines for a low-resource language in a sensitive domain. The authors stress that the classifiers are meant for non-clinical guidance, not diagnosis.

Key points

  • Data: MentalQA, an Arabic mental-health question-answering corpus. The task is to classify both question types and answer types.
  • Four approaches were compared: traditional features + SVM, PLMs as frozen feature extractors, fine-tuned PLMs, and GPT-3.5/GPT-4 prompting in zero-shot and few-shot settings.
  • Best system: fine-tuned MARBERT, with a Jaccard score of 0.80 for questions and 0.86 for answers. AraBERT was also strong.
  • Analysis findings:
    • Fine-tuning clearly helps the PLMs compared with leaving them frozen.
    • The amount of training data strongly affects results.
  • With GPT-3.5, moving from zero-shot to few-shot prompting raised performance by 12% when classifying questions and by 45% when classifying answers.

Interesting ideas

  • I1. Run a whole ladder of learning regimes on one dataset, from classical features through frozen and fine-tuned encoders to LLM prompting, so each gain can be traced to its source rather than credited to "the model" as a whole.
  • I2. Compare a frozen encoder with the same encoder after fine-tuning; this cleanly separates what pretraining gives from what task adaptation gives.
  • I3. Plot performance against training-set size to judge whether more annotation is worth producing.
  • I4. Contrast zero-shot and few-shot prompting of the same LLM; the large jump on answers suggests that worked examples, not model size alone, drive quality.
  • I5. Score set-valued predictions with a Jaccard score, which gives partial credit for overlapping label sets instead of all-or-nothing accuracy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): every new component could be reported against the same ladder of cheaper baselines, so we can see whether its extra complexity actually buys accuracy.
  • I2 (tree): gparse could be evaluated with a frozen versus a fine-tuned transformer backbone, showing how much of its LAS comes from adaptation to UD treebanks.
  • I3 (learn): learning curves over growing slices of agreed silver data would tell us when generating more silver parses stops paying off.
  • I4 (learn): the LLM teacher tiers could always include a few gold exemplars from treebanks or annotated frames, with zero-shot kept as a measured control.
  • I5 (frames): utterances that carry several speech acts or roles at once could be scored by set overlap, making evaluation of prag less brittle.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Umm al-Qura UniversityMakkah al Mukarramah · Saudi Arabia8 works · 87 citations

Also on the byline: King Khalid University.

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Social Psychology · 129 works

🧭 Topics 2 Evaluation & benchmarks · 685 works Semitic languages · 1,484 works

Cite

Cite as Hassan Alhuzali and Ashwag Alasmari. 2024. Evaluating the Effectiveness of the Foundational Models for Q&A Classification in Mental Health care. arXiv:2406.15966.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.15966 · arXiv 2406.15966

@misc{alhuzali2024evaluating,
  title = {{Evaluating the Effectiveness of the Foundational Models for Q\&A Classification in Mental Health care}},
  author = {Hassan Alhuzali and Ashwag Alasmari},
  year = {2024},
  doi = {10.48550/arxiv.2406.15966},
  eprint = {2406.15966},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.15966}
}

References

67 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. World Health Organization. World mental health report: Transforming mental health for all. 2022. find
  2. WHO World Mental Health Survey Consortium et al. Prevalence, severity, and unmet need for treatment of mental disorders in the world health organization world mental health surveys. Jama, 291(21):2581–2590, 2004. find
  3. Patricio V Marquez and Shekhar Saxena. Making mental health a global priority. In Cerebrum: the Dana forum on brain science, volume 2016. Dana Foundation, 2016. find
  4. Aziliz Le Glaz, Yannis Haralambous, Deok-Hee Kim-Dufor, Philippe Lenca, Romain Billot, Taylor C Ryan, Jonathan Marsh, Jordan Devylder, Michel Walter, Sofian Berrouiguet, and Christophe Lemey. Machine learning and natural language processing in mental health: systematic review. Journal of medical Internet research, 23(5):e15708, 2021. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019. our card
  6. Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. arXiv preprint arXiv:2310.05694, 2023. ↗ our card
  7. June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. Chatcounselor: A large language models for mental health support. arXiv preprint arXiv:2309.15461, 2023. ↗
  8. Lennart Brocki, George C Dyer, Anna Gładka, and Neo Christopher Chung. Deep learning mental health dialogue system. In 2023 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 395–398. IEEE, 2023. find
  9. Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. Human–ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1):46–57, 2023. ↗
  10. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507, 2021. our card
  11. Thushari Atapattu, Mahen Herath, Charitha Elvitigala, Piyanjali de Zoysa, Kasun Gunawardana, Menasha Thilakaratne, Kasun de Zoysa, and Katrina Falkner. Emoment: An emotion annotated mental health corpus from two south asian countries. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6991–7001, 2022. ↗
  12. Muhammad Khubayeeb Kabir, Maisha Islam, Anika Nahian Binte Kabir, Adiba Haque, and Md Khalilur Rhaman. Detection of depression severity using bengali social media posts on mental health: Study using natural language processing techniques. JMIR Formative Research, 6(9):e36118, 2022. ↗
  13. Hao Sun, Zhenru Lin, Chujie Zheng, Siyang Liu, and Minlie Huang. Psyqa: A chinese dataset for generating long counseling text for mental health support. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1489–1503, 2021. find
  14. Tianlin Zhang, Annika M Schoene, Shaoxiong Ji, and Sophia Ananiadou. Natural language processing applied to mental illness detection: a narrative review. NPJ digital medicine, 5(1):46, 2022. ↗
  15. Asma Abdulsalam, Areej Alhothali, and Saleh Al-Ghamdi. Detecting suicidality in arabic tweets using machine learning and deep learning techniques. Arabian Journal for Science and Engineering, pages 1–14, 2024. ↗ our card
  16. Shahad Hathal Aldhafer and Mourad Yakhlef. Depression detection in arabic tweets using deep learning. In 2022 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), pages 1–6. IEEE, 2022. find
  17. Norah Al-Musallam and Mohammed Al-Abdullatif. Depression detection through identifying depressive arabic tweets from saudi arabia: Machine learning approach. In 2022 Fifth National Conference of Saudi Computers Colleges (NCCC), pages 11–18. IEEE, 2022. find
  18. Ali Al-Laith and Mamdouh Alenezi. Monitoring people’s emotions and symptoms from arabic tweets during the covid-19 pandemic. Information, 12(2):86, 2021. ↗
  19. Mohammad El-Ramly, Hager Abu-Elyazid, Youseef Mo’men, Gameel Alshaer, Nardine Adib, Kareem Alaa Eldeen, and Mariam El-Shazly. Cairodep: Detecting depression in arabic posts using bert transformers. In 2021 Tenth International Conference on Intelligent Computing and Information Systems (ICICIS), pages 207–212. IEEE, 2021. find
  20. Hassan Alhuzali, Ashwag Alasmari, and Hamad Alsaleh. Mentalqa: An annotated arabic corpus for questions and answers of mental healthcare. 2024. ↗
  21. Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim. Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research, 15(1):3133–3181, 2014. ↗
  22. Sagar S Nikam. A comparative study of classification techniques in data mining algorithms. Oriental Journal of Computer Science and Technology, 8(1):13–19, 2015. find
  23. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 328–339, 2018. find
  24. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. ↗
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. ↗
  26. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019. ↗ our card
  27. Khaled Shaalan, Sanjeera Siddiqui, Manar Alkhatib, and Azza Abdel Monem. Challenges in arabic natural language processing. In World Scientific: Computational linguistics, speech and image processing for arabic language, pages 59–83. 2019. ↗
  28. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. hulmona: The universal language model in arabic. In Proceedings of the fourth Arabic natural language processing workshop, pages 68–77, 2019. find
  29. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9, 2020. ↗ our card
  30. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. An empirical study of pre-trained transformers for arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734, 2020. find
  31. Muhammad Abdul Mageed and Abdelrahim Elmadany. Arbert & marbert: Deep bidirectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, 2021. ↗ our card
  32. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, 2021. our card
  33. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. AlGhafa evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, December 2023. find
  34. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, March 2024. ↗ our card
  35. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020. ↗ our card
  36. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, 2019. ↗ our card
  37. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. Mentalbert: Publicly available pretrained language models for mental healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7184–7190, 2022. find
  38. Zhijun Guo, Alvina Lai, Johan Hilge Thygesen, Joseph Farrington, Thomas Keen, and Kezhi Li. Large language model for mental health: A systematic review. arXiv preprint arXiv:2403.15401, 2024. ↗
  39. Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. Clpsych 2015 shared task: Depression and ptsd on twitter. In Proceedings of the 2nd workshop on computational linguistics and clinical psychology: from linguistic signal to clinical reality, pages 31–39, 2015. our card
  40. Ashwag Alasmari, Luke Kudryashov, Shweta Yadav, Heera Lee, and Dina Demner-Fushman. Chq-socioemo: Identifying social and emotional support needs in consumer-health questions. Scientific Data, 10(1):329, 2023. find
  41. Guangyao Shen, Jiang Jia, Liqiang Nie, Fuli Feng, Cunjun Zhang, Tianrui Hu, Tat-Seng Chua, and Wenwu Zhu. Depression detection via harvesting social media: a multimodal dictionary learning solution. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3838–3844, 2017. find
  42. Elsbeth Turcan and Kathleen Mckeown. Dreaddit: A reddit dataset for stress analysis in social media. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 97–107, 2019. ↗
  43. Aryan Rastogi, Qian Liu, and Erik Cambria. Stress detection from social media articles: New dataset benchmark and analytical study. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022. find
  44. Panchanit Boonyarat, Di Jie Liew, and Yung-Chun Chang. Leveraging enhanced bert models for detecting suicidal ideation in thai social media content amidst covid-19. Information Processing & Management, 61(4):103706, 2024. ↗
  45. Soumitra Ghosh, Asif Ekbal, and Pushpak Bhattacharyya. Cease, a corpus of emotion annotated suicide notes in english. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1618–1626, 2020. our card
  46. Jaya Chaturvedi, Sumithra Velupillai, Robert Stewart, and Angus Roberts. Identifying mentions of pain in mental health records text: A natural language processing approach. Studies in health technology and informatics, 310:695–699, 2024. ↗
  47. Muskan Garg, Chandni Saxena, Sriparna Saha, Veena Krishnan, Ruchi Joshi, and Vijay Mago. Cams: An annotated corpus for causal analysis of mental health issues in social media posts. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6387–6396, 2022. find
  48. Muskan Garg. Wellxplain: Wellness concept extraction and classification in reddit posts for mental health analysis. Knowledge-Based Systems, 284:111228, 2024. ↗
  49. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237, 2018. our card
  50. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509, 2018. ↗ our card
  51. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1756–1765, 2017. our card
  52. Yasmen Wahba, Nazim Madhavji, and John Steinbacher. A comparison of svm against pre-trained language models (plms) for text classification tasks. In International Conference on Machine Learning, Optimization, and Data Science, pages 304–313. Springer, 2022. find
  53. Hassan Alhuzali, Tianlin Zhang, and Sophia Ananiadou. Predicting sign of depression via using frozen pre-trained models and random forest classifier. In CLEF (Working Notes). 2021. find
  54. Hassan Alhuzali and Sophia Ananiadou. Spanemo: Casting multi-label emotion classification as span-prediction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1573–1584, 2021. find
  55. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615–1625, 2017. our card
  56. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. find
  57. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019. ↗ our card
  58. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, August 2021. ↗ our card
  59. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17, 2018. our card
  60. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. ↗
  61. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. ↗
  62. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, 2020. find
  63. Hassan Alhuzali and Sophia Ananiadou. Improving classification of adverse drug reactions through using sentiment analysis and transfer learning. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 339–347, 2019. find
  64. Tianlin Zhang, Kailai Yang, Shaoxiong Ji, and Sophia Ananiadou. Emotion fusion for mental illness detection from social media: A survey. Information Fusion, 92:231–246, 2023. ↗
  65. Hassan Alhuzali. Neural Networks for Textual Emotion Recognition and Analysis. The University of Manchester (United Kingdom), 2022. find
  66. Tianlin Zhang, Kailai Yang, Hassan Alhuzali, Boyang Liu, and Sophia Ananiadou. Phq-aware depressive symptoms identification with similarity contrastive learning on social media. Information Processing & Management, 60(5):103417, 2023. ↗
  67. Alhanoof Althnian, Duaa AlSaeed, Heyam Al-Baity, Amani Samha, Alanoud Bin Dris, Najla Alzakari, Afnan Abou Elwafa, and Heba Kurdi. Impact of dataset size on classification performance: an empirical evaluation in the medical domain. Applied Sciences, 11(2):796, 2021. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alhuzali-2024-evaluating-effectiveness-foundational-models-q/