Elamur Research⌕
☰
home / papers / 2026 / Large Language Models for Mental Health: A Multilingual Evaluation

Large Language Models for Mental Health: A Multilingual Evaluation

Paper · 2026 · Nishat Raihan, Sadiya Sayara Chowdhury Puspo, Ana-Maria Bucur et al.

Summary

Most work on LLMs for mental health text uses English data, so it is unclear how these models perform on languages other than English. This paper benchmarks seven LLMs on eight public social-media and blog datasets in six languages: Arabic, Bengali, Spanish, Portuguese, Russian and Thai. The datasets cover depression and suicidal ideation detection. The models are tested with zero-shot, few-shot and Chain-of-Thought (CoT) prompting. The open-source models are also instruction fine-tuned. Results are compared with the best scores reported in the original dataset papers.

The authors also build back-translated (MT) versions of each dataset. They score translation quality with BLEU, BERTScore and LaBSE and relate it to language typology. The strongest results come from proprietary models with CoT prompting and from fine-tuned open-source models. These often match or beat earlier statistical, neural and BERT-based systems. Scores on MT data are usually lower, and the size of the drop depends on how well the translation keeps meaning and how far the language's structure differs from English. The study shows where MT-based resources can be trusted for sensitive multilingual applications.

Key points

  • Models evaluated: GPT4-omni, Claude3.5-Sonnet and Gemini2-Flash (proprietary), plus LLaMA3.2 11B, Gemma2 27B, Ministral 8B and R1 14B (open-source). Few-shot prompting used 5 examples.
  • Strong results with CoT: Claude3.5-Sonnet with CoT reached F1 0.95 on Thai SIED (Boonyarat et al.), above LFBERT's 0.93. It also scored 0.95 on Portuguese suicide data (Random Forest baseline: 0.94) and 0.91 on Bengali depression data (GRU baseline: 0.76).
  • Baselines not always beaten: On the Arabic dataset of Baghdadi et al., AraElectra's 0.96 stayed ahead of every LLM. Ministral 8B was the weakest model overall, with several F1 scores below 0.5.
  • Fine-tuning: Instruction fine-tuning clearly raised open-source model scores on all datasets and made them less variable than in the zero-shot setting.
  • Effect of typology on MT data: Fusional languages (Portuguese, Russian) kept meaning better in translation and lost less performance. Analytic and templatic languages (Thai, Arabic) showed lower lexical overlap and larger drops. Code is available at github.com/SadiyaPuspo/Multilingual-Mental-Health-Evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Politècnica de ValènciaValencia · Spain34 works · 3,575 citations🎓University of MinnesotaMinneapolis · United States28 works · 2,184 citations🎓University of BucharestBucharest · Romania21 works · 516 citations🎓George Mason UniversityFairfax · United States24 works · 1,288 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Nishat Raihan, Sadiya Sayara Chowdhury Puspo, Ana-Maria Bucur, Stevie Chancellor and Marcos Zampieri. 2026. Large Language Models for Mental Health: A Multilingual Evaluation. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.29

@inproceedings{raihan2026large,
  title = {{Large Language Models for Mental Health: A Multilingual Evaluation}},
  author = {Nishat Raihan, Sadiya Sayara Chowdhury Puspo, Ana-Maria Bucur, Stevie Chancellor and Marcos Zampieri},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.29},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.29}
}

References

52 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nuredin Ali Abdelkadir, Charles Zhang, Ned Mayo, and Stevie Chancellor. 2024. Diverse perspectives, divergent models: Cross-cultural evaluation of depression detection on Twitter. In Proceedings of NAACL. find
  2. Prottay Kumar Adhikary, Aseem Srivastava, Shivani Kumar, Salam Michael Singh, Puneet Manuja, Jini K Gopinath, Vijay Krishnan, Swati Kedia Gupta, Koushik Sinha Deb, and Tanmoy Chakraborty. 2024. Exploring the efficacy of large language models in summarizing mental health counseling sessions: benchmar… find
  3. Kabir Ahuja, Monojit Choudhury, and Sandipan Dandapat. 2022. On the economics of multilingual few-shot learning: Modeling the cost-performance trade-offs of machine translated and manual data. In Proceedings of NAACL. find
  4. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. In Proceedings of ClinicalNLP.
  5. Anthropic. 2023. Claude: The anthropic ai language model. Online documentation. Available at: https://www.anthropic.com. ↗
  6. Nadiah A Baghdadi, Amer Malki, Hossam Magdy Balaha, Yousry AbdulAzeem, Mahmoud Badawy, and Mostafa Elhosseini. 2022. An optimized deep learning approach for suicide detection through arabic tweets. PeerJ Computer Science. find
  7. Outi Bat-El. 2019. Templatic morphology (clippings, word-and-pattern). find
  8. Panchanit Boonyarat, Di Jie Liew, and Yung-Chun Chang. 2024. Leveraging enhanced bert models for detecting suicidal ideation in thai social media content amidst covid-19. Information Processing & Management. find
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Amrita Ganguly, Sadiya Sayara Chowdhury Puspo, Md Nishat Raihan, Dhiman Goswami, Marcos Zampieri, et al. 2024. Masonperplexity at multimodal hate speech event detection 2024: Hate speech and target detection using transformer ensembles. In Proceedings of CASE. find
  11. Muskan Garg. 2024. Towards mental health analysis in social media for low-resourced languages. ACM Transactions on Asian and Low-Resource Language Information Processing, 23(3):1–22. find
  12. Gemini Team. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  13. Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  14. Mozhgan Ghassemiazghandi. 2024. An evaluation of chatgpt’s translation accuracy using bleu score. Theory and Practice in Language Studies, 14(4):985–994. find
  15. Dhiman Goswami, Md Nishat Raihan, Sadiya Sayara Chowdhury Puspo, and Marcos Zampieri. 2023. nlpbdpatriots at blp-2023 task 2: A transfer learning approach towards bangla sentiment analysis. In Proceedings of BLP. find
  16. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  17. Mika Hämäläinen, Pattama Patpong, Khalid Alnajjar, Niko Partanen, and Jack Rueter. 2021. Detecting depression in thai blog posts: a dataset and a baseline. In Proceedings of WNUT. find
  18. Martin Haspelmath. 2005. The world atlas of language structures. Oxford University Press. find
  19. Ana-Maria Bucur, Marcos Zampieri, Tharindu Ranasinghe, and Fabio Crestani. 2025. A survey on multilingual mental disorders detection from social media data. arXiv preprint arXiv:2505.15556. find
  20. AbdelMoniem Helmy, Radwa Nassar, and Nagy Ramdan. 2024. Depression detection for twitter users using sentiment analysis in english and arabic tweets. Artificial intelligence in medicine. find
  21. Richard Oliver Collin. 2010. Ethnologue. Ethnopolitics, 9(3-4):425–432. find
  22. Yining Hua, Fenglin Liu, Kailai Yang, Zehan Li, et al. 2024. Large language models in mental health care: a scoping review. arXiv preprint arXiv:2401.02984. — [arXiv:2401.02984](https://arxiv.org/abs/2401.02984) find
  23. Adonias C de Oliveira, Evandro JS Diniz, Silmar Teixeira, and Ariel S Teles. 2022. How can machine learning identify suicidal ideation from user’s texts? towards the explanation of the boa-mente system. Procedia Computer Science. find
  24. Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. 2024. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. In Proceedings of WWW. find
  25. Rahul Kande, Vasudev Gohil, Matthew DeLorenzo, Chen Chen, and Jeyavijayan Rajendran. 2024. Llms for hardware security: Boon or bane? In Proceedings of IEEE VTS. find
  26. Pitso Walter Khoboko, Vukosi Marivate, and Joseph Sefara. 2025. Optimizing translation for low-resource languages: Efficient fine-tuning with custom prompt engineering in large language models. Machine Learning with Applications, 20:100649. find
  27. Nathaniel Krasner, Justin Vasselli, Belu Ticona, Antonios Anastasopoulos, and Chi-kiu Lo. 2025. Machine translation metrics for indigenous languages using fine-tuned semantic embeddings. In Proceedings of AmericasNLP. find
  28. Gleb Kuzmin, Petr Strepetov, Maksim Stankevich, Artem Shelmanov, and Ivan Smirnov. 2024. Mental disorders detection in the era of large language models. arXiv preprint arXiv:2410.07129. classification for violence inciting text detection in bangla-leveraging back-translation and multilinguality. In… — [arXiv:2410.07129](https://arxiv.org/abs/2410.07129) find
  29. Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. 2025a. mhumaneval–a multilingual benchmark to evaluate large language models for code generation. In Proceedings of NAACL. find
  30. Nishat Raihan, Sadiya Sayara Chowdhury Puspo, Shafkat Farabi, Ana-Maria Bucur, Tharindu Ranasinghe, and Marcos Zampieri. 2024. Mentalhelp: A multi-task dataset for mental health in social media. In Proceedings of LREC-COLING. find
  31. Nishat Raihan, Mohammed Latif Siddiq, Joanna Santos, and Marcos Zampieri. 2025b. Large language models in computer science education: A systematic literature review. In Proceedings of SIGCSE. find
  32. Shervin Malmasi, Marcos Zampieri, and Mark Dras. 2016. Predicting post severity in mental health forums. In Proceedings of CLPsych. find
  33. Konstantinos Skianis, A Seza Doğruöz, and John Pavlopoulos. 2024a. Leveraging llms for translating and classifying mental health data. arXiv preprint arXiv:2410.12985. find
  34. Umasree Mariappan, D Balakrishnan, G Merline, M Sandhia, Dubba Saitej Reddy, and Satti-neni Gagan Teja. 2024. Mental health disorder prediction using recurrent neural network algorithm. In Proceedings of APCIT. find
  35. Konstantinos Skianis, John Pavlopoulos, and A Seza Doğruöz. 2024b. Building multilingual datasets for predicting mental health severity through llms: Prospects and challenges. arXiv preprint arXiv:2409.17397. — [arXiv:2409.17397](https://arxiv.org/abs/2409.17397) find
  36. John Mendonça, Alon Lavie, and Isabel Trancoso. 2023. Towards multilingual automatic dialogue evaluation. arXiv preprint arXiv:2308.16795. — [arXiv:2308.16795](https://arxiv.org/abs/2308.16795) find
  37. Konstantinos Skianis, John Pavlopoulos, and A Seza Dogruöz. 2024c. Severity prediction in mental health: Llm-based creation, analysis, evaluation of a novel multilingual dataset. CoRR. find
  38. David N Milne, Glen Pink, Ben Hachey, and Rafael A Calvo. 2016. find
  39. Clpsych 2016 shared task: Triaging content in online peer-support forums. In Proceedings of CLPsych. find
  40. Sergazy Narynov, Daniyar Mukhtarkhanuly, and Batyrkhan Omarov. 2020. Dataset of depressive posts in russian language collected from social media. Data in brief, 29:105195. find
  41. Thao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt, Jason E Weston, Luke Zettlemoyer, and Xian Li. 2024. Better alignment with instruction back-and-forth translation. In Findings of EMNLP. find
  42. OpenAI. 2023. Gpt-4 technical report. preprint arXiv:2303.08774. arXiv Chen Qiu, Dan Oneat , ă, Emanuele Bugliarello, Stella Frank, and Desmond Elliott. 2022. Multilingual multimodal learning with machine translated text. In Findings of EMNLP.
  43. Md Nishat Raihan, Dhiman Goswami, Sadiya Sayara Chowdhury Puspo, and Marcos Zampieri. 2023. nlpbdpatriots at blp-2023 task 1: Two-step Elsbeth Turcan and Kathleen Mckeown. 2019. Dreaddit: A reddit dataset for stress analysis in social media. In Proceedings of LOUHI. find
  44. Abdul Hasib Uddin, Durjoy Bapery, and Abu Shamim Mohammad Arif. 2019. Depression analysis of bangla social media data using gated recurrent neural network. In Proceedings of IEEE ICASERT. find
  45. Kid Valeriano, Alexia Condori-Larico, and Josè Sulla-Torres. 2020. Detection of suicidal intent in spanish language social networks using machine learning. International Journal of Advanced Computer Science and Applications. find
  46. Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digital Scholarship in the Humanities, 30(1):98–118. find
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of NeurIPS.
  48. Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K Dey, and Dakuo Wang. 2024. Mental-llm: Leveraging large language models for mental health prediction via online text data. Proceedings of IMWUT. find
  49. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. In Proceedings of EMNLP. find
  50. Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Mentallama: interpretable mental health analysis on social media with large language models. In Proceedings of WWW. find
  51. Noureldin Zahran, Aya E Fouda, Radwa J Hanafy, and Mohammed E Fouda. 2025. A comprehensive evaluation of large language models on mental illnesses in arabic context. arXiv preprint arXiv:2501.06859.
  52. Rodolfo Joel Zevallos, Annika Marie Schoene, and John E Ortega. 2025. The first multilingual model for the detection of suicide texts. In Proceedings of SUMEval. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/raihan-2026-large-language-models-mental-health/