Elamur Research⌕
☰
home / papers / 2026 / On the Role of Context in LLM Alignment to Mental Health Counseling Competencies

On the Role of Context in LLM Alignment to Mental Health Counseling Competencies

Paper · 2026 · Sadiya Sayara Chowdhury Puspo, Marcos Zampieri and Özlem Uzuner

Summary

Large language models can reach passing scores on mental health counseling exams. This paper asks whether those scores come from reasoning about the individual patient or from general symptom-to-diagnosis associations. The authors test LLaMA-3-8B-Instruct, LLaMA-3-70B-Instruct and the medically tuned OpenBioLLM-70B on CounselingBench, a benchmark of case vignettes drawn from NCMHCE licensing exam practice material.

They split each vignette's patient-specific context into four parts: demographics, presenting problem, mental status examination and other contexts. They then remove these parts step by step, shuffle answer options, reorder question and answer, assign the model different roles, and apply Thread-of-Thought (ThoT) prompting.

Removing context lowers accuracy only slightly. Option shuffling barely changes predictions, while question ordering has a larger effect. Neither role framing nor ThoT helps. Even when ThoT reasoning surfaces the relevant cue, the final answer often ignores it.

The work matters because benchmark-level competence may overstate how well models integrate context. This argues for evaluations built around context, such as contrastive vignettes, before LLMs are deployed in mental health care.

Key points

  • Dataset: CounselingBench has 1,612 questions across 138 case studies, covering five competencies:
    • CS&I: 599 questions
    • IA&D: 460 questions
    • PP&E: 274 questions
    • TP: 253 questions
    • CCA: 23 questions
  • Passing threshold: The typical NCMHCE pass mark is about 63% accuracy, and the chosen models had already reached or exceeded it in earlier work.
  • Role framing: Assigned personas had no measurable effect on accuracy. In 30.2% of cases, models gave the same wrong answer regardless of persona.
  • Error analysis: The authors examined 490 cases where all four role-framed prompts agreed on an incorrect answer. They found three recurring failure patterns:
    • broad-to-specific errors when selecting assessments
    • substitution of generic priorities at intake
    • neglect of relational context in ethics questions
  • Limitations: Only open models were tested, and the benchmark is US-specific, English-only and multiple-choice. Errors clustered in PP&E, and the authors flag possible benchmark contamination.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓George Mason UniversityFairfax · United States24 works · 1,288 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Sadiya Sayara Chowdhury Puspo, Marcos Zampieri and Özlem Uzuner. 2026. On the Role of Context in LLM Alignment to Mental Health Counseling Competencies. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.clpsych-1.6

@inproceedings{puspo2026role,
  title = {{On the Role of Context in LLM Alignment to Mental Health Counseling Competencies}},
  author = {Sadiya Sayara Chowdhury Puspo, Marcos Zampieri and Özlem Uzuner},
  booktitle = {Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.clpsych-1.6},
  url = {https://doi.org/10.18653/v1/2026.clpsych-1.6}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohsen Ghorbian and Mostafa Ghobaei-Arani. 2025. Large language models for mental health diagnosis and treatment: a survey. Artificial Intelligence Review, 59(1):9. find
  2. Hiba Arnaout, Anmol Goel, H Andrew Schwartz, Steffen T Eberhardt, Dana Atzil-Slonim, Gavin Doherty, Brian Schwartz, Wolfgang Lutz, Tim Althoff, Munmun De Choudhury, and 1 others. 2026. Responsible evaluation of ai for mental health. arXiv preprint arXiv:2602.00065. find
  3. Candida M Greco, Andrea Simeri, Andrea Tagarelli, and Ester Zumpano. 2023. Transformer-based language models for mental health issues: a survey. Pattern Recognition Letters, 167:204–211. find
  4. Ana-Maria Bucur. 2026. Computational approaches to mental health disorders detection from social media texts, images and videos. Procesamiento del Lenguaje Natural, 76:311–314. find
  5. Zhijun Guo, Alvina Lai, Johan H Thygesen, Joseph Farrington, Thomas Keen, Kezhi Li, and 1 others. 2024. Large language models for mental health applications: systematic review. JMIR mental health, 11(1):e57400. find
  6. Ana-Maria Bucur, Andreea Moldovan, Krutika Parvatikar, Marcos Zampieri, Ashiqur Khudabukhsh, and Liviu P Dinu. 2025a. Datasets for depression modeling in social media: An overview. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025), pages 116–126. find
  7. Zainab Iftikhar, Amy Xiao, Sean Ransom, Jeff Huang, and Harini Suresh. 2025. How llm counselors violate ethical standards in mental health practice: A practitioner-informed framework. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 1311–1323. find
  8. Ana-Maria Bucur, Andreea-Codrina Moldovan, Kru-tika Parvatikar, Marcos Zampieri, Ashiqur R KhudaBukhsh, and Liviu P Dinu. 2025b. On the state of nlp approaches to modeling depression in social media: A post-covid-19 outlook. IEEE Journal of Biomedical and Health Informatics. find
  9. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  10. Ana-Maria Bucur, Marcos Zampieri, Tharindu Ranasinghe, and Fabio Crestani. 2026. A survey on multilingual mental disorders detection from social media data. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8… find
  11. Yoon Kyung Lee, Inju Lee, Minjung Shin, Seoyeon Bae, and Sowon Hahn. 2023. Chain of empathy: Enhancing empathetic response of large language models based on psychotherapy models. arXiv preprint arXiv:2311.04915. — [arXiv:2311.04915](https://arxiv.org/abs/2311.04915) find
  12. Inbar Levkovich. 2025. Evaluating diagnostic accuracy and treatment efficacy in mental health: A comparative analysis of large language model tools and mental health professionals. European Journal of Investigation in Health, Psychology and Education, 15(1):9. find
  13. Mohit Chandra, Siddharth Sriraman, Harneet Singh Khanuja, Yiqiao Jin, and Munmun De Choudhury. 2025. Reasoning is not all you need: examining llms for multi-turn mental health conversations. arXiv preprint arXiv:2505.20201. find
  14. Siyuan Brandon Loh and Aravind Sesagiri Raamkumar. 2023. Harnessing large language models’ empathetic response generation capabilities for online mental health counselling support. arXiv preprint arXiv:2310.08017. — [arXiv:2310.08017](https://arxiv.org/abs/2310.08017) find
  15. Ha Na Cho, Jiayuan Wang, Di Hu, and Kai Zheng. 2026. Large language model–based chatbots and agentic ai for mental health counseling: Systematic review of methodologies, evaluation frameworks, and ethical safeguards. JMIR AI, 5(1):e80348. find
  16. Danilo Moggia, Wolfgang Lutz, Eva-Lotta Brakemeier, and Leonard Bickman. 2024. Treatment personalization and precision mental health care: Where are we and where do we want to go? Administration and Policy in Mental Health and Mental Health Services Research, 51(5):611–616. find
  17. Sarah C Cook, Ann C Schwartz, and Nadine J Kaslow. 2017. Evidence-based psychotherapy: Advantages and challenges. Neurotherapeutics, 14(3):537–545. find
  18. Ahmed Tamer Elboardy, Ziad Mohamed, Ghada Khoriba, Tamer Arafa, and Essam A Rashed. 2025. Bridging gender disparities in mental health: A bilingual large language model for multilingual therapeutic chatbots. In 2025 22nd International Learning and Technology Conference (L&T), volume 22, pages 251–25… find
  19. Viet Cuong Nguyen, Mohammad Taher, Dongwan Hong, Vinicius Konkolics Possobom, Vibha Thirunel-layi Gopalakrishnan, Ekta Raj, Zihang Li, Heather J Soled, Michael L Birnbaum, Srijan Kumar, and 1 others. 2024. Do large language models align with core mental health counseling competencies? arXiv preprint… find
  20. Zhuohan Ge, Nicole Hu, Darian Li, Yubo Wang, Shihao Qi, Yuming Xu, Han Shi, and Jason Zhang. 2025. A survey of large language models in mental health disorder detection on social media. In 2025 IEEE 41st International Conference on Data Engineering Workshops (ICDEW), pages 164–176. IEEE. find
  21. APA Presidential Task Force on Evidence-Based Practice and 1 others. 2006. Evidence-based practice in psychology. The American Psychologist, 61(4):271– 285. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/puspo-2026-role-context-llm-alignment-mental/