Elamur Research⌕
☰
home / papers / 2025 / CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy

CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy

Paper · 2025 · Mian Zhang, Xianjun Yang, Xinlu Zhang et al.

Summary

Demand for mental health care far exceeds the number of available professionals. This paper asks how well large language models could support therapists practicing cognitive behavioral therapy (CBT). The authors worked with clinical psychologists, social work professors and hired CBT experts to build CBT-Bench, a benchmark of five new datasets arranged in three tiers of increasing difficulty:

  1. Recalling CBT knowledge through multiple-choice questions.
  2. Analysing a patient's cognitive model by classifying cognitive distortions and core beliefs at two levels of granularity.
  3. Writing therapist-style replies to patient statements, using Deliberate Practice exercises as a privacy-friendly stand-in for real session transcripts.

Several popular LLMs were tested. Larger models answer knowledge questions well, but scaling brings little improvement in understanding patients' cognitive structures. All models struggle with fine-grained distortion and belief categories. In response generation, models tend to reason rigidly and challenge clients instead of respecting their autonomy and building rapport. These shortcomings are most visible when clients show resistance. The benchmark exposes concrete gaps that must be closed before AI can meaningfully assist psychotherapy.

Key points

  • Level I (CBT-QA): 220 test questions drawn from graduate CBT exams and expert-written items, with 50 more reserved for in-context examples. Human experts scored 90.7% accuracy.
  • Level II, three classification datasets:
    • CBT-CD: 146 cognitive distortion examples.
    • CBT-PC: 184 examples across 3 primary core belief categories.
    • CBT-FC: 112 examples across 19 fine-grained core belief categories.
  • Level III (CBT-DP): 156 Deliberate Practice exercises covering 10 aspects of CBT sessions at three difficulty levels. Model outputs were rated by experts against human reference responses.
  • Models tested: six LLMs were evaluated on Levels I–II, including GPT-4o, Llama-3.1-8B and Llama-3.1-405B. All models performed worst on the "Responding to Client Resistance" exercise.
  • Availability and ethics: the project was IRB-approved and used 8 US-based experts recruited via UpWork. Data and code are released on GitHub (mianzhang/CBT-Bench) for research use only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Princeton UniversityPrinceton · United States90 works · 39,109 citations🎓University of PittsburghPittsburgh · United States59 works · 22,324 citations🎓The University of Texas at DallasRichardson · United States61 works · 5,614 citations🎓University of California, Santa BarbaraSanta Barbara · United States57 works · 9,752 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 worksApplied Psychology · 21 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Mian Zhang, Xianjun Yang, Xinlu Zhang, Travis Labrum, Jamie C. Chiu, Shaun M. Eack, Fei Fang, William Yang Wang and Zhiyu Chen. 2025. CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.196

@inproceedings{zhang2025cbt,
  title = {{CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy}},
  author = {Mian Zhang, Xianjun Yang, Xinlu Zhang, Travis Labrum, Jamie C. Chiu, Shaun M. Eack, Fei Fang, William Yang Wang and Zhiyu Chen},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.196},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.196}
}

References

30 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achini Adikari, Daswin De Silva, Harsha Moraliyage, Damminda Alahakoon, Jiahui Wong, Mathew Gan-carz, Suja Chackochan, Bomi Park, Rachel Heo, and Yvonne Leung. 2022. Empathic conversational agents for real-time monitoring and co-facilitation of patient-centered healthcare. Future Generation Computer… find
  2. Dae-Young Kim, Rebecca Hwa, and Muhammad Mahbubur Rahman. 2024. MhGPT: A lightweight generative pre-trained transformer for mental health text analysis. arXiv [cs.CL]. find
  3. Willem Kuyken, Christine A Padesky, and Robert Dudley. 2011. Collaborative case conceptualization: Working effectively with clients in cognitive-behavioral therapy. Guilford Press. find
  4. Aaron T Beck and Emily AP Haigh. 2014. Advances in cognitive theory and therapy: The generic cognitive model. Annual review of clinical psychology, 10(1):1– 24. find
  5. June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023. ChatCounselor: A large language models for mental health support. arXiv [cs.CL]. find
  6. Judith S Beck. 2020. Cognitive behavior therapy: Basics and beyond. Guilford Publications. find
  7. Llama Team. 2024. The llama 3 herd of models. arXiv [cs.AI].
  8. James F Boswell and Michael J Constantino. 2022. Deliberate practice in cognitive behavioral therapy. American Psychological Association. find
  9. Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, and Diyi Yang. 2024. Roleplay-doh: Enabling domain-experts to create LLM-simulated patients via eliciting and adhering to principles.
  10. Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023a. LLM-empowered chatbots for psychiatrist and patient simulation: Application and evaluation. arXiv [cs.CL]. find
  11. Zhiyu Chen, Yujie Lu, and William Wang. 2023b. Empowering psychotherapy with large language models: Cognitive distortion detection through diagnosis of thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4295–4304. find
  12. Shuang Gao, Vince D Calhoun, and Jing Sui. 2018. Machine learning in major depression: From classification to treatment outcome prediction. CNS neuroscience & therapeutics, 24(11):1037–1052. find
  13. Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv [cs.CL]. find
  14. Declan Grabb, Max Lamparth, and Nina Vasan. 2024. Risks from language models for automated mental healthcare: Ethics and structure for implementation. arXiv [cs.CY]. find
  15. Shang-Ling Hsu, Raj Sanjay Shah, Prathik Senthil, Zahra Ashktorab, Casey Dugan, Werner Geyer, and Diyi Yang. 2023. Helping the helper: Supporting peer counselors via ai-empowered practice and feedback. arXiv preprint arXiv:2305.08982. find
  16. Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ananiadou, and Erik Cambria. 2023. Rethinking large language models in mental health applications. arXiv [cs.CL]. find
  17. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  18. Mounica Maddela, Megan Ung, Jing Xu, Andrea Madotto, Heather Foran, and Y-Lan Boureau. 2023. Training models to generate, recognize, and reframe unhelpful thoughts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Stroudsburg, PA, US… find
  19. Saskia Senn, ML Tlachac, Ricardo Flores, and Elke Rundensteiner. 2022. Ensembles of bert for depression classification. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 4691– 4694. IEEE. find
  20. Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. 2021. Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach. In Proceedings of the Web Conference 2021, pages 194–205. find
  21. Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. 2023a. Human–ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1):46–57. find
  22. Ashish Sharma, Kevin Rushton, Inna Lin, David Wadden, Khendra Lucas, Adam Miner, Theresa Nguyen, and Tim Althoff. 2023b. Cognitive reframing of negative thoughts through human-language model interaction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volu… find
  23. Ashish Sharma, Kevin Rushton, Inna Lin, David Wadden, Khendra Lucas, Adam Miner, Theresa Nguyen, and Tim Althoff. 2023c. Cognitive reframing of negative thoughts through human-language model interaction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volu… find
  24. Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, Theresa Nguyen, and Tim Althoff. 2024. Facilitating self-guided mental health interventions through human-language model interaction: A case study of cognitive restructuring. In Proceedings of the CHI Conference on Human Factors in Computing Systems, vo… find
  25. Sagarika Shreevastava and Peter Foltz. 2021a. Detecting cognitive distortions from patient-therapist interactions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  26. Sagarika Shreevastava and Peter Foltz. 2021b. Detecting cognitive distortions from patient-therapist interactions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 151–158. find
  27. Bichen Wang, Pengfei Deng, Yanyan Zhao, and Bing Qin. 2023. C2D2 dataset: A resource for the cognitive distortion analysis and its impact on mental health. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10149– 10160, Stroudsburg, PA, USA. Association for Computationa… find
  28. Ruiyi Wang, Stephanie Milani, Jamie C Chiu, Shaun M Eack, Travis Labrum, Samuel M Murphy, Nev Jones, Kate Hardy, Hong Shen, Fei Fang, et al. 2024. Patient-{\\Psi}: Using large language models to simulate patients for training mental health professionals. arXiv preprint arXiv:2405.19660. find
  29. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. arXiv [cs.CL]. find
  30. Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. MentaL-LaMA: Interpretable mental health analysis on social media with large language models. In Proceedings of the ACM on Web Conference 2024, volume 35 of WWW ’24, page 4489–4500. ACM. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-cbt-bench-evaluating-large-language/