Elamur Research⌕
☰
home / papers / 2026 / Preference Learning Unlocks LLMs' Psycho-Counseling Skills

Preference Learning Unlocks LLMs' Psycho-Counseling Skills

Paper · 2026 · Mian Zhang, Shaun M. Eack and Zhiyu Chen

Summary

The paper addresses a central gap in LLM-based mental-health support. Models often answer clients inconsistently, and real counseling transcripts that could guide training are rarely available because of privacy. Their quality also varies from therapist to therapist. Together with social-work and psychiatry experts, the authors defined seven criteria for judging a counselor's reply. Four are basic (empathy, relevance, clarity, safety) and three are client-centered (encouraging self-exploration, supporting autonomy, and recognizing the client's stage of change). They gathered client messages from several public sources, collected replies from a pool of 20 LLMs, and had GPT-4o rate each reply against the criteria. Pairs of replies with clearly different scores became PsyCoPref, a preference dataset of about 36k comparisons. They trained reward models and applied both offline and online preference optimization. The resulting PsyCo-Llama3-8B is preferred over GPT-4o in most test comparisons. The work shows that expert-informed synthetic preference data can substitute for scarce real sessions. The released data and models are intended to help therapists draft responses, not to replace them.

Key points

  • Dataset scope: PsyCoPref is built from 26,483 de-duplicated client speeches of 100–1,000 characters. They cover 8 coarse and 42 fine-grained topics, with 23,192 used for training and 3,291 held out for testing.
  • Pair construction: Four models are drawn at random from the 20-LLM pool (including the hybrid AI21-Jamba-1.5-Mini) to answer each speech. GPT-4o scores every reply on a 1–5 scale for each of the seven principles, and training pairs require an average-score gap of at least 1.
  • Expert validation: Professional psychotherapists checked a sample of the data and agreed with the preference labels in 88.5% of cases.
  • Main result: PsyCo-Llama3-8B achieves an 87% win rate against GPT-4o. Online preference learning that uses fresh generations from the current policy outperformed offline training, and reward models trained on PsyCoPref beat earlier general-purpose reward models.
  • Release and limitations: Data and models are available at huggingface.co/Psychotherapy-LLM. The dataset covers only single-turn exchanges, and reward hacking remains an open problem.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PittsburghPittsburgh · United States59 works · 22,324 citations🎓The University of Texas at DallasRichardson · United States61 works · 5,614 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Mian Zhang, Shaun M. Eack and Zhiyu Chen. 2026. Preference Learning Unlocks LLMs' Psycho-Counseling Skills. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.521

@inproceedings{zhang2026preference,
  title = {{Preference Learning Unlocks LLMs' Psycho-Counseling Skills}},
  author = {Mian Zhang, Shaun M. Eack and Zhiyu Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.521},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.521}
}

References

44 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, and 32 others. 20… — Constitutional AI: Harmlessness from AI Feedback
  4. Andrew P Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit., 30(7):1145–1159. find
  5. Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li, Yongqiang Liu, Haihui Pan, Leichao Xu, Dacheng Zhang, Zhipeng Zhang, and Kun Han. 2024. Orion-14B: Open-source multilingual large language models. arXiv [cs.CL].
  6. Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023a. LLM-empowered chatbots for psychiatrist and patient simulation: Application and evaluation. arXiv [cs.CL]. find
  7. Zhiyu Chen, Yujie Lu, and William Yang Wang. 2023b. Empowering psychotherapy with large language models: Cognitive distortion detection through diagnosis of thought prompting. arXiv [cs.CL]. find
  8. Neo Christopher Chung, George Dyer, and Lennart Brocki. 2023. Challenges of large language models for mental health counseling. arXiv [cs.CL]. find
  9. DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, and 68 others. 2024. DeepSeek LLM: Scaling open-source language models wit… find
  10. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z F Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs…
  11. Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv [cs.CL]. find
  12. Gerard E Hogarty. 2002. Personal therapy for schizophrenia and related disorders: A guide to individualized treatment. Guilford Press. find
  13. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, and 6 others. 2024. MiniCPM: Unveiling the potential of small lan…
  14. Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, and 42 others. 2024. Jamba-1.5: Hybri…
  15. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L J Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A Smith, and Hannaneh Hajishirzi. 2024. Reward-Bench: Evaluating reward models for language modeling. arXiv [cs.LG]. find
  16. Yeonji Lee, Sangjun Park, Kyunghyun Cho, and Jinyeong Bak. 2024. MentalAgora: A gateway to advanced personalized care in mental health through multi-agent debating and attribute control. arXiv [cs.CL]. find
  17. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in LLMs. arXiv [cs.AI].
  18. June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023. ChatCounselor: A large language models for mental health support. arXiv [cs.CL]. find
  19. Llama Team. 2024. The llama 3 herd of models. arXiv [cs.AI].
  20. Mounica Maddela, Megan Ung, Jing Xu, Andrea Madotto, Heather Foran, and Y-Lan Boureau. 2023. Training models to generate, recognize, and reframe unhelpful thoughts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Stroudsburg, PA, US… find
  21. Luke Marks, Amir Abdullah, Clement Neo, Rauno Arike, Philip Torr, and Fazl Barez. 2023. Beyond training objectives: Interpreting reward model divergence in large language models. arXiv [cs.LG]. find
  22. Miller and William Stephen Rollnick. Motivational interviewing: Helping people change. Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden,… find
  23. Hongbin Na, Yining Hua, Zimu Wang, Tao Shen, Beibei Yu, Lilin Wang, Wei Wang, John Torous, and Ling Chen. 2025. A survey of large language models in psychotherapy: Current landscape and future directions. arXiv [cs.CL]. find
  24. Mahdi Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. Proc. Conf. AAAI Artif. Intell., 2015:2901–2907. find
  25. OpenAI. 2024. GPT-4o system card. arXiv [cs.CL].
  26. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E Miller, Maddie Simens, Amanda Askell, P Welinder, P Christiano, J Leike, and Ryan J Lowe. 2022. Training la…
  27. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. arXiv [cs.CL].
  28. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2024. Qwen2.5 technical report. arXiv [cs.CL]. find
  29. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. 2024. Scaling laws for reward model overoptimization in direct alignment algorithms. find
  30. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv [cs.LG].
  31. Diego Rocco, Alessandro Gennaro, Lorena Filugelli, Patrizia Squarcina, and Elena Antonelli. 2019. Key factors in psychotherapy training: an analysis of trainers’, trainees’ and psychotherapists’ points of view. Res. Psychother. Psychopathol. Process Outcome, 22(3). find
  32. Glenda Dewberry Rooney, Ronald H Rooney, Dean H Hepworth, and Kim Strom-Gottfried. 2017. Direct social work practice: Theory and skills. Cengage Learning. find
  33. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv [cs.LG]. find
  34. Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, Theresa Nguyen, and Tim Althoff. 2024. Facilitating self-guided mental health interventions through human-language model interaction: A case study of cognitive restructuring. In Proceedings of the CHI Conference on Human Factors in Computing Systems, vo… find
  35. Sagarika Shreevastava and Peter Foltz. 2021. Detecting cognitive distortions from patient-therapist interactions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  36. Elizabeth C Stade, Shannon Wiltsey Stirman, Lyle H Ungar, Cody L Boland, H Andrew Schwartz, David B Yaden, João Sedoc, Robert J DeRubeis, Robb Willer, and Johannes C Eichstaedt. 2024. Large language models could change the future of behavioral healthcare: a proposal for responsible development and e… find
  37. Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. 2024. Understanding the performance gap between online and offline alignment algorithms. arXiv [cs.LG]. find
  38. Kristine D Tower. 1994. Consumer-centered social work practice: Restoring client self-determination. Social Work, 39(2):191–196. find
  39. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024. HelpSteer2-preference: Complementing ratings with preferences. arXiv [cs.LG]. find
  40. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024. Is DPO superior to PPO for LLM alignment? a comprehensive study. arXiv [cs.CL]. find
  41. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, and 36 others. 2023. Baichuan 2: Open large-scale language models. arXiv [cs.CL].
  42. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. Qwen2 technical report. arXiv [cs.CL]. find
  43. Mian Zhang, Xianjun Yang, Xinlu Zhang, Travis Labrum, Jamie C Chiu, Shaun M Eack, Fei Fang, William Yang Wang, and Zhiyu Zoey Chen. 2024. CBT-bench: Evaluating large language models on assisting cognitive behavior therapy. arXiv [cs.CL].
  44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv [cs.CL].

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-preference-learning-unlocks-llms-psycho/