Elamur Research⌕
☰
home / papers / 2026 / Mostly Grounded, Occasionally Risky: Expert Evaluation of LLM-Generated Supervisory Feedback in a Psychotherapy Training Simulator

Mostly Grounded, Occasionally Risky: Expert Evaluation of LLM-Generated Supervisory Feedback in a Psychotherapy Training Simulator

Paper · 2026 · Adrian Montesano, Justin K. Bloomberg and Marc Pérez-Buriel

Summary

AI-based psychotherapy training simulators often promote automatic feedback as a major benefit, but no one had tested whether LLM-written supervisory comments are actually supported by what happened in a free-form session. The authors examine PRACTICE, a web platform where novice trainees converse openly with an LLM-driven virtual patient. After every turn, the system updates how open, emotionally intense, collaborative and ready to change the patient is. When a trainee requests feedback, gpt-5 produces it through a RAG pipeline (LangChain) that draws on the Facilitative Interpersonal Skills (FIS) coding manual and a skills manual written for the platform. Four trainees generated 21 feedback instances in Spanish. These were split into 167 atomic propositions, and two experienced clinicians independently checked each one against the session transcripts. Most claims were grounded, but more than half of the feedback instances contained at least one problematic statement. Trainees nonetheless rated the feedback almost perfectly, so such errors could go unnoticed and influence real clinical habits. The authors call for traceable, expert-auditable feedback architectures and present their annotated Spanish data as an early resource toward that goal.

Key points

  • Coding and reliability: Two experts labelled 167 propositions as Justified, Unjustified or Unsure. Agreement was 98.2% (Cohen's κ = 0.902), and the few disagreements were borderline Justified-vs-Unsure cases.
  • Overall accuracy: 149 propositions (89.2%) were Justified, 9 were Unjustified and 6 were Unsure.
  • Errors per instance: 11 of 21 instances (52.4%) contained at least one non-justified claim. Most of these had only a single error.
  • Where and how errors occurred: Errors clustered in the Strengths component (15.5%, versus 4.0% in Area for Improvement). Three failure types were identified:
    • unsupported clinical inference (n = 8)
    • pragmatic failure (n = 4)
    • referential grounding error (n = 3), e.g., attributing the patient's words to the trainee
  • Trainee perception and limitations: Trainees rated feedback quality at a mean of 9.88/10. The study is a small pilot with no human-supervisor baseline and a text-only setting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Oberta de CatalunyaBarcelona · Spain3 works

Also on the byline: Universitat de Girona.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,772 worksPsychology · 1,552 worksSafety Research · 543 worksMedicine · 241 worksHealth Informatics · 184 worksClinical Psychology · 20 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Adrian Montesano, Justin K. Bloomberg and Marc Pérez-Buriel. 2026. Mostly Grounded, Occasionally Risky: Expert Evaluation of LLM-Generated Supervisory Feedback in a Psychotherapy Training Simulator. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.clpsych-1.24

@inproceedings{montesano2026mostly,
  title = {{Mostly Grounded, Occasionally Risky: Expert Evaluation of LLM-Generated Supervisory Feedback in a Psychotherapy Training Simulator}},
  author = {Adrian Montesano, Justin K. Bloomberg and Marc Pérez-Buriel},
  booktitle = {Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.clpsych-1.24},
  url = {https://doi.org/10.18653/v1/2026.clpsych-1.24}
}

References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Alsahafi, D. L. X. Ling, M. Newell, and T. Krop-mans. 2023. A systematic review of effective quality feedback measurement tools used in clinical skills assessment. MedEdPublish, 12(11). find
  2. T. Anderson, M. E. J. Crowley, L. Himawan, J. K. Holmberg, and B. D. Uhlin. 2016. Therapist facilitative interpersonal skills and training status: A randomized clinical trial on alliance and outcome. Psychotherapy Research, 26(5):511–529. find
  3. T. Anderson, B. M. Ogles, C. L. Patterson, M. J. Lambert, and D. A. Vermeersch. 2009. Facilitative interpersonal skills coding manual. Ohio University. find
  4. E. T. Beeson, E. Counselman-Carpenter, T. A. Field, and M. Ghoston. 2025. Artificial intelligence role-play for counselor skill development: A qualitative investigation. Counselor Education and Supervision. find
  5. J. M. Bernard and R. K. Goodyear. 2019. Fundamentals of Clinical Supervision, 6 edition. Pearson. find
  6. D. Cabrera Lozoya, E. Hernandez Lua, J. A. Barajas Perches, M. Conway, and S. D’Alfonso. 2025. Synthetic empathy: Generating and evaluating artificial psychotherapy dialogues to detect empathy in counseling sessions. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psych… find
  7. D. L. Chow, S. D. Miller, J. A. Seidel, R. T. Kane, J. A. Thornton, and W. P. Andrews. 2015. The role of deliberate practice in the development of highly effective psychotherapists. Psychotherapy, 52(3):337– 345. find
  8. A. C. Del Re, C. Flückiger, A. O. Horvath, and B. E. Wampold. 2021. Examining therapist effects in the alliance-outcome relationship: A multilevel meta-analysis. Journal of Consulting and Clinical Psychology, 89(5):371–378. find
  9. C. Flückiger, A. C. Del Re, B. E. Wampold, and A. O. Horvath. 2018. The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy, 55(4):316–340. find
  10. C. Hannan, M. J. Lambert, C. Harmon, S. L. Nielsen, D. W. Smart, K. Shimokawa, and S. W. Sutton. 2005. A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61(2):155–163. find
  11. D. Hatfield, L. McCullough, S. H. B. Frantz, and K. Krieger. 2010. Do we know when our clients get worse? an investigation of therapists’ ability to detect negative client change. Clinical Psychology & Psychotherapy, 17(1):25–32. find
  12. C. E. Johnson, J. L. Keating, M. Leech, P. Congdon, F. Kent, M. K. Farlie, and E. K. Molloy. 2021. Development of the feedback quality instrument: A guide for health professional educators in fostering learner-centred discussions. BMC Medical Education, 21(1):382. find
  13. S. L. Kopelovich, K. Doud, M. Ofer, and B. Stiles. 2025a. Development and validation of a machine learning system for rating CBT for psychosis competencies. Psychotherapy, 62(1):1–11. find
  14. S. L. Kopelovich, K. Doud, M. Ofer, and B. Stiles. 2025b. Trainee and supervisor perspectives on ai-generated feedback in CBT for psychosis training. Psychotherapy, 62(1):12–21. find
  15. J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  16. D. Milne. 2007. An empirical definition of clinical supervision. British Journal of Clinical Psychology, 46(4):437–447. find
  17. F. Schmidt, K. Hammerfald, H. H. Jahren, and V. Vlassov. 2025. CFiCS: Graph-based classification of common factors and microcounseling skills. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025), pages 106–115. Association for Computational Linguis… find
  18. S. H. Stige, A. Montesano, and K. Aafjes-van Doorn. under review. Use of generative AI as a path to strengthen clinical training: First steps and emerging challenges for active learning. Active Learning in Higher Education. Under review. find
  19. Z. Tian, L. Yang, and W. Liang. 2025. Ai-simulated clients in counseling training: Effects on self-efficacy and skill development. Training and Education in Professional Psychology. find
  20. L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1– 55.
  21. T. J. G. Tracey, B. E. Wampold, J. W. Lichtenberg, and R. K. Goodyear. 2014. Expertise in psychotherapy: An elusive goal? American Psychologist, 69(3):218– 229. find
  22. H. Jeong, S. Lee, and B. K. Sohn. 2025. Exploring ai-based role-play simulation for mental health counseling training. Psychiatric Services. find
  23. T. Tseriotou, J. Chim, A. Klein, A. Shamir, G. Dvir, I. Ali, C. Kennedy, G. S. Kohli, A. Hills, A. Zirikly, D. Atzil-Slonim, and M. Liakata. 2025. Overview of the CLPsych 2025 shared task: Capturing mental health dynamics from social media timelines. In Proceedings of the 10th Workshop on Computatio… find
  24. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.
  25. S. Walfish, B. McAlister, P. O’Donnell, and M. J. Lambert. 2012. An investigation of self-assessment bias in mental health providers. Psychological Reports, 110(2):639–644. find
  26. Z. E. Imel, M. Steyvers, and D. C. Atkins. 2024. Computational psychotherapy research: Current status and future directions. Behaviour Research and Therapy, 172:104430. 6.“Avoid quickly resorting to exercise as a solution. Instead, focus more on their language and motivation (e.g., ‘turn down the vo… find
  27. B. E. Wampold and J. Owen. 2021. Therapist effects: History, methods, magnitude, and characteristics of effective therapists. In M. Barkham, W. Lutz, and L. G. Castonguay, editors, Bergin and Garfield’s handbook of psychotherapy and behavior change, 7th edition, pages 297–326. Wiley. find
  28. S. Yosef, M. Zisquit, B. Cohen, A. Brunstein Klomek, K. Bar, and D. Friedman. 2024. Assessing motivational interviewing sessions with ai-generated patient simulations. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 1–11. Association for… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/montesano-2026-mostly-grounded-occasionally-risky-expert/