Elamur Research⌕
☰
home / papers / 2026 / The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7

The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7

Paper · 2026 · Qian Shen and Yu Han

Summary

LLMs are increasingly used to draft new psychological items and questionnaires, but validating these instruments with human respondents is expensive. This paper asks whether open-weight LLMs can stand in for people as synthetic respondents on two clinical screening scales, GAD-7 (anxiety) and PHQ-9 (depression). Four models answered both questionnaires many times. Generation varied three things: a severity persona (healthy, mild or severe), a randomly added situational cue such as poor sleep or a stressful event, and the prompt style (Standard or Calibrated). Two sampling temperatures were also used. The synthetic answers were compared with human survey data from NHIS and NHANES using multi-group CFA and ordinal logistic DIF. The models gave very internally consistent answers, often more consistent than humans. However, their factor structure departed from the human one, scalar invariance did not hold, and item-level bias was widespread. The authors call this a "reliability illusion": high Cronbach's α can hide structural invalidity. They argue that measurement invariance, not reliability, should be the benchmark for synthetic psychometric data.

Key points

  • Models tested: Llama-3.1-8B-Instruct, Mistral-7B-Instruct, Llama-3.3-70B-Instruct and GPT-OSS-120B.
  • Human baselines: GAD-7 data from the 2019 NHIS and PHQ-9 data from NHANES 2017–2019, with incomplete cases removed.
  • Generation design:
    • 3 personas × 2 prompt conditions × 2 temperatures (0.7 and 1.0), giving 12 conditions.
    • 500 generations per condition, yielding 6,000 valid responses per model per scale.
    • Answers were constrained to strict JSON with ordinal 0–3 scores.
  • Synthetic data reached Cronbach's α of 0.86–0.96, yet showed severe structural misfit, failed scalar invariance, extensive DIF and strong sensitivity to prompt wording.
  • The authors attribute the pattern to stereotype-based matching of symptom wording rather than a coherent latent construct. Stated limitations include:
    • only open-weight models were tested;
    • the simulated severity mix was roughly balanced, unlike real populations;
    • no ablation of individual design components;
    • a single human sample was used for comparison.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of FloridaGainesville · United States33 works · 364 citations

Also on the byline: University of Mississippi.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Psychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Qian Shen and Yu Han. 2026. The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.clpsych-1.7

@inproceedings{shen2026reliability,
  title = {{The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7}},
  author = {Qian Shen and Yu Han},
  booktitle = {Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.clpsych-1.7},
  url = {https://doi.org/10.18653/v1/2026.clpsych-1.7}
}

References

33 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gregorio Ferreira, Jacopo Amidei, Rubén Nieto, and Andreas Kaltenbrunner. 2025. Matching gpt-simulated populations with real ones in psychological studies—the case of the epqr-a personality test. ACM Transactions on Computing for Healthcare, 6(2):1– 33. find
  2. Divya Mani Adhikari, Alexander Hartland, Ingmar Weber, and Vikram Kamath Cannanure. 2025. Exploring llms for automated generation and adaptation of questionnaires. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, pages 1–23. find
  3. Silvia García-Méndez, Francisco de Arriba-Pérez, Julen Beiro-Suso, and Francisco J González-Castaño. 2026. Real-time anxiety and depression detection by combining large language models and machine learning with explainability capabilities on a user-centric, engaging conversational assistant. Expert… find
  4. Michael F. Armey, Heather T. Schatten, Natasha Harad-hvala, and Ivan W. Miller. 2015. Ecological momentary assessment (ema) of depression-related phenomena. Current Opinion in Psychology, 4:21–25. find
  5. Fabiana Battista, Tiziana Lanciano, Raffaella Maria Ri-batti, and Antonietta Curci. 2026. Malingering depression: a comparative study of human and gpt-3.5 performance. Current Psychology, 45(4):379. find
  6. Aura-Nidia Herrera and Juana Gómez. 2008. Influence of equal or unequal comparison group sample sizes on the detection of differential item functioning using the mantel–haenszel and logistic regression techniques. Quality & Quantity, 42(6):739–755. find
  7. Renzo Bianchi, Jay Verkuilen, Sharon Toker, Irvin Sam Schonfeld, Markus Gerber, Elmar Braehler, and Kurt Li-tze Hu and Peter M Bentler. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidiscipli… find
  8. Sriya Mantena, Anders Johnson, Marily Oppezzo, Narayan Schütz, Alexander Tolas, Ritu Doijad, C Mikael Mattson, Allan Lawrie, Mariana Ramirez-Posada, Paul Schmiedmayer, and 1 others. 2025. Fine-tuning llms in behavioral psychology for scalable health coaching. NPJ Cardiovascular Health, 2(1):48. find
  9. Hyolim Jeon, Dongje Yoo, Daeun Lee, Sejung Son, Seungbae Kim, and Jinyoung Han. 2024. A dual-prompting for interpretable mental health language models. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 247–255, St. Julians, Malta. Associati… find
  10. Summer Mengelkoch, Daniel P. Moriarity, Anne M. Novak, Michael P. Snyder, George M. Slavich, and Shahar Lev-Ari. 2024. Using ecological momentary assessments to study how daily fluctuations in psychological states impact stress, well-being, and health. Journal of Clinical Medicine, 13(1):24. find
  11. Laura J. Julian. 2011. Measures of anxiety: State-trait anxiety inventory (stai), beck anxiety inventory (bai), and hospital anxiety and depression scale-anxiety (hads-a). Arthritis Care & Research, 63(S11):S467– S472. find
  12. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. find
  13. Kurt Kroenke, Robert L Spitzer, and Janet BW Williams. 2001. The phq-9: validity of a brief depression severity measure. Journal of general internal medicine, 16(9):606–613. find
  14. Adrianos Pavlopoulos, Theodoros Rachiotis, and Ilias Maglogiannis. 2024. An overview of tools and technologies for anxiety and depression management using ai. Applied Sciences, 14(19):9068. find
  15. Hannah R Lawrence, Renee A Schneider, Susan B Rubin, Maja J Matarić, Daniel J McDuff, and Megan Jones Bell. 2024. The opportunities and risks of large language models in mental health. JMIR Mental Health, 11(1):e59479. find
  16. Diane L Putnick and Marc H Bornstein. 2016. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental review, 41:71–90. find
  17. Chihao Li and Yue Qi. 2025. Toward accurate psychological simulations: Investigating llms’ responses to personality and cultural variables. Computers in human behavior, 170:108687. find
  18. Aadesh Salecha, Molly E Ireland, Shashanka Subrahmanya, João Sedoc, Lyle H Ungar, and Johannes C Eichstaedt. 2024. Large language models display human-like social desirability biases in big five personality surveys. PNAS nexus, 3(12):pgae533. find
  19. Hauke Licht, Rupak Sarkar, Patrick Y Wu, Pranav Goel, Niklas Stoehr, Elliott Ash, and Alexander Miserlis Hoyle. 2025. Measuring scalar constructs in social science with llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32132–32159. find
  20. Qian Shen, Fanghua Cao, Min Yao, Shlok Gilda, Bonnie J Dorr, and Walter L Leite. 2026. Children’s english reading story generation via supervised fine-tuning of compact llms with controllable difficulty and safety. arXiv preprint arXiv:2605.13709.
  21. June M Liu, Mengxia Gao, Sahand Sabour, Zhuang Chen, Minlie Huang, and Tatia MC Lee. 2025a. Enhanced large language models for effective screening of depression and anxiety. Communications Medicine, 5(1):457. find
  22. Robert L Spitzer, Kurt Kroenke, Janet BW Williams, and Bernd Löwe. 2006. A brief measure for assessing generalized anxiety disorder: the gad-7. Archives of internal medicine, 166(10):1092–1097. find
  23. Yunting Liu, Shreya Bhandari, and Zachary A Pardos. 2025b. Leveraging llm respondents for item evaluation: A psychometric analysis. British Journal of Educational Technology, 56(3):1028–1052. find
  24. Bin Tan, Nour Armoush, Elisabetta Mazzullo, Okan Bulut, and Mark Gierl. 2025. A review of automatic item generation techniques leveraging large language models. International Journal of Assessment Tools in Education, 12(2):317–340. find
  25. Bernd Löwe, Oliver Decker, Stefanie Müller, Elmar Brähler, Dieter Schellberg, Wolfgang Herzog, and Philipp Yorck Herzberg. 2008. Validation and standardization of the generalized anxiety disorder screener (gad-7) in the general population. Medical care, 46(3):266–274. find
  26. Yongfeng Tao, Minqiang Yang, Hao Shen, Zhichao Yang, Ziru Weng, and Bin Hu. 2023. Classifying anxiety and depression through llms virtual interactions: a case study with chatgpt. In 2023 IEEE international conference on bioinformatics and biomedicine (BIBM), pages 2259–2264. IEEE. find
  27. Erfan Loweimi, Sofia de la Fuente Garcia, and Saturnino Luz. 2026. Predicting psychological well-being from spontaneous speech using llms. arXiv preprint arXiv:2605.11303. — [arXiv:2605.11303](https://arxiv.org/abs/2605.11303) find
  28. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Surabhi Adhikari, Hariram Veeramani, Mehwish Nasim, and Usman Naseem. 2025. Large language models (llm) in computational social science: prospects, current state, and challenges. Social Network Analysis and Mining, 15(1):4. find
  29. Robert J Vandenberg and Charles E Lance. 2000. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational research methods, 3(1):4–70. find
  30. Bushi Xiao and Qian Shen. 2026. Personality-driven student agent-based modeling in mathematics education: How well do student agents align with human learners? arXiv preprint arXiv:2603.21358. — [arXiv:2603.21358](https://arxiv.org/abs/2603.21358) find
  31. Shihao Xu, Yiming Yan, Yanli Ding, Feng Li, Shu Zhang, Haoyun Tang, Chao Luo, Yan Li, Hao Liu, Yu Mei, and 1 others. 2025. Identifying psychiatric manifestations in outpatients with depression and anxiety: a large language model-based approach. npj Mental Health Research, 4(1):63. find
  32. Yuqing Zhao, Wei Qian, Yaru Chen, Donghong Wu, Yujia Luo, Cong Gao, Kankan Wu, and Zhengkui Liu. 2025. Effect of an ai agent trained on a large language model (llm) as an intervention for depression and anxiety symptoms in young adults: A 28-day randomized controlled trial. Applied Psychology: Healt… find
  33. Bruno D Zumbo. 1999. A handbook on the theory and methods of differential item functioning (dif). Ottawa: National Defense Headquarters, 160:53. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-reliability-illusion-synthetic-patients-psychometric/