Elamur Research⌕
☰
home / papers / 2026 / GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing

GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing

Paper · 2026 · Ming Wang, Shuang Wu, Bixuan Wang et al.

Summary

Psychological profiling of persona-conditioned LLM agents (PC-Agents) usually relies on self-report questionnaires such as BFI, BDI or MBTI. This approach has two weaknesses. The items have probably been seen during pretraining, and model answers drift toward socially acceptable responses. The authors propose GenPT, which adapts three projective techniques (TAT, Rorschach and SCT) using freshly generated, expert-reviewed stimuli: scenes, inkblots and sentence stems. Assessment runs in three steps. An Examinee agent gives free-form responses, an LLM Interpreter converts them into scores based on clinical rubrics (e.g., SCORS-G, Exner-style coding), and a Diagnostician aggregates these scores into trait or risk labels. Agents built from CharacterRAG and AnnaAgent profiles were tested on Big Five, MBTI, depression and suicide ideation, with GenPT compared against classical questionnaires. Questionnaire scores moved in a consistent direction under social-desirability framing, while GenPT stayed close to neutral. GenPT was also far more responsive to longitudinal counselling context. However, questionnaires remained slightly better on clean personality-trait tasks. The work positions projective testing as a contamination-resistant complement to self-report.

Key points

  • The three-stage pipeline (Examinee → Interpreter → Diagnostician) is evaluated with three small-to-mid open-weights backbones: Qwen3-8B, Phi-4-mini-reasoning (3.84B) and Intern-S1-mini (~8B).
  • Criterion validity was mixed:
    • On personality tasks, the questionnaire baseline won (Big Five accuracy 0.37 vs. 0.24–0.33; MBTI mean Hamming distance 0.73 vs. 1.20–2.20).
    • On depression, GenPT scored 0.20–0.40 accuracy vs. 0.13 for the questionnaire.
  • Under social-desirability framing, questionnaires showed systematic directional bias, strongest for suicide ideation, while GenPT stayed near the symmetric baseline.
  • In a longitudinal counselling setting with Qwen3, the GenPT depression score shifted roughly ten times more than the questionnaire score.
  • Code and stimuli are released at github.com/sci-m-wang/GenPT. The authors list higher compute cost and limited model and cultural coverage as limitations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 worksApplied Psychology · 21 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ming Wang, Shuang Wu, Bixuan Wang, Lu Lin, Yuxin Chen, Xiaocui Yang, Daling Wang, Shi Feng, Yifei Zhang and Yufan Sun. 2026. GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1901

@inproceedings{wang2026genpt,
  title = {{GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing}},
  author = {Ming Wang, Shuang Wu, Bixuan Wang, Lu Lin, Yuxin Chen, Xiaocui Yang, Daling Wang, Shi Feng, Yifei Zhang and Yufan Sun},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1901},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1901}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranav Bhandari, Usman Naseem, Amitava Datta, Nicolas Fay, and Mehwish Nasim. 2025. Evaluating personality traits in large language models: Insights from psychological questionnaires. In Companion Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025 -2 May 2… find
  2. Black Forest Labs. 2025. FLUX.1-dev. find
  3. Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. CoRR, abs/2406.20094.
  4. Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. 2024. From persona to personalization: A survey on role-playing language agents.
  5. Trans. Mach. Learn. Res., 2024. find
  6. Jiaqi Chen, Ming Wang, Tingna Xie, Shi Feng, and Yongkang Liu. 2026. A systematic analysis of the impact of persona steering on llm capabilities. Preprint, arXiv:2604.11048. — [arXiv:2604.11048](https://arxiv.org/abs/2604.11048) find
  7. John M Digman. 1990. Personality structure: Emergence of the five-factor model. Annual review of psychology, 41(1):417–440. find
  8. Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023. Towards measuring t…
  9. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, and 1 others. 2024. Scaling rectified flow transformers for high-resolution image synthesis. Preprint, arXiv:2403.03206. — [arXiv:2403.03206](https://arxiv.org/abs/2403.03206) find
  10. John E. Exner Jr. 1993a. The Rorschach: A comprehensive system. Wiley series in personality processes. Place: Oxford, England Publisher: John Wiley & Sons. find
  11. John E. Exner Jr. 1993b. The Rorschach: A comprehensive system: Basic foundations. The Rorschach: A comprehensive system: Basic foundations, Vol. 1, 3rd ed. John Wiley & Sons, Oxford, England. Pages: xxiii, 642. find
  12. Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, and 1 others. 2025. Syceval: Evaluating llm sycophancy. Preprint, arXiv:2502.08177. — [arXiv:2502.08177](https://arxiv.org/abs/2502.08177) find
  13. Lawrence K. Frank. 1939. Projective methods for the study of personality. The Journal of Psychology, 8(2):389–413. find
  14. Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations.
  15. LR Goldberg. 1990. An alternative "description of personality": the big-five factor structure. Journal of personality and social psychology, 59(6):1216—1229. find
  16. Ankur Joshi, Saket Kale, and Satish Chandel. 2015. Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396. find
  17. Leon O. H. Kroczek, Alexander May, Selina Het-tenkofer, Andreas Ruider, Bernd Ludwig, and Andreas Mühlberger. 2025. The influence of persona and conversational task on social interactions with a llm-controlled embodied conversational agent. Comput. Hum. Behav., 172:108759. find
  18. Kun Li, Chenwei Dai, Wei Zhou, and Songlin Hu. 2025a. APEE: assessing the personality expressions of llm-driven role play agent beyond self-perception. In 28th International Conference on Computer Supported Cooperative Work in Design, CSCWD 2025, Compiegne, France, May 5-7, 2025, pages 1406– 1411. find
  19. Yuan Li, Yue Huang, Hongyi Wang, Ying Cheng, Xiangliang Zhang, James Zou, and Lichao Sun. 2025b. Evaluating large language models with psychometrics. In Large Language Models for Scientific and Societal Advances. find
  20. CHRISTIANA D. MORGAN and HENRY A. MUR-RAY. 1935. A METHOD FOR INVESTIGAT-ING FANTASIES: THE THEMATIC APPERCEP-TION TEST. Archives of Neurology & Psychiatry, 34(2):289–306. find
  21. Jeiyoon Park, Yongshin Han, Minseop Kim, and Kisu Yang. 2025. Dynamic context adaptation for consistent role-playing agents with retrieval-augmented generations. CoRR, abs/2508.02016. — [arXiv:2508.02016](https://arxiv.org/abs/2508.02016) find
  22. PDB Community. 2022. Database. find
  23. PDB: The Personality Changyong Qi, Longwei Zheng, Anna He, Haoxin Xu, Linzhao Jia, Yuang Wei, Bingqian Jiang, and Xiaoqing Gu. 2026. Simulating student learning behaviors with llm-based role-playing agents: A data-driven and cognitively inspired framework. Expert Systems with Applications, 304:13075… find
  24. Hermann Rorschach. 1922. Psychodiagnostik. The Journal of Nervous and Mental Disease, 56(3). find
  25. Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja J. Mataric. 2023. Personality traits in large language models. CoRR, abs/2307.00184.
  26. Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2025. Personality traits in large language models. Preprint, arXiv:2307.00184.
  27. Woojung Song, Dongmin Choi, Yoonah Park, Jongwook Han, and Yohan Jo. 2026. Human psychometric questionnaires mischaracterize llm psychology: Evidence from generation behavior. Preprint, arXiv:2509.10078. — [arXiv:2509.10078](https://arxiv.org/abs/2509.10078) find
  28. George Stricker and Bede J Healey. 1990. Projective assessment of object relations: A review of the empirical literature. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 2(3):219. find
  29. Ming Wang, Peidong Wang, Lin Wu, Xiaocui Yang, and 1 others. 2025a. Annaagent: Dynamic evolution agent system with multi-session memory for realistic seeker simulation. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 23221–2323… find
  30. Noah Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. 2024a. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of la…
  31. Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, Wei Wang, and Yanghua Xiao. 2025b. Coser: Coordinating llm-based persona simulation of established roles. In Forty-second International Conference on Machine Learning, ICML… find
  32. Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. 2024b. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Proceedings of the 62nd Annual…
  33. Drew Westen. 1991. Social cognition and object relations. Psychological Bulletin, 109:429–455. find
  34. Binwei Yao, Chao Shi, Likai Zou, Lingfeng Dai, and 1 others. 2022. D4: a Chinese dialogue dataset for depression-diagnosis-oriented chat. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2438–2459, Abu Dhabi, United Arab Emirates. Association for Compu… find
  35. Haoran Ye, Jing Jin, Yuhang Xie, Xin Zhang, and Guojie Song. 2025. Large language model psychometrics: A systematic review of evaluation, validation, and enhancement. Preprint, arXiv:2505.08245. — [arXiv:2505.08245](https://arxiv.org/abs/2505.08245) find
  36. Xi Zheng, Zhuoyang Li, Xinning Gui, and Yuhan Luo. 2025. Customizing emotional support: How do individuals construct and interact with llm-powered chatbots. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025, Yokohama, Japan, 26 April 2025- 1 May 2025, pages 37… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-genpt-beyond-self-report-reliable/