Elamur Research⌕
☰
home / papers / 2024 / InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

Paper · 2024 · Xintao Wang, Yunze Xiao, Jen-tse Huang et al.

Summary

The paper looks at how to check whether role-playing agents (RPAs) built on large language models capture the personality of the character they imitate. Earlier evaluations focused on a character's knowledge and speaking style. Those evaluations need a separate dataset for every character, and they do not test how the agent thinks. The authors measure "personality fidelity" with established psychological scales instead. Asking agents to fill in questionnaires directly works poorly: agents often step out of character or pick options that contradict how they actually behave. So the proposed framework, InCharacter, runs the test as an interview in two steps:

  1. Scale items are rewritten as open-ended questions, and each question is asked in a fresh context.
  2. An LLM scores the answers, either by mapping them to Likert options or by judging traits the way a psychologist would.

Across 32 characters and 14 scales, InCharacter measures personality more reliably than self-report prompting. It shows that state-of-the-art RPAs largely match how humans perceive the characters. This gives a way to evaluate character fidelity that does not depend on character-specific test sets.

Key points

  • There are two assessment variants: option conversion (OC) maps answers to Likert options, and expert rating (ER) has an LLM judge personality the way a psychologist would.
  • The benchmark covers 32 characters and 14 scales, including the Big Five Inventory (BFI), 16Personalities (16P) and the Dark Triad Dirty Dozen (DTDD).
  • Reference labels for BFI and 16P come from the Personality Database (PDb). Labels for the other scales were collected from annotators familiar with the characters.
  • The best RPAs agree with the human-perceived character personalities with accuracy reaching 80.7%.
  • The authors release 18,304 interview dialogues and suggest them as fine-tuning data for RPA foundation models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Fudan UniversityShanghai · China65 works · 19,402 citations🎓Renmin University of ChinaBeijing · China35 works · 14,757 citations🎓Chinese University of Hong KongHong Kong · China53 works · 12,921 citations🎓Boston UniversityBoston · United States10 works · 3,261 citations🏢SensetimeBeijing · China7 works · 1,846 citations🎓Beijing University of TechnologyBeijing · China4 works · 1,819 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Social Psychology · 129 works

Cite

Cite as Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li and Yanghua Xiao. 2024. InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.102

@inproceedings{wang2024incharacter,
  title = {{InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews}},
  author = {Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li and Yanghua Xiao},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.102},
  url = {https://doi.org/10.18653/v1/2024.acl-long.102}
}

References

50 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. ArXiv preprint, abs/2309.16609. our card
  2. Murray R Barrick and Michael K Mount. 1991. The big five personality dimensions and job performance: a meta-analysis. Personnel psychology, 44(1):1–26. find
  3. Sandra L Bem. 1981. Bem sex role inventory. Journal of personality and social psychology. find
  4. Bojana Bodroza, Bojana M Dinic, and Ljubisa Bojic. 2023. Personality testing of gpt-3: Limited temporal reliability, but highlighted social desirability of gpt-3’s personality instruments results. ArXiv preprint, abs/2306.04308. — [arXiv:2306.04308](https://arxiv.org/abs/2306.04308) find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  6. Patrick Butlin, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Constant, George Deane, Stephen M Fleming, Chris Frith, Xu Ji, et al. 2023. Consciousness in artificial intelligence: Insights from the science of consciousness. ArXiv preprint, abs/2308.08708. — [arXiv:2308.08708](https://arxiv.org/abs/2308.08708) find
  7. Julian Coda-Forno, Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. 2023. Inducing anxiety in large language models increases exploration and bias. ArXiv preprint, abs/2304.11111. — [arXiv:2304.11111](https://arxiv.org/abs/2304.11111) find
  8. Jacob. Cohen. 1968. Weighed kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213– 220. find
  9. Pim Cuijpers, Juan Li, Stefan G Hofmann, and Gerhard Andersson. 2010. Self-reported versus clinician-rated symptoms of depression as outcome measures in psychotherapy research on depression: a meta-analysis. Clinical psychology review, 30(6):768–778. find
  10. Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, and Kai Chen. 2023. Botchat: Evaluating llms’ capabilities of having multi-turn dialogues. ArXiv preprint, abs/2310.13650. find
  11. Michael B First. 2014. Structured clinical interview for the dsm (scid). The encyclopedia of clinical psychology, pages 1–6. find
  12. Jingsheng Gao, Yixin Lian, Ziyi Zhou, Yuzhuo Fu, and Baoyuan Wang. 2023. LiveChat: A large-scale personalized dialogue dataset automatically constructed from live streaming. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1538… find
  13. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. Available at SSRN 4316084. find
  14. Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. 2023a. Emotionally numb or empathetic? evaluating how llms feel using emotionbench. ArXiv preprint, abs/2308.03656. — [arXiv:2308.03656](https://arxiv.org/abs/2308.03656) find
  15. Jen-tse Huang, Wenxuan Wang, Man Ho Lam, Eric John Li, Wenxiang Jiao, and Michael R Lyu. 2023b. Chatgpt an enfj, bard an istj: Empirical study on personalities of large language models. ArXiv preprint, abs/2305.19926. — [arXiv:2305.19926](https://arxiv.org/abs/2305.19926) find
  16. Jen-tse Huang, Wenxuan Wang, Man Ho Lam, Eric John Li, Wenxiang Jiao, and Michael R Lyu. 2023c. Revisiting the reliability of psychological scales on large language models. ArXiv preprint, abs/2305.19926. — [arXiv:2305.19926](https://arxiv.org/abs/2305.19926) find
  17. Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. 2024. Who is chatgpt? benchmarking llms’ psychological portrayal using psychobench. In Proceedings of the Twelfth International Conference on Learning Representations. find
  18. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. ArXiv preprint, abs/2310.06825. find
  19. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. ArXiv preprint, abs/2401.04088. find
  20. Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2022. Mpi: Evaluating and inducing personality in pre-trained language models. ArXiv preprint, abs/2206.07550. find
  21. Saketh Reddy Karra, Son The Nguyen, and Theja Tulabandhula. 2022. Estimating the personality of white-box language models. ArXiv preprint, abs/2204.12000. — [arXiv:2204.12000](https://arxiv.org/abs/2204.12000) find
  22. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. find
  23. Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. 2023. Chatharuhi: Reviving anime character in reality via large language model. ArXiv preprint, abs/2308.09597. our card
  24. Xingxuan Li, Yutong Li, Shafiq Joty, Linlin Liu, Fei Huang, Lin Qiu, and Lidong Bing. 2022. Does gpt-3 demonstrate psychopathy? evaluating large language models from a psychological perspective. ArXiv preprint, abs/2212.10529. — [arXiv:2212.10529](https://arxiv.org/abs/2212.10529) find
  25. Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is GPT-3? an exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), pages 218–227, Abu Dhabi, UAE. Association for Computatio… find
  26. Danko Nikolić. 2010. The brain is a context machine. Review of psychology, 17(1):33–38. find
  27. OpenAI. 2022. Openai: Introducing chatgpt. find
  28. OpenAI. 2023. Gpt-4 technical report. our card
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… our card
  30. Keyu Pan and Yawen Zeng. 2023. Do llms possess a personality? making the mbti test an amazing evaluation for large language models. ArXiv preprint, abs/2307.16180. find
  31. Karl Pearson. 1920. Notes on the history of correlation. Biometrika, 13(1):25–45. find
  32. Peter Romero, Stephen Fitz, and Teruo Nakatsuma. 2023. Do gpt language models suffer from split personality disorder? the advent of substrate-free psychometrics. ResearchSquare preprint. find
  33. A John Rush, William Hiser, and Donna E Giles. 1987. A comparison of self-reported versus clinician-related symptoms in depression. The Journal of clinical psychiatry, 48(6):246–248. find
  34. Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. 2024. The self-perception and political biases of chatgpt. Human Behavior and Emerging Technologies, 2024(1):7115633. find
  35. Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. ArXiv preprint, abs/2307.00184. our card
  36. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023a. Character-LLM: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13153–13187, Singapore. Association for Computational Linguistics. our card
  37. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023b. Character-LLM: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13153–13187, Singapore. Association for Computational Linguistics. our card
  38. Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc. find
  39. Vera Sorin, Danna Brin, Yiftach Barash, Eli Konen, Alexander Charney, Girish Nadkarni, and Eyal Klang. 2023. Large language models (llms) and empathy-a systematic review. medRxiv, pages 2023–08. find
  40. Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, et al. 2023. Characterglm: Customizing chinese conversational ai characters with large language models. ArXiv preprint, abs/2311.16832. our card
  41. Charles Spearman. 1961. The proof and measurement of association between two things. find
  42. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. our card
  43. Timothy J Trull, Thomas A Widiger, J David Useda, Jay Holcomb, Bao-Tran Doan, Seth R Axelrod, Barry L Stern, and Beth S Gershuny. 1998. A structured interview for the assessment of the five-factor model of personality. Psychological assessment, 10(3):229. find
  44. Quan Tu, Chuanqi Chen, Jinpeng Li, Yanran Li, Shuo Shang, Dongyan Zhao, Ran Wang, and Rui Yan. 2023. Characterchat: Learning towards conversational ai with personalized social support. ArXiv preprint, abs/2308.10278. — [arXiv:2308.10278](https://arxiv.org/abs/2308.10278) find
  45. Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. 2024. Charactereval: A chinese benchmark for role-playing conversational agent evaluation. ArXiv preprint, abs/2401.01275. our card
  46. Rudolf Uher, Roy H Perlis, Anna Placentino, Mojca Zvezdana Dernovšek, Neven Henigsberg, Ole Mors, Wolfgang Maier, Peter McGuffin, and Anne Farmer. 2012. Self-report and clinician-rated measures of depression severity: can one replace the other? Depression and anxiety, 29(12):1043–1049. find
  47. Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2024. Openchat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations. our card
  48. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291. find
  49. Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. 2023b. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. ArXiv preprint, abs/2310.00746. our card
  50. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-incharacter-evaluating-personality-fidelity-role/