Elamur Research⌕
☰
home / papers / 2023 / PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

Paper · 2023 · Hang Jiang, Xiajie Zhang, Xubo Cao et al.

Summary

Chatbots with assigned personalities are increasingly common, yet it is rarely checked whether such "LLM personas" actually behave in line with the traits they are given. The authors test this for GPT-3.5 and GPT-4 using the Big Five model. They prompt 320 personas, covering every binary combination of the five traits, to take the 44-item Big Five Inventory (BFI) and to write an 800-word personal story without naming their traits. The stories are analyzed with LIWC-22 and compared against human texts from the Essays dataset. Human annotators and LLM evaluators then rate a sample of the stories on six quality dimensions and guess the author's personality. The personas' self-reported BFI scores match their assigned traits, with large effect sizes on all five dimensions. Their writing also shows trait-related linguistic markers resembling those found in human writing. Human readers identify some traits with accuracy up to 80%, but they do noticeably worse when told an AI wrote the text. This highlights the importance of disclosing AI authorship.

Key points

  • Models tested: GPT-3.5-turbo-0613 and GPT-4-0613, run at temperature 0.7. Results for LLaMA-2 appear in the appendices, but its stories were judged unsuitable for human evaluation because they were repetitive and often named traits directly.
  • Design: 32 personality types × 10 personas = 320 personas, each created with a simple system prompt listing its five trait descriptors (e.g., extroverted/introverted).
  • A lexicon-based check found explicit trait mentions in 96.56% of GPT-3.5 stories versus 31.87% for GPT-4, so the human study used only GPT-4 stories.
  • Human evaluation covered 32 stories, each judged by five annotators recruited on Prolific. The six dimensions were readability, personalness, redundancy, cohesiveness, likeability, and believability.
  • The human baseline for the LIWC comparison is the Essays corpus (Pennebaker and King, 1999), which contains texts from 2,467 participants. Code, data, and annotations are released at github.com/hjian42/PersonaLLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPsychology · 1,649 worksSafety Research · 626 worksClinical Psychology · 21 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy and Jad Kabbara. 2023. PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits. arXiv:2305.02547.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.02547 · arXiv 2305.02547

@misc{jiang2023personallm,
  title = {{PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits}},
  author = {Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy and Jad Kabbara},
  year = {2023},
  doi = {10.48550/arxiv.2305.02547},
  eprint = {2305.02547},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.02547}
}

References

66 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337–371. PMLR. find
  2. Jacob Andreas. 2022. Language models as agent models. arXiv preprint arXiv:2212.01681.
  3. Jerry B Ayers, W Louis Bashaw, and James A Wash. 1969. A study of the validity of the sixteen personality factor questionnaire in predicting high school academic achievement. Educational and Psychological Measurement, 29(2):479–484. find
  4. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. find
  5. Stephen R Briggs. 1992. Assessing the five-factor model of personality description. Journal of personality, 60(2):253–293. find
  6. Alessandro Bruno and Gurmeet Singh. 2022. Personality traits prediction from text via machine learning. In 2022 IEEE World Conference on Applied Intelligence and Computing (AIC), pages 588–594. IEEE. find
  7. Xubo Cao and Michal Kosinski. 2023. Chatgpt can accurately predict public figures’ perceived personalities without any training. find
  8. Graham Caron and Shashank Srivastava. 2022. Identifying and manipulating the personality traits of language models. arXiv preprint arXiv:2212.10276. — [arXiv:2212.10276](https://arxiv.org/abs/2212.10276) find
  9. Heather EP Cattell and Alan D Mead. 2008. The sixteen personality factor questionnaire (16pf). The SAGE handbook of personality theory and assessment, 2:135–159. find
  10. Raymond B Cattell. 1957. Personality and motivation structure and measurement. find
  11. Cheng-Han Chiang and Hung-yi Lee. 2023a. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics… find
  12. Cheng-Han Chiang and Hung-yi Lee. 2023b. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937. find
  13. Paul T Costa and Robert R McCrae. 2008. The revised neo personality inventory (neo-pi-r). The SAGE handbook of personality theory and assessment, 2(2):179– 198. find
  14. James Patrick Curran Jr. 1970. Analysis of factors effecting interpersonal attraction in the dating situation. University of Illinois at Urbana-Champaign. find
  15. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051.
  16. Boele De Raad. 2000. The big five personality factors: the psycholexical approach to personality. Hogrefe & Huber Publishers. find
  17. Cassandra DiRienzo, Jayoti Das, Wonhi Synn, Jeremy Kitts, and Kyle McGrath. 2010. The relationship between mbti® and academic performance: A study across academic disciplines. Journal of Psychological Type. find
  18. Golnoosh Farnadi, Susana Zoghbi, Marie-Francine Moens, and Martine De Cock. 2013. Recognising personality traits using facebook status updates. In Proceedings of the international AAAI conference on web and social media, volume 7, pages 14–18. find
  19. Ali-Reza Feizi-Derakhshi, Mohammad-Reza Feizi-Derakhshi, Majid Ramezani, Narjes Nikzad-Khasmakhi, Meysam Asgari-Chenaghlu, Taymaz Akan, Mehrdad Ranjbar-Khadivi, Elnaz Zafarni-Moattar, and Z Jahanbakhsh-Naghadeh. 2021. The state-of-the-art in text-based automatic personality prediction. arXiv preprin… — [arXiv:2110.01186](https://arxiv.org/abs/2110.01186) find
  20. Adithya V Ganesan, Yash Kumar Lal, August Håkan Nilsson, and H Andrew Schwartz. 2023. Systematic evaluation of gpt-3 for zero-shot personality estimation. arXiv preprint arXiv:2306.01183. find
  21. Lewis R Goldberg. 2013. An alternative “description of personality”: The big-five factor structure. In Personality and Personality Disorders, pages 34–47. Routledge. find
  22. Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S Bernstein. 2021. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–14. find
  23. Denise A Hines and Kimberly J Saudino. 2008. Personality and intimate partner aggression in dating relationships: the role of the “big five”. Aggressive Behavior: Official Journal of the International Society for Research on Aggression, 34(6):593–604. find
  24. Jacob B Hirsh and Jordan B Peterson. 2009. Personality and language use in self-narratives. Journal of research in personality, 43(3):524–527. find
  25. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research. find
  26. Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2022a. Evaluating and inducing personality in pre-trained language models.
  27. Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2022b. Mpi: Evaluating and inducing personality in pre-trained language models. arXiv preprint arXiv:2206.07550.
  28. Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. IMaT: Unsupervised text attribute transfer via iterative matching and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on N… find
  29. Oliver P John, Sanjay Srivastava, et al. 1999. The big-five trait taxonomy: History, measurement, and theoretical perspectives. find
  30. Jad Kabbara and Jackie Chi Kit Cheung. 2016. Stylistic transfer in natural language generation systems using recurrent neural networks. In Proceedings of the Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods, pages 43–47, Austin, TX. Association for Comp… find
  31. Trina Ana Kajzer. 2023. Exploring the role of personality traits and attachment styles in shaping dating app user experience. find
  32. Saketh Reddy Karra, Son The Nguyen, and Theja Tulabandhula. 2022. Estimating the personality of white-box language models. arXiv preprint arXiv:2204.12000. — [arXiv:2204.12000](https://arxiv.org/abs/2204.12000) find
  33. R Bryan Kennedy and D Ashley Kennedy. 2004. Using the myers-briggs type indicator® in career counseling. Journal of employment counseling, 41(1):38–43. find
  34. Chang H Lee, Kyungil Kim, Young Seok Seo, and Cindy K Chung. 2007. The relations between personality and language use. The Journal of general psychology, 134(4):405–413. find
  35. Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. 2023. Tailoring personality traits in large language models via unsupervisedly-built personalized lexicons. arXiv preprint arXiv:2310.16582. — [arXiv:2310.16582](https://arxiv.org/abs/2310.16582) find
  36. Xingxuan Li, Yutong Li, Linlin Liu, Lidong Bing, and Shafiq Joty. 2022. Is gpt-3 a psychopath? evaluating large language models from a psychological perspective. arXiv preprint arXiv:2212.10529. — [arXiv:2212.10529](https://arxiv.org/abs/2212.10529) find
  37. Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, et al. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arxiv abs/2303.16634 (2023).
  38. John W Lounsbury, Teresa Hutchens, and James M Loveland. 2005. An investigation of big five personality traits and career decidedness among early and middle adolescents. Journal of career assessment, 13(1):25–39. find
  39. Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page…
  40. François Mairesse and Marilyn Walker. 2007. Personage: Personality generation for dialogue. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 496–503. find
  41. François Mairesse and Marilyn A Walker. 2011. Controlling user perceptions of linguistic style: Trainable generation of personality traits. Computational Linguistics, 37(3):455–488.
  42. François Mairesse, Marilyn A Walker, Matthias R Mehl, and Roger K Moore. 2007. Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of artificial intelligence research, 30:457–500.
  43. Shengyu Mao, Ningyu Zhang, Xiaohan Wang, Mengru Wang, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2023. Editing personality for llms. arXiv preprint arXiv:2310.02168. — [arXiv:2310.02168](https://arxiv.org/abs/2310.02168) find
  44. Isabel Briggs Myers. 1962. The myers-briggs type indicator: Manual (1962). find
  45. Isabel Briggs Myers. 1985. A Guide to the Development and Use of the Myers-Briggs Type Indicator: Manual. Consulting Psychologists Press. find
  46. Jon Oberlander and Scott Nowson. 2006. Whose thumb is it anyway? classifying author personality from weblog text. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 627– 634. find
  47. Melissa C O’Connor and Sampo V Paunonen. 2007. Big five personality predictors of post-secondary academic performance. Personality and Individual differences, 43(5):971–990. find
  48. Keyu Pan and Yawen Zeng. 2023. Do llms possess a personality? making the mbti test an amazing evaluation for large language models. arXiv preprint arXiv:2307.16180.
  49. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442. find
  50. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001. find
  51. James W Pennebaker and Anna Graybeal. 2001. Patterns of natural language use: Disclosure, personality, and social integration. Current Directions in Psychological Science, 10(3):90–93. find
  52. James W Pennebaker and Laura A King. 1999. Linguistic styles: language use as an individual difference. Journal of personality and social psychology, 77(6):1296. find
  53. Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Assigning personality/profile to a chatting machine for coherent conversation generation. In Ijcai, pages 4279–4285. find
  54. Haocong Rao, Cyril Leung, and Chunyan Miao. 2023. Can chatgpt assess human personalities? a general evaluation framework. arXiv preprint arXiv:2303.01248. find
  55. Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. arXiv preprint arXiv:2307.00184.
  56. JM Schuerger. 1995. Career assessment and the sixteen personality factor questionnaire. Journal of Career Assessment, 3(2):157–175. find
  57. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158.
  58. Marcin Skowron, Marko Tkalčič, Bruce Ferwerda, and Markus Schedl. 2016. Fusing social media cues: personality prediction from twitter and instagram. In Proceedings of the 25th international conference companion on world wide web, pages 107–108. find
  59. Michael M Tadesse, Hongfei Lin, Bo Xu, and Liang Yang. 2018. Personality predictions based on user behavior on the facebook social media platform. IEEE Access, 6:61959–61969. find
  60. Xintao Wang, Yaying Fei, Ziang Leng, and Cheng Li. 2023a. Does role-playing chatbots capture the character personalities? assessing personality traits for role-playing chatbots. arXiv preprint arXiv:2310.17976. — [arXiv:2310.17976](https://arxiv.org/abs/2310.17976) find
  61. Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. 2023b. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746.
  62. Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541. find
  63. Lionel Wong, Gabriel Grand, Alexander K Lew, Noah D Goodman, Vikash K Mansinghka, Jacob Andreas, and Joshua B Tenenbaum. 2023. From word models to world models: Translating from natural language to the probabilistic language of thought. arXiv preprint arXiv:2306.12672. find
  64. Tao Yang, Tianyuan Shi, Fanqi Wan, Xiaojun Quan, Qifan Wang, Bingzhe Wu, and Jiaxiang Wu. 2023. Psycot: Psychological questionnaire as powerful chain-of-thought for personality detection. arXiv preprint arXiv:2310.20256. find
  65. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243.
  66. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large language models transform computational social science? arXiv preprint arXiv:2305.03514.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2023-personallm-investigating-ability-large-language/