Elamur Research⌕
☰
home / papers / 2026 / Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

Summary

This LREC 2026 paper asks whether LLMs can be given a stable, human-like Big Five personality. It also criticizes the common practice of scoring each OCEAN trait separately, where random guessing already reaches 50%. The authors fine-tune five models on the Essays Dataset (Pennebaker and King, 1999), in which every long essay carries binary labels for the five traits. The models are Gemma-2-2B, Llama3.2-3B, Gemma-7B, Llama 3.1-8B and GPT-3.5-turbo-0125. Training uses SFT and the preference methods DPO and ORPO. Each tuned model then writes an essay and answers IPIP-NEO items. Three prompt formulations (S1–S3) test how robust the answers are to rewording.

Post-training makes questionnaire answers much less sensitive to rephrasing, which fixes a known weakness of pre-trained models. However, matching the complete five-trait profile stays roughly at chance, even when individual traits improve. The authors conclude that unguided essays carry too little signal for this task. They call for scenario-based data or interactive elicitation, and warn that per-trait metrics can overstate progress.

Key points

  • Data: 2,467 essays, about 1.9 million words in total and roughly 770 words each. Labels are close to balanced for every trait (e.g., Openness has 1,271 True and 1,196 False).
  • Three research questions: whether fine-tuning lowers response variance, whether SFT or preference tuning on unguided text induces personality, and whether safety alignment matters. The alignment question compares uncensored variants of Llama 3.1-8B, Llama 3.2-3B and Gemma-2-2B; Gemma-7B is excluded because it has no uncensored version.
  • Answers that break the response template are counted as NaN, which happens in about 6–10% of cases. OpenAI moderation removed around 300 samples, leaving an SFT set of about 2.1k.
  • Two SFT setups were tried: essays only, and essays with questionnaire items and answers mixed into the prompt.
  • Code, data and prompts are released under the MIT License at github.com/pkrajput/personality_induction. The authors note limitations: no human evaluation, a risk of overfitting, and possible mismatch between human self-report tests and LLM behaviour.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4zdw5tmzx58h

@inproceedings{rajput2026evaluation,
  title = {{Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?}},
  author = {Rajput, Prateek Kumar and Song, Yewei and Olatunji, Iyiola Emmanuel and Klein, Jacques and Bissyande, Tegawendé},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4zdw5tmzx58h},
  url = {https://doi.org/10.63317/4zdw5tmzx58h}
}

References

44 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Graham Caron and Shashank Srivastava. 2023. Manipulating the perceived personality traits of language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2370–2386, Singapore. Association for Computational Linguistics. find
  2. Yanquan Chen, Zhen Wu, Junjie Guo, Shujian Huang, and Xinyu Dai. 2024. Extroversion or introversion? controlling the personality of your large language models. arXiv preprint arXiv:2406.04583. — [arXiv:2406.04583](https://arxiv.org/abs/2406.04583) find
  3. Hans Christian, Derwin Suhartono, Andry Chowanda, and Kamal Z Zamli. 2021. Text based personality prediction from multiple social media data sources using pre-trained language model and model averaging. Journal of Big Data, 8(1):68. find
  4. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115. find
  5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  6. Golnoosh Farnadi, Susana Zoghbi, Marie-Francine Moens, and Martine De Cock. 2013. How well do your facebook status updates express your personality. In Proceedings of the 22nd edition of the annual Belgian-Dutch conference on machine learning (BENELEARN), page 88. find
  7. Ivar Frisch and Mario Giulianelli. 2024. Llm agents in interaction: Measuring personality consistency and linguistic alignment in interacting populations of large language models. arXiv preprint arXiv:2402.02896. find
  8. Matej Gjurković and Jan Šnajder. 2018. Reddit: A gold mine for personality prediction. In Proceedings of the second workshop on computational modeling of people’s opinions, personality, and emotions in social media, pages 87–97. find
  9. Jennifer Golbeck, Cristina Robles, Michon Edmondson, and Karen Turner. 2011. Predicting personality from twitter. In 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing, pages 149–156. IEEE. find
  10. Lewis R Goldberg. 1993. The structure of phenotypic personality traits. American psychologist, 48(1):26. find
  11. Lewis R Goldberg et al. 1999. A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality psychology in Europe, 7(1):7–28. find
  12. Alan J Gow, Martha C Whiteman, Alison Pattie, and Ian J Deary. 2005. Goldberg’s ‘ipip’big-five factor markers: Internal consistency and concurrent validation in scotland. Personality and Individual Differences, 39(2):317–329. find
  13. Akshat Gupta, Xiaoyang Song, and Gopala Anumanchipalli. 2024. Self-assessment tests are unreliable measures of llm personality. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 301–314. find
  14. Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–19. find
  15. Songqiao Han, Hailiang Huang, and Yuqing Tang. 2020. Knowledge of words: An interpretable approach for personality recognition from social media. Knowledge-Based Systems, 194:105550. find
  16. Airlie Hilliard, Cristian Munoz, Zekun Wu, and Adriano Soares Koshiyama. 2024. Eliciting big five personality traits in large language models: A textual analysis with classifier-driven approach. arXiv preprint arXiv:2402.08341. — [arXiv:2402.08341](https://arxiv.org/abs/2402.08341) find
  17. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189. find
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  19. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):… find our card
  20. Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2024. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36. find
  21. Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. 2023. Personallm: Investigating the ability of large language models to express personality traits. arXiv preprint arXiv:2305.02547. find
  22. Oliver P John and Sanjay Srivastava. 1999. The big-five trait taxonomy: History, measurement, and theoretical perspectives. In Lawrence A Pervin and Oliver P John, editors, Handbook of Personality: Theory and Research, 2nd edition, pages 102–138. Guilford Press, New York. find
  23. Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155. find our card
  24. François Mairesse, Marilyn A Walker, Matthias R Mehl, and Roger K Moore. 2007. Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of artificial intelligence research, 30:457–500. find
  25. Shengyu Mao, Ningyu Zhang, Xiaohan Wang, Mengru Wang, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2023. Editing personality for llms. arXiv preprint arXiv:2310.02168. — [arXiv:2310.02168](https://arxiv.org/abs/2310.02168) find
  26. Jessica L Maples, Li Guan, Nathan T Carter, and Joshua D Miller. 2014. A test of the international personality item pool representation of the revised neo personality inventory and development of a 120-item ipip-based measure of the five-factor model. Psychological assessment, 26(4):1070. find
  27. Robert R McCrae and Oliver P John. 1992. An introduction to the five-factor model and its applications. Journal of personality, 60(2):175–215. find
  28. Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is gpt-3? an exploration of personality, values and demographics. arXiv preprint arXiv:2209.14338. find
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. find
  30. Gregory Park, H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Michal Kosinski, David J Stillwell, Lyle H Ungar, and Martin EP Seligman. 2015. Automatic personality assessment through social media language. Journal of personality and social psychology, 108(6):934. find
  31. James W Pennebaker and Laura A King. 1999. Linguistic styles: language use as an individual difference. Journal of personality and social psychology, 77(6):1296. find
  32. Philip M Podsakoff, Scott B MacKenzie, and Nathan P Podsakoff. 2012. Sources of method bias in social science research and recommendations on how to control it. Annual review of psychology, 63(1):539–569. find
  33. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. find
  34. Matthew Renze. 2024. The effect of sampling temperature on problem solving in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346–7356. find
  35. Aadesh Salecha, Molly E Ireland, Shashanka Subrahmanya, João Sedoc, Lyle H Ungar, and Johannes C Eichstaedt. 2024. Large language models show human-like social desirability biases in survey responses. arXiv preprint arXiv:2405.06058. — [arXiv:2405.06058](https://arxiv.org/abs/2405.06058) find
  36. H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin EP Seligman, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9)… find
  37. Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. arXiv preprint arXiv:2307.00184. find our card
  38. Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role play with large language models. Nature, 623(7987):493–498. find our card
  39. MEDIANTA TARIGAN et al. 2024. The validity and reliability of the indonesian ipip-neo-120. TPM: Testing, Psychometrics, Methodology in Applied Psychology, 31(1). find
  40. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  41. Tal Yarkoni. 2010. Personality in 100,000 words: A large-scale analysis of personality and word use among bloggers. Journal of research in personality, 44(3):363–373. find
  42. Shiwei Zhang, Yiyang Du, Guanzhong Liu, Zhao Yan, and Yunbo Cao. 2022. G4: grounding-guided goal-oriented dialogues generation with multiple documents. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 108–114. find
  43. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find
  44. Code-Grounded Reproducibility Details A. To facilitate reproducibility, this appendix reports the key implementation details extracted directly from the released codebase. 2 We document the training hyperparameters, inference configuration, prompt templates, and dataset statistics used across all ex… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rajput-2026-evaluation-drift-llm-personality-induction/