Elamur Research⌕
☰
home / papers / 2026 / HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns

HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns

Paper · 2026 · Xintao Wang, Jian Yang, Weiyuan Li et al.

Summary

LLM-based role-playing agents usually treat personality as a set of independent labels, each tied to a stereotyped behavior. This misses how several psychological tendencies combine or clash in real people, and it leads to persona drift and inconsistent behavior. HumanLLM instead treats these tendencies as causal forces that interact. The authors compiled 244 patterns: 100 traits from Goldberg's Big Five markers and 144 social-cognitive phenomena, each distilled from roughly 50 papers. They then generated 11,359 multi-character scenarios in which 2–5 patterns support, oppose, or moderate one another. Each scenario has a multi-turn dialogue whose turns contain inner thoughts, actions, and speech. Models are trained with supervised fine-tuning on one target character's lines. An LLM judge scores them using checklists at both the pattern level and the scenario level. This evaluation agrees closely with human ratings. The fine-tuned HumanLLM-8B beats the much larger Qwen3-32B on multi-pattern dynamics. The authors argue that convincing anthropomorphism requires modeling the mental processes behind behavior, not just imitating its surface.

Key points

  • Resource: 244 patterns grounded in about 12,000 academic papers, plus 11,359 scenarios with 2–6 characters each. Dataset, code, and models (HumanLLM-8B/32B) are released at github.com/YJGoodbye2024/HumanLLM.
  • Dual-level checklists:
    • pattern-level, with 12–15 indicators per pattern;
    • scenario-level, with 2–6 expected tendencies per character.
  • The checklist evaluation correlates strongly with human judgments (r = 0.90). The authors also observe that holistic ratings conflate how faithfully a model simulates behavior with how socially desirable that behavior appears.
  • On multi-pattern dynamics, HumanLLM-8B scores higher than Qwen3-32B even though it is 4× smaller in parameter count. Models are tested in in-domain, out-of-domain, and mixed settings, and on the external benchmarks LifeChoice and CroSS-MR.
  • Limitations and ethics:
    • The judge is GPT-5-mini, run three times with standard deviation below 2.1%.
    • Training data is fully synthetic, though human-rated source faithfulness is a mean of 3.50 (α = 0.73).
    • The authors flag risks of WEIRD cultural bias, manipulation, and parasocial attachment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Xintao Wang, Jian Yang, Weiyuan Li, Rui Xie, Jen-tse Huang, Jun Gao, Shuai Huang, Yueping Kang, Yuanli Guo, Hongwei Feng and Yanghua Xiao. 2026. HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1783

@inproceedings{wang2026humanllm,
  title = {{HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns}},
  author = {Xintao Wang, Jian Yang, Weiyuan Li, Rui Xie, Jen-tse Huang, Jun Gao, Shuai Huang, Yueping Kang, Yuanli Guo, Hongwei Feng and Yanghua Xiao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1783},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1783}
}

References

51 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Anthropic’s transparency hub: Model report. https://www.anthropic.com/ transparency/model-report. Accessed: 2026-01-03. ↗
  2. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. LongAlign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376–1395, Miami, Florida, USA. Association f…
  3. David M Buss. 2024. Evolutionary psychology: The new science of the mind. Routledge. find
  4. Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. 2024a. From Persona to Personalization: A Survey on Role-Playing Language Agents. Preprint,…
  5. Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. Persona vectors: Monitoring and controlling character traits in language models. Preprint, arXiv:2507.21509. — [arXiv:2507.21509](https://arxiv.org/abs/2507.21509) find
  6. Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. 2024b. ToMBench: Benchmarking Theory of Mind in Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling…
  7. Vanessa Cheung, Maximilian Maier, and Falk Lieder. 2025. Large language models show amplified cognitive biases in moral decision-making. Proceedings of the National Academy of Sciences, 122(25):e2412015122. find
  8. Robert B Cialdini. 2009. Influence: Science and practice, volume 4. Pearson education Boston. find
  9. Lee J Cronbach and Paul E Meehl. 1955. Construct validity in psychological tests. Psychological bulletin, 52(4):281. find
  10. Edward L Deci and Richard M Ryan. 2000. The" what" and" why" of goal pursuits: Human needs and the self-determination of behavior. Psychological inquiry, 11(4):227–268. find
  11. Google DeepMind. 2025. Gemini 3 pro model card. Model card, Google DeepMind. find
  12. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025a. Deepseek-r1: Incentivizing reasoning capability in l…
  13. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025b. Deepseek-v3.2: Pushing the frontier of ope…
  14. John M. Digman. 1990. Personality structure: Emergence of the five-factor model. Annual Review of Psychology, 41:417–440. find
  15. Lewis R Goldberg. 1992. The development of markers for the big-five factor structure. Psychological assessment, 4(1):26. find
  16. Pengrui Han, Rafal Kocielnik, Peiyang Song, Ramit Debnath, Dean Mobbs, Anima Anandkumar, and R. Michael Alvarez. 2025. The personality illusion: Revealing dissociation between self-reports & behavior in llms. Preprint, arXiv:2509.03730. — [arXiv:2509.03730](https://arxiv.org/abs/2509.03730) find
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2023. Aligning ai with shared human values. Preprint, arXiv:2008.02275.
  18. Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3):61–83. find
  19. Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael Lyu. 2024a. On the reliability of psychological scales on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6152–6173. find
  20. Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. 2024b. Apathetic or empathetic? evaluating llms’ emotional alignments with humans. Advances in Neural Information Processing Systems, 37:97053–97087. find
  21. Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael Lyu. 2024c. On the humanity of conversational ai: Evaluating the psychological portrayal of llms. In The Twelfth International Conference on Learning Representations. find
  22. Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2025. Moralbench: Moral evaluation of llms. Preprint, arXiv:2406.04428. — [arXiv:2406.04428](https://arxiv.org/abs/2406.04428) find
  23. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  24. Kurt Lewin. 2013. Principles of topological psychology. Read Books Ltd. find
  25. Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. 2023. Chatharuhi: Reviving anime character in reality via large language model. Preprint, arXiv:2308.09597.
  26. Weiyuan Li, Xintao Wang, Siyu Yuan, Rui Xu, Jiangjie Chen, Qingqing Dong, Yanghua Xiao, and Deqing Yang. 2025. Curse of knowledge: When complex evaluation context benefits yet biases llm judges. Preprint, arXiv:2509.03419. — [arXiv:2509.03419](https://arxiv.org/abs/2509.03419) find
  27. Olivia Macmillan-Scott and Mirco Musolesi. 2024. (ir)rationality and cognitive biases in large language models. Preprint, arXiv:2402.09193. — [arXiv:2402.09193](https://arxiv.org/abs/2402.09193) find
  28. Man Tik Ng, Hui Tung Tse, Jen-tse Huang, Jingjing Li, Wenxuan Wang, and Michael R Lyu. 2024. How well can llms echo us? evaluating ai chatbots’ role-play ability with echo. arXiv preprint arXiv:2404.13957. — [arXiv:2404.13957](https://arxiv.org/abs/2404.13957) find
  29. OpenAI. 2025. Gpt-5 system card. System card, OpenAI. find
  30. Samuel J. Paech. 2024. Eq-bench: An emotional intelligence benchmark for large language models. Preprint, arXiv:2312.06281.
  31. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. Preprint, arXiv:2304.03442. find
  32. Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. Ai psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19(5):808–826. find
  33. John F Rauthmann, David Gallardo-Pujol, Esther M Guillaume, Elysia Todd, Christopher S Nave, Ryne A Sherman, Matthias Ziegler, Ashley Bell Jones, and David C Funder. 2014. The situational eight diamonds: a taxonomy of major dimensions of situation characteristics. Journal of personality and social p… find
  34. Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. 2024. EmoBench: Evaluating the Emotional Intelligence of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (… find
  35. Sarah Schröder, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. 2025. Large language models do not simulate human psychology. Preprint, arXiv:2508.06950. — [arXiv:2508.06950](https://arxiv.org/abs/2508.06950) find
  36. Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2025. Personality traits in large language models. Preprint, arXiv:2307.00184.
  37. Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role play with large language models. Nature, 623(7987):493–498.
  38. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. Preprint, arXiv:2302.08399. find
  39. Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, Wei Wang, and Yanghua Xiao. 2025. CoSER: Coordinating LLM-Based Persona Simulation of Established Roles. Preprint, arXiv:2502.09082. — [arXiv:2502.09082](https://arxiv.org/abs/2502.09082) find
  40. Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. 2024. InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews. In Proceedings of the 62nd Annual M…
  41. Rui Xu, Dakuan Lu, Xiaoyu Tan, Xintao Wang, Siyu Yuan, Jiangjie Chen, Wei Chu, and Yinghui Xu. 2024a. MINDECHO: Role-Playing Language Agents for Key Opinion Leaders. arXiv preprint. find
  42. Rui Xu, Xintao Wang, Jiangjie Chen, Siyu Yuan, Xinfeng Yuan, Jiaqing Liang, Zulong Chen, Xiaoqing Dong, and Yanghua Xiao. 2024b. Character is destiny: Can role-playing language agents make persona-driven decisions? Preprint, arXiv:2404.12138. — [arXiv:2404.12138](https://arxiv.org/abs/2404.12138) find
  43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  44. Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, and Deqing Yang. 2024. Evaluating character understanding of large language models via character profiling from fictional works. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processi… find
  45. Yutong Zhang, Dora Zhao, Jeffrey T. Hancock, Robert Kraut, and Diyi Yang. 2025. The rise of ai companions: How human-chatbot relationships influence well-being. Preprint, arXiv:2506.12605. find
  46. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A Trainable Agent for Role-Playing. Preprint, arXiv:2310.10158.
  47. Jiaxu Zhou, Jen-tse Huang, Xuhui Zhou, Man Ho Lam, Xintao Wang, Hao Zhu, Wenxuan Wang, and Maarten Sap. 2025a. The pimmur principles: Ensuring validity in collective behavior of llm societies. arXiv preprint arXiv:2509.18052. — [arXiv:2509.18052](https://arxiv.org/abs/2509.18052) find
  48. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124–1131. find
  49. Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2023. Characterglm: Customizing chinese conversational ai characters with large…
  50. Xuhui Zhou, Zhe Su, Sophie Feng, Jiaxu Zhou, Jen-tse Huang, Hsien-Te Kao, Spencer Lynch, Svitlana Volkova, Tongshuang Wu, Anita Woolley, and 1 others. 2025b. Sotopia-s4: a user-friendly system for flexible, customizable, and large-scale social simulation. In Proceedings of the 2025 Conference of the… find
  51. Huiqi Zou, Pengda Wang, Zihan Yan, Tianjun Sun, and Ziang Xiao. 2025. Can llm "self-report"?: Evaluating the validity of self-report scales in measuring personality design in llm-based chatbots. Preprint, arXiv:2412.00207. — [arXiv:2412.00207](https://arxiv.org/abs/2412.00207) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-humanllm-benchmarking-improving-llm-anthropomorphism/